Unable to set container_base in config

[mtwest2718]

• OS Version: Red Hat Enterprise Linux 8.8 (Ootpa)
• Kernel: 4.18.0-477.21.1.el8_8.x86_64
• System Python3: 3.6.8
• Lmod version: Release 4.5.2 (2020-07-30)
• singularity-ce version 3.11.5-1.el8
• Installing shpc within a venv environment from the public git repo
  • shpc version: 0.1.26

On my single node server, I am unable to specify a new location to put pulled & managed container images. shpc config set container_base /usr/share/shpc/image_files/

The error I get is below, which is rather strange as I have successfully done this on an el9 system with Python 3.9 using the same release version (0.1.26). And I set up that machine yesterday, so I don't think anything has changed within ~24 hours to have borked this functionality.

  File "/opt/shpc/bin/shpc", line 11, in <module>
    load_entry_point('singularity-hpc==0.1.26', 'console_scripts', 'shpc')()
  File "/opt/shpc/lib64/python3.6/site-packages/shpc/client/__init__.py", line 556, in run_shpc
    main(args=args, parser=parser, extra=extra, subparser=helper)
  File "/opt/shpc/lib64/python3.6/site-packages/shpc/client/config.py", line 31, in main
    quiet=args.quiet, settings_file=args.settings_file, validate=validate
  File "/opt/shpc/lib64/python3.6/site-packages/shpc/main/__init__.py", line 36, in get_client
    from shpc.main.modules.lmod import Client
  File "/opt/shpc/lib64/python3.6/site-packages/shpc/main/modules/lmod.py", line 5, in <module>
    from .base import ModuleBase
  File "/opt/shpc/lib64/python3.6/site-packages/shpc/main/modules/base.py", line 14, in <module>
    import shpc.main.container as container
  File "/opt/shpc/lib64/python3.6/site-packages/shpc/main/container/__init__.py", line 1, in <module>
    from .config import ContainerConfig
  File "/opt/shpc/lib64/python3.6/site-packages/shpc/main/container/config.py", line 8, in <module>
    import shpc.main.container.update as update
AttributeError: module 'shpc.main' has no attribute 'container'

[vsoch]

Are you able to test with python 3.7? This looks to me like a circular import issue, and one that would be OK with 3.7. 3.6 is from 2016 so we have never tested on it (and I'm tempted to not support it). If you are saying the same thing worked on 3.6 a day earlier, try doing a sanity check as follows:

# Do this where the import is failing at the top, so main/container/config 
import sys

# Look for shpc.main here
print(sys.modules)

# Also look at what it seen
print(dir(shpc.main))

And try that between a working and non working version. Usually you can take an error like this directly - it doesn't see that module name "container" as a member of shpc.main. Figuring out what it does see is the first step, and then the reason for it the second step.

[vsoch]

And one general suggestion - it looks like you installed shpc to site packages? I'd normally recommend an install to a custom location to ensure you aren't writing any assets to the site-packages location.,

[mtwest2718]

So this is a el8 system, which comes with python 3.6 as its default python. I need to check with the owner of the machine to see if they want to upgrade to 3.7 or higher and if this will break anything else.

which is rather strange as I have successfully done this on an el9 system with Python 3.9 using the same release version (0.1.26).

No, I was just remarking that I had successfully set it up on another system with python 3.9 the day before using the same shpc release version. This made me suspect it might be a python version issue.

[mtwest2718]

Also I installed shpc in a python venv virtual environment in /opt exclusively just for shpc. I specified the module_base and container_base to be elsewhere to keep them away from this install location.

[mtwest2718]

From a working version on an el9 system.

>>> import shpc.main
>>> print(dir(shpc.main))
['Settings', '__author__', '__builtins__', '__cached__', '__copyright__', '__doc__', '__file__', '__license__', '__loader__', '__name__', '__package__', '__path__', '__spec__', 'get_client', 'logger', 'schemas', 'settings', 'shpc']

Also don't see container as an attribute of shpc.main.

[vsoch]

Also don't see container as an attribute of shpc.main.

The reason is because we are binding a submodule to a name, namely container to shpc.main. That was the release note I linked above:

Circular imports involving absolute imports with binding a submodule to a name are now supported. (Contributed by Serhiy Storchaka in bpo-30024.)

I was curious if it might show up (but does not) thank you for checking.

Also I installed shpc in a python venv virtual environment in /opt exclusively just for shpc. I specified the module_base and container_base to be elsewhere to keep them away from this install location.

Gotcha - I see that. I think I normally just clone and then do pip install . so I could easily pull and update the software.

Without being able to debug it myself I don't have any strong inkling for why this behavior arises, but I suspect it's Python version, and if it works for one but not the other, the likely solution is to use "the other" and not the older Python from 2016. Maybe @marcodelapierre has ideas.

[mtwest2718]

it looks like you installed shpc to site packages

So the oddity the above comment is referencing just clicked in my head.

• I pip install git+https://url.git directly into a python venv located at /opt/shpc
• Set the container_base and module_base to be NOT within that directory structure.
• BUT, I did not heed the warning about moving the wrapper_base as well.

Hence

[cloud-user@minicondor ~]$ module load shpc/julia
[cloud-user@minicondor ~]$ which julia
/opt/shpc/lib64/python3.9/site-packages/modules/julia/1.9.3/bin/julia

which will indeed make updating the shpc repo a pain. So will definitely be changing that.

[mtwest2718]

Because this issue came up due to what we all expect is a python version problem, could some minimal dependency requirements be put in the docs? Lots of folks on el8 or el7 systems are using older base-install versions of python (usually 3.6.8) and I don't want anyone getting caught out like I did.

[vsoch]

We should update here, which is the standard place to put that: https://github.com/singularityhub/singularity-hpc/blob/e1e66adeb56035ce4b3c933b96fe2963df4e9d09/setup.py#L103

I very likely copy pasted that metadata from elsewhere. I'll update to 3.7 shortly.

[mtwest2718]

Not being familiar with the setup.py syntax, if someone tries to install singularity-hpc with < Python3.7, will it spit out an informative error as to what the problem is?

[vsoch]

No, the setup.py cannot enforce that. However, it's a standard setup classifier (e.g., the entire listing https://vsoch.github.io/pypi-classifiers/) that is rendered on pypi and wherever else the package is distributed, so it's a convention.

[marcodelapierre]

I think I had similar errors recently, and I have the vague memory the key was indeed the Python version. My goldfish memory had retained the essential info to always start with Python >= 3.7, to be safe with newer versions of typical dependency packages from pip. And I suspect this might need to be lifted pretty soon... see e.g. https://devguide.python.org/versions/

[vsoch]

This picture is epic. Wow. And I didn't realize how long 2.7 stuck around! I always harped on the "still using python 2.x" crowd but... they weren't being absolutely terrible?

[marcodelapierre]

It is epic, agree!!

[mtwest2718]

The problem for system maintainers using "stable" LTS OS releases is that we run a number of years behind the bleeding edge. And also wish to avoid python version ridiculousness if possible.

So any call-outs, reminders or automated flagging for unsupported python language versions for a package is ❤️.

[marcodelapierre]

Many thanks for your feedback @mtwest2718 .

I hear you, in my previous role we did have a number of on-prem with 4-5 yo Linux distros, whose system Python was stuck to versions that can create version conflicts in present days.

• @vsoch has just updated the version requirement in setup.py ; this will be soon reflected in the PYPI page for Singularity HPC (https://pypi.org/project/singularity-hpc/)
• @mtwest2718 do you reckon we should also highlight this requirement somewhere else, e.g. in the repo README?
• finally, do you think that providing links to how to install more recent Python versions can be beneficial to the user base you are aware of?

Thanks, Marco

[vsoch]

Look at some of these new beautiful features in 3.9 (which is already kind of old? I have 3.11 on my system). So nice though!

I know I love programming because I still get excited/happy about the small things. <3

[marcodelapierre]

I love the small things, too ... I must keep advocating for Nextflow to migrate to Python!!

[vsoch]

haha but @marcodelapierre ... it's so Groovy!!!

Can't help myself, haha. I've definitely made that joke one too many times...

[marcodelapierre]

Aahhaha first time with me LOL