Python 3.11 so3g import not working

[kmharrington]

(Edit because I realize the first version wasn't the actual error).

Trying to get sotodlib built with python 3.11. (Already gave up on 3.12). Running import so3g import the first time fails with the error below. But is works if you just run it again

Python 3.11.10 (main, Oct  3 2024, 07:29:13) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from sotodlib import core
RuntimeError: dynamic loading error: loading 'so3g' from '/home/kharrington/.local/bebop/anaconda3/2024.06/lib/python3.11/site-packages/so3g/libso3g.so': /home/kharrington/.conda/envs/so-tod/lib/python3.11/site-packages/mkl/../../../libiomp5.so: undefined symbol: __tgt_register_ptask_services

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/kharrington/.local/bebop/anaconda3/2024.06/lib/python3.11/site-packages/sotodlib/core/__init__.py", line 13, in <module>
    from .flagman import FlagManager
  File "/home/kharrington/.local/bebop/anaconda3/2024.06/lib/python3.11/site-packages/sotodlib/core/flagman.py", line 5, in <module>
    from so3g.proj import Ranges, RangesMatrix
  File "/home/kharrington/.local/bebop/anaconda3/2024.06/lib/python3.11/site-packages/so3g/__init__.py", line 21, in <module>
    load_pybindings([__path__[0] + '/libso3g'], name='so3g')
  File "/home/kharrington/.local/bebop/anaconda3/2024.06/lib/python3.11/site-packages/so3g/load_pybindings.py", line 46, in load_pybindings
    m = dload.load_dynamic(name, name, path + lib_suffix)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
SystemError: <built-in function load_dynamic> returned a result with an exception set

so putting this at the top of scripts is now my fix

try:
    import so3g
except:
    import so3g

[mhasself]

Short answer, this isn't really an sotodlib problem... There are shared library conflicts, likely between so3g and numpy or scipy or another heavy mathematical package coming from conda.

Where did you get so3g? Are you using soconda, or pip installing so3g, or building so3gf it from source? soconda is worth the investment, and the installation docs are quite good. This library issue (likely) is exactly the kind of problem it was set up to solve!

The import order of dependencies is what is mattering here -- for example in the past we had a conflict where scipy and so3g would only import in one order, not the other. But when that is the case, it's a sign something is going to be weird downstream, even if the import seems to work.

[tskisner]

Based on the errors, it looks like this is on a laptop / workstation using commercial anaconda installed to your user account. For consideration, here are some things to try:

Option 1

• Uninstall anaconda completely and make sure any cruft that it put in your shell resource files is gone.
• Install a base conda environment using the miniforge installer. If this is macos, use the x86-64 installer, even if you are on an arm64 system
• Either allow the installer to modify your shell resource file so that the conda init is sourced, or manually do source path_to_conda/etc/profile.d/conda.sh
• Then create a new environment with as many packages as possible installed through conda: conda create -n sodev python=3.11 "openblas=*=*openmp*" "numpy<2" pyaml requests pluggy lmfit urllib3 scipy scikit-image h5py numba matplotlib astropy healpy ephem pytz sqlalchemy tqdm cmake
• Activate the conda env and install non-conda stuff conda activate sodev; pip install so3g; cd sotodlib; pip install .

I just tested these commands and the import in your original post works.

Option 2

Use soconda to do this. Use the "binary" config so that it does not try to build any packages. Let me know if you want more details.

[kmharrington]

This is actually on an Argonne cluster so I'm not sure about installing my own conda from scratch. The anaconda environment I'm starting with there is very empty though. I'd need to go talk to support about how to do more complicated installs. FWIW I appear to have fixed my issue by just going down to python 3.10.15

conda create -n sodev310 python=3.10
conda activate sodev310
pip install so3g
cd sotodlib/
pip install .

This works whether or not I install so3g first. So really just seems like current install setup isn't good enough for 3.11?

[tskisner]

Note that running those commands exactly as you have written will install so3g somewhere else, since the sodev310 environment was not activated after you created it.

I believe your current environment probably has a mix of incompatible packages installed. If you don't want to install your own conda (you don't need root to do that), then at least do conda env list and see what environments exist and wipe any old ones. And then go wipe the "user" package locations that it seems to be installing in places like /home/kharrington/.local/bebop/anaconda3/2024.06/lib/python3.11/site-packages.

Then you could try to run the commands I previously posted using their anaconda base environment:

conda create -n sodev python=3.11 "openblas=*=*openmp*" "numpy<2" pyaml requests pluggy lmfit urllib3 scipy scikit-image h5py numba matplotlib astropy healpy ephem pytz sqlalchemy tqdm cmake

conda activate sodev
pip install so3g
cd sotodlib
pip install .

[kmharrington]

oh true, I did activate it, just forgot to copy the command here (editing comment)

your conda throws errors. it's a least weird because healpy and lmfit are installed

(base) [kharrington@beboplogin3 ~]$ conda create -n sodev python=3.11 "openblas=*=*openmp*" "numpy<2" pyaml requests pluggy lmfit urllib3 scipy scikit-image h5py numba matplotlib astropy healpy ephem pytz sqlalchemy tqdm cmake

Channels:
 - defaults
Platform: linux-64
Collecting package metadata (repodata.json): done
Solving environment: failed

PackagesNotFoundError: The following packages are not available from current channels:

  - openblas=[build=*openmp*]
  - lmfit
  - healpy

Current channels:

  - defaults

[tskisner]

OK, my commands were tested with a conda-forge base environment (which has all those packages in the conda-forge channel). Anaconda only has a smaller set of packages in their commercial default channels. If I were you I would abandon the Anaconda that ALCF or whatever cluster has installed. Anaconda basically has no added value except some commercial support contract. They even use conda-forge infrastructure for building the packages in their commercial channels. NERSC has switched to using a conda-forge base environment. You can just install miniforge to someplace in your home directory.

[kmharrington]

Ok, can confirm that Ted's instructions above did work when I actually went and did them :-D