Conflict between python==3.11 and numba~=0.56.0 for installation on osx-arm64

[tibaldo]

I'm trying to install nectarchain on a new laptop with a Mac ARM chip and I get an incompatibility issue. On this platform numba~=0.56.0 is only compatible with python<3.11, and python 3.1 requires numba >=0.60. Is is possible to relax these version constraints or otherwise is there a known solution to this bug?

[tibaldo]

I verified that the requirement numba~=0.56.0 comes from ctapipe 0.19 (that only requires python >=3.8), while the requirement python=3.11 is from nectarchain itself. Unfortunately these requirements do not seem to be compatible on the osx-arm64 platform.

[jlenain]

Hi @tibaldo , Thanks for reporting this. Could you maybe try relaxing the constraint on the python version, like >=3.9 as in https://github.com/cta-observatory/nectarchain/blob/main/pyproject.toml ?

[tibaldo]

In that case I can successfully build nectarchain (with python 3.10.14) but then the tests fail with error messages alike

ERROR src/nectarchain/dqm/bokeh_app/tests/test_app_hooks.py - TypeError: '<' not supported between instances of 'NoneType' and 'str'
ERROR src/nectarchain/makers/calibration/tests/test_core.py - TypeError: '<' not supported between instances of 'NoneType' and 'str'
ERROR src/nectarchain/makers/calibration/tests/test_pedestal_tool.py - TypeError: '<' not supported between instances of 'NoneType' and 'str'
ERROR src/nectarchain/makers/tests/test_charges_makers.py - TypeError: '<' not supported between instances of 'NoneType' and 'str'
ERROR src/nectarchain/makers/tests/test_core.py - TypeError: '<' not supported between instances of 'NoneType' and 'str'
ERROR src/nectarchain/makers/tests/test_waveforms_makers.py - TypeError: '<' not supported between instances of 'NoneType' and 'str'

What was the reason to pin to python=3.11 for nectarchain? I guess some of the code in the core calibration package is written for python 3.11?

[jlenain]

OK, now this is weird because our tests run fine in CI, for several python versions. We test with python 3.9, 3.10 and 3.11 (see https://github.com/cta-observatory/nectarchain/blob/74dbc107f633d6a92660b91e86827b0ec62792a4/.github/workflows/ci.yml#L37), but on amd64 only.

I don't recall if there was any specific reason to pin python=3.11 in the environment specifications. In pyproject.toml, the constraint is more relaxed, with python>=3.9.

[tibaldo]

Okay, weird indeed. I'll dig into this tomorrow.

[tibaldo]

The latter problem seems unrelated to the subject of this issue and shows up also in the CI for another OS. See #128

[tibaldo]

After the transient problem discussed in #128 was solved, I did manage to install nectarchain on my Mac ARM laptop. I had to relax the constraint on python>=3.9 as suggested by @jlenain, and also to add to the environment file the requirement on numba ~=0.56.0 (otherwise a higher python version is picked and the installation of nectarchain fails). I also had to move the zeo and zodb packages to the pip section because these packages do not seem to be installable via conda on osx-arm64. I published the changes on a dedicated branch. This is probably just a messy fix but I cannot figure out how to do it more elegantly. At least it's there for other people encountering the same problem for the time being.

[jlenain]

Hi again, @tibaldo ! With #128 closed, we can go back to this one. So if I am correct, for nectarchain to work on a new Mac with ARM chip, the constraints on the python and numba versions might be too tight in our environment.yml file: python pinned to version 3.11, and numba to 0.56 as per dependency with ctapipe (through https://github.com/cta-observatory/ctapipe/blob/3f2178084855b3fc3d5cf68ad69665c1eebb4de2/environment.yml#L22). Could you please try relaxing the constraint in environment.yml by changing:

  - python=3.11

to:

  - python>=3.9

? Thanks!

[jlenain]

Oups, cross-posts...

[jlenain]

OK, thanks a lot for the check, @tibaldo , and this new branch! Overall, this should not cause any problem. zeo and zodb are only needed to run the bokeh app for the DQM, and should be considered as optional only - unfortunately, conda does not support (yet) optional packages (see https://github.com/conda/conda/issues/12098, opened by our fellow colleague @HealthyPear, btw). However, they should not be platform-dependent (see https://anaconda.org/conda-forge/zeo/files and https://anaconda.org/conda-forge/zodb/files) for the versions we use. Just for my curiosity, could you please post the log from conda when you try to install them via conda-forge? Many thanks again!

[tibaldo]

It looks like a problem with dependency versions.

LibMambaUnsatisfiableError: Encountered problems while solving:

• nothing provides btrees >=4.2.0 needed by zodb-5.4.0-py_1
• nothing provides persistent >=4.1.0 needed by zeo-5.1.1-py_0

Could not solve for environment specs The following packages are incompatible ├─ zeo is not installable because it requires │ └─ persistent >=4.1.0 , which does not exist (perhaps a missing channel); └─ zodb is not installable because it requires └─ btrees >=4.2.0 , which does not exist (perhaps a missing channel).

When I install with pip I get higher versions of both packages (zeo 6.0.0 and zodb 6.0) which do not seem to have dependency issues (I get persistent 6.0 and btrees 6.0)

[jlenain]

I see, thanks... Indeed, persistent and btrees, which are at the core of ZODB, are provided as compiled for specific platforms, osx-arm64 excluded. I think we can work with a pull request based on your proposed branch, @tibaldo.

[tibaldo]

Okay, thank you so much @jlenain ! Please let me know if I can assist with anything, do some tests etc.

[jlenain]

It seems numba 0.56 is only compatible with python<=3.10. With python 3.9, the installation is successful, see https://github.com/cta-observatory/nectarchain/actions/runs/10100292566/job/27933584864. With python 3.10, the installation is also successful, see https://github.com/cta-observatory/nectarchain/actions/runs/10100292566/job/27933585597. With python 3.11, the installation is failing, see https://github.com/cta-observatory/nectarchain/actions/runs/10100292566/job/27933586037.

These three behaviors can also be reproduced locally on an amd64 platform (aka linux-64, running Ubuntu_ 22.04 LTS).

Also, installing a new conda environment with ctapipe 0.19 does not come with version 0.56 of numba but 0.60:

mamba create --name delme ctapipe=0.19
[...]
mamba activate delme
mamba list
[...]
ctapipe                   0.19.3             pyhd8ed1ab_0    conda-forge
numba                     0.60.0          py312h83e6fd3_0    conda-forge
python                    3.12.4          h194c7f8_0_cpython    conda-forge

The same applies if we further pin python to version 3.11:

mamba create --name delme ctapipe=0.19 python=3.11
[...]
mamba activate delme
mamba list
[...]
ctapipe                   0.19.3             pyhd8ed1ab_0    conda-forge
numba                     0.60.0          py311h4bc866e_0    conda-forge
python                    3.11.9          hb806964_0_cpython    conda-forge

so it seems the conda package for version 0.19 of ctapipe is actually not pinning the version for the numba package.

@tibaldo , do you remember at which exact step your installation was failing in the first place ?

[jlenain]

In the meantime, I discovered that GitHub-hosted runners now also include macOS 14 on M1 (https://github.com/actions/runner-images/issues/9254). Is it similar to your new hardware, @tibaldo ?

I thus included a test in CI using this platform (https://github.com/cta-observatory/nectarchain/compare/main...tibaldo:nectarchain:126_installation_problems#diff-b803fcb7f17ed9235f1e5cb1fcd2f5d3b2838429d4368ae4c57ce4436577f03fR60), which runs fine (without pinning numba to any version): https://github.com/cta-observatory/nectarchain/actions/runs/10101732685

[tibaldo]

Hi @jlenain, thanks a lot! The installation runs successfully with the latest version on 126_installation_problems. I cannot reproduce the error with numba anymore. Last week I had tried to create an environment from the command line rather than from the environment.yml file for testing (to get around the issue with zeo and zodb). Perhaps I had done some error with the channel configuration? Anyway, the modifications to the environment.yml file make a seamless installation possible on osx-arm64 now! M1 and M3 are different generations of the same chip architecture, so including M1 in the CI pipeline should guarantee that nothing breaks for users with recent Apple machines. Shall we go ahead with the PR?

[jlenain]

Hi @tibaldo! That's great! Yes, we can definitely go ahead with the PR, and am very glad to have finally found a way to automatically test the installation on osx-arm64 as well!