Closed qiyunzhu closed 3 months ago
@wasade It will be very nice if this can also be resolved prior to the new release. You are probably the most familiar with the issue. But if necessary, I can motivate a student to help you on it.
@wasade I tried to pip install biom-format 2.1.16 along with numpy 2.0rc1 but it ran into issues. Procedures to reproduce:
> conda create -y -n biom-np2 -c conda-forge python
> conda activate biom-np2
> pip install numpy==2.0.0rc1
...
Successfully installed numpy-2.0.0rc1
> echo "import numpy; print(numpy.__version__)" | python
2.0.0rc1
> pip install biom-format
...
Successfully installed biom-format-2.1.16 click-8.1.7 h5py-3.11.0 pandas-2.2.2 python-dateutil-2.9.0.post0 pytz-2024.1 scipy-1.13.0 six-1.16.0 tzdata-2024.1
> echo "import biom; print(biom.__version__)" | python
...
File "biom/_filter.pyx", line 1, in init biom._filter
ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject
After some testing, I found that the following method works:
> conda create -y -n biom-np2 -c conda-forge python
> conda activate biom-np2
> pip cache purge
> pip install numpy==2.0.0rc1 cython
> pip install biom-format --no-build-isolation
> echo "import biom; print(biom.__version__)" | python
2.1.16
Some notes:
--no-build-isolation
. Otherwise biom-format will be built using the NumPy 1.x logic.ModuleNotFoundError: No module named 'Cython'
.pip cache purge
helps to ensure reproducibility.Comment:
--no-build-isolation
may not be favorable. This behavior is not seen in other dependencies of biom-format. Perhaps this can be worked on within the biom-format packaging mechanism?The use of --no-build-isolation
is consistent though with the guidance from @rgommers?
I would welcome a PR to resolve the Cython installation step
It's because of the numpy
requirement in https://github.com/biocore/biom-format/blob/master/pyproject.toml#L2. The advised requirement here is numpy>=2.0.0rc1,<3
, see: https://github.com/numpy/numpy/issues/24300#issuecomment-2030603395. That will make pip install biom-format
work as expected. It's then still possible to buid against numpy 1.x through --no-build-isolation
, but the default is then such that you'll produce wheels that work with 1.x and 2.x.
Ah okay, thank you!
I guess once NumPy 2.0 is officially released, the current package (biom-format) will automatically build against 2.0 when it is being installed, so there is nothing we need to do to this package. Is my understanding correct? @rgommers
I guess once NumPy 2.0 is officially released, the current package (biom-format) will automatically build against 2.0 when it is being installed
Indeed, it will.
so there is nothing we need to do to this package.
If you've worked around the problem in scikit-bio and this package isn't used much outside of that, you could leave it as is indeed. There is an annoying failure modes though, which is that any user who is building a release before numpy 2.0 is out will get a wheel built that doesn't work with 2.0. And those are cached by pip
, so they may see issues at runtime since pip
will pick a wheel from its cache rather than rebuild. If you want to avoid that, adding numpy>=2.0.0rc1,<3
is still better.
@rgommers Thanks for the clarification!
Is my understanding correct that if biom-format or scikit-bio is built against numpy>=2.0.0rc1, it will be compatible with both numpy 1.x and numpy 2.x. Therefore, even in an environment with numpy 1.x pre-installed, one can install scikit-bio via a simple pip install scikt-bio
, and that doesn't trigger a mandatory upgrade of numpy 1.x into 2.x?
Yes indeed, that is correct.
FWIW it looks like biom-format
has been migrated for NumPy 2: https://github.com/conda-forge/biom-format-feedstock/pull/30
scikit-bio
depends on hdmedians
, which is pending migration ( https://github.com/conda-forge/hdmedians-feedstock/pull/20 ). Once that is in there should be a new bot PR for scikit-bio
Thanks!!
@wasade @jakirkham We recently removed hdmedians from scikit-bio's dependencies. It won't be an issue in the next release.
Hi @wasade I was able to set up a development environment with NumPy 2.0, following the instruction provided in #955 . Now as I run
python -m unittest
, there are two errors. I think you may be more familiar with these. Would you like to address them?