search

scipy / oldest-supported-numpy

Meta-package providing the oldest supported Numpy for a given Python version and platform
BSD 2-Clause "Simplified" License
55 stars 33 forks source link

Fails on alpine due to musl wheel being in latest version #75

Closed daniel-brenot closed 2 months ago

daniel-brenot commented 1 year ago

The latest version of numpy(1.25.0) added musl wheels that support use in alpine. I'm trying to use this in an alpine docker container, but struggling to due to a dependency relying on this library.

I believe this should return the latest version of numpy when the target is musl.

Additionally, if anyone could offer a way for me to install the version of numpy and use it as a dependency when downloading scikit I would appreciate it. I'm currently running: pip wheel --no-cache-dir --wheel-dir /usr/src/app/wheels -r requirements.txt -c constraints.txt

with the following in my constraints.txt file: numpy>=1.25.0

and yet still numpy 1.23 is being installed, which is incompatible with my alpine image.

h-vetinari commented 1 year ago

This already came up in #69, but the problem is that the metadata does not allow us to distinguish musl

NumPy 1.25.0 also supports musl, @rgommers how do you want to handle that?

Environment markers don't allow selecting on libc version AFAIK (see PEP 508), so this can't be dealt with. Luckily it shouldn't be needed anymore either once we update things for the C API exporting changes in 1.25.0 (which should be done separately from this update).

Might be worth starting a thread on https://discuss.python.org/ how this can be improved.

daniel-brenot commented 1 year ago

Is there some other way then that downstream users of this can affect the numpy version? This is used in scikit-learn and aside from forking their library and changing the dependency to the newest numpy AND forking the library that uses it, i'm at a loss for how to use the latest numpy

rgommers commented 1 year ago

You can disable build isolation to change versions. E.g.:

pip install numpy==1.25.0
pip install scikit-learn --no-build-isolation  # also need to make sure all other dependencies are installed in the build env

It's quite fiddly to get that right unfortunately once you get higher up in the stack and have lots of dependencies, because there's no great way to install only the dependencies, nor to override build constraints. There are very long threads on both topics on the Pip issue tracker, but neither seems to be moving forward.

The real solution I guess is for scikit-learn to upload musllinux wheels, so that there's no need to build from source. pyproject.toml metadata is inadequate for building from source in this kind of circumstance I'm afraid.

However, the next thing we'll do here is to remove the pins from oldest-supported-numpy, because of the change in 1.25.0 to how the C API is exported: https://numpy.org/devdocs/dev/depending_on_numpy.html#build-time-dependency. That will in practice solve the problem for musllinux here.

I may try to find time soon to do this. I suspect that it's worth testing the waters here with a new oldest-supported-numpy release, and if it doesn't work as designed then to yank that release.

daniel-brenot commented 1 year ago

Great to hear this is getting addressed and i'm excited for this to get merged so we can go back to alpine.

As for your suggestion, it doesn't seem to work when i run the pip wheel command, which is how I do dependencies in our case. We use pip wheel in a separate stage in docker so that build dependencies don't make it to the final build.

If you have any recommendations for how to do this when using the wheel command i'm happy with that. If not i'll just wait until this issue is resolved.

Thank you!

rgommers commented 1 year ago

To build a wheel, prefer build over pip wheel. So here, use python -m build -nxw (-w for wheel, and -n -x for turning off build isolation).