Patch existing python wheels that depend on numpy to use numpy <2.0

[mboisson]

This is because of the breaking ABI with numpy 2.0. https://numpy.org/doc/stable/dev/depending_on_numpy.html#numpy-2-0-specific-advice

Using https://github.com/ComputeCanada/wheels_builder/pull/113

List of such wheels being extracted right now using for w in $(find /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/ -name '*.whl'); do echo $w; ./manipulate_wheels.py --print_req --wheel $w | grep numpy; done |& tee wheel_numpy_req.txt

[mboisson]

This is the list of wheels which we have and which contain numpy.* as requirement:

wheel_numpy_req_filtered.txt

[ccoulombe]

It should only be the wheels that binds on numpy. For example, a pure generic wheel that requires numpy and do not contains any extensions should probably not restrict the numpy version.

Plus, it can only be 2023 and python 3.10+ wheels since the numpy 2.0 wheels will be under 2023 only.

If we really want to constrain all wheels we have, than a rule in a constraints file is easy to manage.

[mboisson]

It should only be the wheels that binds on numpy. For example, a pure generic wheel that requires numpy and do not contains any extensions should probably not restrict the numpy version.

Are you sure of that ?

Plus, it can only be 2023 and python 3.10+ wheels since the numpy 2.0 wheels will be under 2023 only.

If we really want to constrain all wheels we have, than a rule in a constraints file is easy to manage.

This is not a forever situation. A rule in a constraints-file does not work. We will have to build numpy 2.0 soon and start building wheels with it. This is to ensure that older wheels keep working.

[ccoulombe]

Yes : https://numpy.org/devdocs/dev/depending_on_numpy.html#numpy-2-abi-handling

NumPy 2.0 is an ABI-breaking release, however it does contain support for building wheels that work on both 2.0 and >1.xx releases. It’s important to understand that: When you build wheels for your package using a NumPy 1.xx version at build time, those will not work with NumPy 2.0. When you build wheels for your package using a NumPy 2.x version at build time, those will work with NumPy 1.xx.

If you rely on the NumPy C API (e.g. via direct use in C/C++, or via Cython code that uses NumPy), add a numpy<2.0 requirement

numpy 2.0 will only be under 2023, thus python 3.10 and higher, hence we can focus only on 2023.

Generic wheels that do not contains extensions that binds on Numpy can be ignored since they only rely on the API not the ABI. Some generic wheels do contains extensions, hence we need to look inside of each wheels for objects thats binds on numpy, not only at their requirements.

[mboisson]

We do not have a distinction between 2020 and 2023 for ~most~ many wheels (under generic), so we can't necessarily split by that.

We can however ignore 1) All cp27, cp36, cp37, cp38, cp39 wheels ~2) All -py3-none-any.whl~ 3) All wheels under gentoo2020

?

[ccoulombe]

Most wheels under generic do not bind on numpy abi, ie are pure python, thus most can be ignored.

It comes down at looking inside the wheels if there's objects that binds to the numpy abi, for each wheels.

Some [py2|py3|py2.py3]-none-any.whl may have objects/extensions, for instance bitsandbytes-0.42.0+computecanada-py3-none-any.whl that contains extensions that binds to cuda and others. The example of bitsandbytes is interesting, since it requires numpy for the API but not the ABI (it does not bind to it).

torch
numpy
scipy

[mboisson]

Ah, so we can't ignore all -none-any.whl and we need to test them for extensions 😢

[ccoulombe]

En effet! I can point the ones (generic ones) I know contains objects, but I might miss a few, so we really need to look inside.

[ccoulombe]

To look for symbols of numpy in objects, something similar would do :

unzip -d RIPSER /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo2023/x86-64-v3/ripser-0.6.10+computecanada-cp312-cp312-linux_x86_64.whl
cd RIPSER/
for so in $(find . -name '*.so'); do nm $so | fgrep -i -e npy -e numpy; done

it finds

0000000000023d40 r _ZL39__pyx_k_numpy_core_multiarray_failed_to
0000000000023d00 r _ZL40__pyx_k_numpy_core_umath_failed_to_impor

I would create a list of wheels that have numpy has requirement, filter it, then

for wheel in listoffilteredwheels ; do
    tmpdir=$(mktemp -du)
    echo "### $wheel"

    unzip -qn -d $tmpdir $wheel
    for so in $(find $tmpdir -name '*.so'); do nm $so | fgrep -i -e npy -e numpy; done

    echo "###"
    echo ""

    rm -fr $tmpdir
done

[ccoulombe]

The list of wheels is now filtered down to : 731 wheels that bind to Numpy ABI.

The list is contains wheels from :

• gentoo/generic/
• generic
• gentoo2023/

filtered to keep wheels for : cp??-abi3, cp31[012]. eg : wheels that are compatible with 3.10, 3.11 and 3.12

[ccoulombe]

This is done, and the wheels are being synced to the systems at the moment.