search

zarr-developers / numcodecs

A Python package providing buffer compression and transformation codecs for use in data storage and communication applications.
http://numcodecs.readthedocs.io
MIT License
121 stars 82 forks source link

NumPy 2.0 support #521

Closed jakirkham closed 2 weeks ago

jakirkham commented 2 months ago

NumPy 2.0 is coming out soon ( https://github.com/numpy/numpy/issues/24300 ). NumPy 2.0.0rc1 packages for conda & wheels came out 3 weeks back ( https://github.com/numpy/numpy/issues/24300#issuecomment-2030603395 )

To prepare for NumPy 2.0, it might be worthwhile to start testing against NumPy 2 in CI

NumPy has put out a migration guide. More details are in the release notes

If Numcodecs make use of NumPy's C API (and produces wheels that use it), having a release of Numcodecs with wheels built against NumPy 2.0.0rc1 would be helpful to ensure NumPy 1 & 2 compatible wheels (as wheels built against NumPy 1 won't be compatible with NumPy 2). More details in this NumPy 2 ABI doc

Also as NumPy is tracking ecosystem support for NumPy 2.0, it would be helpful to share Numcodecs current support status in issue (with any plans): https://github.com/numpy/numpy/issues/26191

jakirkham commented 2 months ago

As Numcodecs has relied primarily on the Python Buffer Protocol and not really used NumPy's C API, this shouldn't involve rebuilding Numcodecs. Have just searched through the code to reconfirm this

The thing to check for would be if Numcodecs is affected by any NumPy Python API breaks. This could be done by adding it to CI here

jakirkham commented 2 months ago

FWIW attempted running the test suite locally with NumPy 2.0.0rc1 from conda-forge and it appeared to pass. So doesn't look like any changes are needed

Still would be good to add to CI

jakirkham commented 2 months ago

We do have an optional dependency on zfpy

https://github.com/zarr-developers/numcodecs/blob/4abe4be2d32a1e5743982f3056aac937f9664da6/pyproject.toml#L66-L68

zfpy still needs to do some work to migrate: https://github.com/LLNL/zfp/issues/210

However this just means users won't be able to install zfpy until that is resolved. No impact on installing Numcodecs itself

dstansby commented 2 weeks ago

It might be worth doing a release with the numpy<2 pin first (which is now in main) while this gets sorted out?

joshmoore commented 2 weeks ago

:+1: Certainly what is happening in several other of my repos.

Edit: see https://github.com/zarr-developers/numcodecs/pull/528/files#diff-50c86b7ed8ac2cf95bd48334961bf0530cdc77b5a56f852c5c61b89d735fd711

rgommers commented 2 weeks ago

Didn't @jakirkham confirm above already that everything works as expected for numcodecs? If so, no need to add a pin now. Based on that testing, we marked numcodecs as compatible in https://github.com/numpy/numpy/issues/26191 a long time ago.

The linked PR (gh-535) only shows test failures because of zfpy, which @jakirkham had also commented on already.

dstansby commented 2 weeks ago

👍 without zfpy looks like it tests work fine with numpy 2.0.

jakirkham commented 1 week ago

Yep all we needed was CI. Thanks for adding that! 🙏

NumPy 2 support in zfpy is a separate issue. Presumably this would be solved with upstream PR ( https://github.com/LLNL/zfp/pull/231 ) and a new tag. In any event it is not blocking for numcodecs