Open aMarcireau opened 1 month ago
@adamreichold I just pushed a new version. Here's a brief overview of breaking changes.
numpy::npyffi::PyArray_Descr
now only contains public fields with the same position in numpy 1 and numpy 2. The remaining fields (elsize
, alignment
, metadata
, subarray
, names
, fields
, and c_metadata
) may be accessed with the functions PyDataType_$field
, where $field
is ELSIZE
, ALIGNMENT
and so on.numpy::PyArrayDescr
and the trait numpy::PyArrayDescrMethods
now have an extra parameter py: Python<'py>
in first position: itemsize
, alignment
, flags
, ndim
, has_object
, is_aligned_struct
, has_subarray
, and has_fields
. u64
to support numpy 2. It was c_char
before . Negative values were technically possible but the numpy 1 to numpy 2 back port suggests to simply cast flag values to unsigned (C) chars, which is what we do here (in the accessor function).
- The following functions of the struct
numpy::PyArrayDescr
and the traitnumpy::PyArrayDescrMethods
now have an extra parameterpy: Python<'py>
in first position:itemsize
,alignment
,flags
,ndim
,has_object
,is_aligned_struct
,has_subarray
, andhas_fields
.
I don't think it is neccessary to change the API here. Since these are Python types we already have the proof that the GIL is held. You can just get the token via self.py()
. For the trait you probably have to remove the default implementation and move them into the impl on Bound
to get access to the token.
@Icxolu Good point! I made the changes that your suggested.
@adamreichold Let me know if you would like me to squash the commits that modified src/dtype, since I ended up rolling back many of the edits.
Sorry for not getting to this yet. We are in crunch mode at $DAYJOB until next week. Will look into as soon as I can.
Thanks @stinodego. I replicated your changes in this PR.
The dtype::tests::test_dtype_names
unit test appears to fail in the CI running NumPy 2. More generally, we might want to have at least one test matrix entry that explicitly builds against NumPy 1 to keep ourselves honest in supporting both major versions.
(I am not what the macOS issue about installing Python is but I would be surprised if it were actually related to this PR. @davidhewitt Is this something that already affected PyO3 and which we could shamelessly copy here?)
(I am not what the macOS issue about installing Python is but I would be surprised if it were actually related to this PR. @davidhewitt Is this something that already affected PyO3 and which we could shamelessly copy here?)
Going to tackle this separately in #436...
@aMarcireau I merged the necessary fixes so that the CI passes on the main branch now. You should rebase and then make sure to relax the various numpy<2
requirements I added when you add one CI job to explicitly test the older NumPy version. (I would suggest doing this on a Linux runner.)
The changes are based on recommendations from https://numpy.org/devdocs/numpy_2_0_migration_guide.html#c-api-changes.
The most visible user-facing change is the addition of two feature flags (
numpy-1
andnumpy-2
). By default, both features are enabled and the code is compiled with support for both ABI versions (with runtime checks to select the right function offsets). Functions that are only available in numpy 1 or 2 are not exposed in this case. Disabling default features (for instancenumpy = {version = "0.21.0", default-features = false, features = ["numpy-1"]}
) exposes version-specific functions and fields but the library will panic if the runtime numpy version does not match.I have not done much testing, this should be tried on different code bases (ideally ones that use low-level field access) before merging.
This currently uses
std::sync::OnceLock
to cache the runtime version. I realised too late that this is not compatible with the Minimum Supported Rust Version (it was introduced in 1.70.0). Usingpyo3::sync::GILOnceCell
isn't straightforward sincepy
is not always available in functions that need to check the version to pick an implementation.