search

Cantera / cantera

Chemical kinetics, thermodynamics, and transport tool suite
https://cantera.org
Other
581 stars 342 forks source link

Upcoming breaking changes in Numpy 2.0 #1675

Open corykinney opened 3 months ago

corykinney commented 3 months ago

NumPy 2.0 hasn't been released yet, but the initial release notes from the pre-release mentions some breaking changes that they address in their migration guide. I don't know how the Cantera team wants to handle compatibility going forward, but I thought I would go ahead and make a thread in case it's helpful.

Previous Cantera builds

I installed NumPy 2.0.0b1 and attempted to import Cantera 3.0.0 which yields the following error:

...
    from ._cantera import *
  File "cantera/_cantera.pyx", line 33, in init cantera._cantera
  File "cantera/_utils.pyx", line 1, in init cantera._utils
ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject

Clearly previous builds of Cantera can only be used with NumPy 1.x and will not work with 2.x releases. I don't know if there is a way to enforce numpy<2.0.0 as a package requirement on PyPI, or if that can't be done retroactively, but it might need to be added to some documentation somewhere.

Future Cantera builds

I attempted to build Cantera from source from the latest commit using NumPy 2.0.0b1 to see what changes would need to be made. The first build error that I get from the breaking changes is below:

build\python\cantera\_onedim.cpp(8544): error C2039: 'subarray': is not a member of '_PyArray_Descr'
...
build\python\cantera\_onedim.cpp(8545): error C2039: 'subarray': is not a member of '_PyArray_Descr'

The changes to the PyArray_Descr struct are covered in this section of the migration guide. It seems the PyDataType_SUBARRAY accessor function needs to be used. They also provide a npy2_compat.h file that can be included in the code base that will allow compilation on NumPy 1.x and not affect 2.x builds.

I haven't gotten any further yet, but I will edit this if I do.

speth commented 3 months ago

Clearly previous builds of Cantera can only be used with NumPy 1.x and will not work with 2.x releases. I don't know if there is a way to enforce numpy<2.0.0 as a package requirement on PyPI, or if that can't be done retroactively, but it might need to be added to some documentation somewhere.

We would need to rebuild the PyPI packages to add that version restriction. How many things are broken in that workflow is an interesting question.

It seems the PyDataType_SUBARRAY accessor function needs to be used. They also provide a npy2_compat.h file that can be included in the code base that will allow compilation on NumPy 1.x and not affect 2.x builds.

Those errors are coming from Cython-generated code, so I think we just need to wait for Cython to be updated on this one. What version of Cython are you using?

corykinney commented 3 months ago

Those errors are coming from Cython-generated code, so I think we just need to wait for Cython to be updated on this one. What version of Cython are you using?

Sorry, I'm still getting used to Cython, so I did not realize that the error was coming from that instead of the onedim Cantera file, which explains why I couldn't find it. I am using Cython 3.0.9, which I thought should work based on this from the migration guide:

Where not possible, new accessor functions are required: ...

  • PyDataType_FIELDS, PyDataType_NAMES, PyDataType_SUBARRAY

... Cython code should use Cython 3, in which case the change is transparent.

Perhaps I should open an issue on Cython instead?

ischoegl commented 3 months ago

To add my 2 cents, I don't think that we need to jump on implementing changes based on a beta release yet. At the same time, we have an established workflow for prerelease dependencies in our post-merge-tests.yml CI tests. While it is released now, Cython 3.0 was in pre-release until recently, and we had a test ensuring that things would compile (see prerelease-cython runner); while this used to be part of the main CI job, we moved it to a post-merge test as it was considered non-critical.

I would suggest to simply modify the (now no longer needed) CI runner for pre-release Cython to test for NumPy 2.x instead, so we can monitor whether things clear up (breaking NumPy API changes may or may not be breaking for Cantera due to the Cython layer, which at this point should default to 3.0.x). This modification is presumably quite simple - a PR is welcome and a good intro to learn about our GH actions tests.

Beyond, NumPy v2.0.0b1 was just released 2 days ago. While I found this statement "The tentative release date for the first release candidate of NumPy 2.0 is around 1 Mar 2024, and the final release 6-8 weeks later," it is not necessarily easy to predict how fast downstream packages are expected/able to move. I consider Cantera as downstream relative to Cython and other packages that themselves depend on NumPy, so we may have to pin to numpy<2.0 until things stabilize upstream. As an aside, I personally do not believe that we need to make the upcoming Cantera 3.1 release contingent on support for NumPy 2.0.

bryanwweber commented 3 months ago

My 2 cents as well... :smile: If this dependency cap is added to 3.0 (by creating .post1, most likely) this may also be a good time to add Python 3.12 support to the builds, and GitHub Actions supports native ARM Mac runners now, which should simplify cross-compiling. See https://github.com/Cantera/enhancements/issues/198 I suppose it depends on the timeline for 3.1.

ischoegl commented 3 months ago

My 2 cents as well... 😄 If this dependency cap is added to 3.0 (by creating .post1, most likely) this may also be a good time to add Python 3.12 support to the builds, and GitHub Actions supports native ARM Mac runners now, which should simplify cross-compiling. See Cantera/enhancements#198 I suppose it depends on the timeline for 3.1.

Great to hear that you're volunteering for .post1! 😜

Jests aside the fact that we pretty much decided to remove the legacy MATLAB interface (#1670) may mean that 3.0 may be around for a while for some users. At the same time I am very hesitant to spend time on back ports.

bryanwweber commented 3 months ago

Great to hear that you're volunteering for .post1

Happy to help as much as I can with this. I know the packaging stuff is still pretty arcane. ❤️

bryanwweber commented 1 month ago

As a point of reference, conda-forge attempted an automatic upgrade to NumPy 2.0 and as expected it failed. https://github.com/conda-forge/cantera-feedstock/pull/38

bryanwweber commented 1 week ago

With https://github.com/conda-forge/cantera-feedstock/pull/38 and https://github.com/Cantera/cantera/pull/1719 merged, there should be packages published for 3.0.1 on PyPI that back port changes to support NumPy 2.0 and Python 3.12 and 3.13. The last step is to update the Cantera channel conda recipes to build 3.0.1 and pin NumPy.