search

conda-forge / conda-forge.github.io

The conda-forge website.
https://conda-forge.org
BSD 3-Clause "New" or "Revised" License
123 stars 272 forks source link

NumPy 2 bringup #1997

Open jakirkham opened 1 year ago

jakirkham commented 1 year ago

NumPy is currently working on a NumPy 2.0 release, which is planned to come out later this year. Here are the current (draft) release notes. Also here's the upstream tracking issue ( https://github.com/numpy/numpy/issues/24300 ), and ecosystem compatibility tracker.

Some questions worth discussing:

Todos:

cc @conda-forge/core

hmaarrfk commented 3 months ago

Just so it doesn't get lost in the notifications, something seems to be off with the python 3.12 pinning. I know it is confusing to do dual migrations at the same time so I'm not sure there is an easy answer https://github.com/conda-forge/conda-forge-pinning-feedstock/pull/5790#discussion_r1590007898

h-vetinari commented 3 months ago

Yeah, thanks for the note @hmaarrfk & @xhochy. In previous numpy bumps, we always did them for the open python 3.x & pypy migrations as well, and this wasn't an issue. Here however, bumping the 3.12 migrator gets immediately picked up by a given feedstock rerendering and so runs into breakage if that package isn't yet numpy 2.0-compatible (basically all of them).

I undid the changes to the 312 migrator to unbreak this, but I'm kind of unsure how to square this circle now. I think for now the least bad option is to just accept that we don't get numpy 2.0 builds for 3.12. Alternatives might be:

  1. trying to override the whole numpy/python/python_impl zip in the numpy2 migrator to include 3.12 directly.
    • I tested this and it seems to work
    • Downside there is that this will open PRs to all python-related feedstocks, because the migrator bot sees python: as being updated.
  2. Close Python 3.12 migration (which would be waaaaay earlier than we usually do this)

With 1., the numpy2 migrator would look like

numpy:
  - 1.22  # no py38 support for numpy 2.0
  - 2.0
  - 2.0
  - 2.0
  - 2.0
python:
  - 3.8.* *_cpython
  - 3.9.* *_cpython
  - 3.10.* *_cpython
  - 3.11.* *_cpython
  - 3.12.* *_cpython
python_impl:
  - cpython
  - cpython
  - cpython
  - cpython
  - cpython

while the python312 one would stay untouched.

h-vetinari commented 3 months ago

Next snag: bot isn't opening PRs 😑

jakirkham commented 3 months ago

We are 87% of the easy through the Python 3.12 migration. Maybe we are close to closing this out? Perhaps with a little effort?

Wonder if there are other configuration options available to us if we make the NumPy 2 migration depend on the Python 3.12 migration? IOW the "no ice cream until you eat your vegetables" approach 😉

h-vetinari commented 3 months ago

Wonder if there are other configuration options available to us if we make the NumPy 2 migration depend on the Python 3.12 migration?

Yeah, there's wait_for_migrators, but this is not exactly well tested and it would still run into the issue that we'd have to override the whole zip, which would then open PRs for all non-noarch python feedstocks.

h-vetinari commented 3 months ago

We are 87% of the easy through the Python 3.12 migration.

The migrator status page is somewhat misleading on this - it reports the percentage of "PRs opened", not "PRs closed". So we're actually at ~67% with the 3.12 migration, with about ~900 feedstocks still to go. That's IMO quite a bit too early to close?

I guess the only nice alternative to points 1. & 2. above would be to implement something in cf-scripts like skip_unless_present:, where we could say that a feedstock should only be migrated if it depends on numpy, in order to force it to ignore the feedstocks that wrongly get picked up due to the presence of the python: key (that we need to override the zip) if we do point 1.

I guess there aren't really any other zips that are affected in this way, so it'd be a bit tailor-made for the numpy/python situation. From that POV, maybe we could also just deal with the hassle of closing the superfluous PRs this one time, because once we're on numpy 2.0 (and have dropped 3.8 around October), we can completely remove numpy from the python zip[^1] going forward.

[^1]: except PyPy, but that's a separate discussion for various reasons.

h-vetinari commented 3 months ago

FWIW, I think we need to migrate 3.12 as well, as we won't simply be able to switch this one later (due to lots of missing dependencies then).

So if we don't do it as part of the overall numpy 2.0 migration, we'd need a separate migrator (now or later) that catches up 3.12 specifically. I feel that a second such migration wastes more time than it would do to deal with the overzealous PRs to python-but-not-numpy-related feedstocks (which by themselves aren't harmful even if merged).

h-vetinari commented 3 months ago

I've paused the numpy2 migration for now until we decide how we want to handle this (as all PRs that are merged now would have to be cleaned up again later).

I've also opened a PR proposing to include python 3.12 in the current migrator.

jakirkham commented 3 months ago

After some discussion, we decided to soft-close the Python 3.12 migration. Meaning that the Python 3.12 migration is still running, but we now also include Python 3.12 in the global pinning. As a result, re-rendering a feedstock could add Python 3.12 (even if its dependencies are not available on Python 3.12 yet). Meaning the feedstock would get Python 3.12 builds that would fail. These could simply be ignored for now. Based on the Python 3.12 migration status, this is only relevant for ~11% of feedstocks (admittedly that is ~500 feedstocks). The rest have either already been migrated or have a migration PR

With Python 3.12 now addressed, we restarted the NumPy 2 migration today ( https://github.com/conda-forge/conda-forge-pinning-feedstock/pull/5851 ). Here is the status of NumPy 2 migration. It may take it a bit to start adding PRs to all the feedstocks that can now take a PR. So please be patient 🙂

Thanks all! 🙏

bashtage commented 3 months ago

statsmodels still seeling some NumPy 1.22 builds on my NumPy 2 migration. Have these not been migrated yet? Or will they never be migrated (and so need to be skipped (and if so, any pointers would be helpful)).

image

These all fail. All of the NumPy 2 targeted build pass (as do a few NumPy 1.22, although most of these should be disabled when the recipe is rerendered).

statsmodels NumPY 2 PR

h-vetinari commented 3 months ago

These all fail.

Those are all the PyPy builds, which is not participating in the numpy 2.0 migration (IOW, they should run just as before, but have broken for unrelated reasons). I haven't opened an issue yet, but PyPy support is (unfortunately!) being wound down, so don't hesitate to skip those. I did just that in the statsmodels PR even before seeing your comment here... :)

jakirkham commented 3 months ago

JFYI NumPy is planning to release 2.0.0 on June 16th

xref: https://github.com/numpy/numpy/issues/24300#issuecomment-2125643448

jakirkham commented 2 months ago

More updates: