Drop support for Python 3.9

bdice commented 1 month ago

This issue tracks dropping Python 3.9, following from the discussion on https://github.com/rapidsai/build-planning/issues/40:

When should we drop Python 3.9?

Typically RAPIDS has kept the matrix of supported Python minor versions to 2 or 3 versions at a time. When adding Python 3.12, we should probably drop Python 3.9 as well.

SPEC 0 recommended dropping support for Python 3.9 in 2023Q4. Meanwhile, NEP 29 recommended dropping support for Python 3.9 as of Apr 05, 2024. Both of these deadlines have passed and several large Python libraries are now moving towards dropping Python 3.9, so it is probably reasonable to drop Python 3.9 around the same time that we add Python 3.12.

We are beyond the SPEC 0 date for dropping Python 3.9 now, and expect that RAPIDS dependencies will gradually drop support. We are exploring dropping Python 3.9 in RAPIDS 24.10.

Tasks:

[x] Announce deprecation: https://github.com/rapidsai/docs/pull/528
[x] Drop Python 3.9 from shared-workflows: https://github.com/rapidsai/shared-workflows/pull/235
[x] For each repo, update Python requirements
- [x] Update matrix filters referencing Python 3.9
- [x] Edit dependencies.yaml to remove py: 3.9 sections like this
- [x] Edit dependencies.yaml to raise the fallback matrix's Python lower bound
- [x] Update CMake references like find_package(Python 3.9 ...)
- [x] Update pyproject.toml: requires-python and specifiers
- [x] Check for special cases, like ucxx's Python build matrix in conda_build_config.yaml
- [x] Update README, CONTRIBUTING, and docs as needed
- [x] grep for 3.9 to make sure nothing was missed!

Things to be careful about:

Python version embedded in Jupyter notebook metadata (like this)
avoid changing benchmarking results and the code to reproduce them
pynvjitlink is used by more than just RAPIDS (like https://github.com/NVIDIA/nvmath-python), and uses its own CI matrix to support a broader range of versions than RAPIDS (link)... we may sometimes drop a Python version across RAPIDS but still support it in pynvjitlink
generic tools like rapids-reviser, rapids-build-backend, rapids-metadata, etc. shouldn't have their floors bumped unless there's a compelling reason to do so... that'd make it harder to fix bugs in those tools that affect older releases

### before making changes in individual repos
- [x] Announce deprecation: https://github.com/rapidsai/docs/pull/528
- [x] Drop Python 3.9 from shared-workflows: https://github.com/rapidsai/shared-workflows/pull/235

### Updates for libraries
- [x] cucim (https://github.com/rapidsai/cucim/pull/766)
- [x] cudf (https://github.com/rapidsai/cudf/pull/16637)
- [x] cugraph (https://github.com/rapidsai/cugraph/pull/4625)
- [x] cugraph-gnn (https://github.com/rapidsai/cugraph-gnn/pull/32)
- [x] cugraph-pg (https://github.com/rapidsai/cugraph-pg/pull/25 ... this repo is not being actively developed right now, not waiting for that to be mered to consider this done)
- [x] cuml (https://github.com/rapidsai/cuml/pull/6040)
- [x] cuspatial (https://github.com/rapidsai/cuspatial/pull/1443)
- [x] cuvs (https://github.com/rapidsai/cuvs/pull/301)
- [x] cuxfilter (https://github.com/rapidsai/cuxfilter/pull/623)
- [x] dask-cuda (https://github.com/rapidsai/dask-cuda/pull/1377)
- [x] kvikio (https://github.com/rapidsai/kvikio/pull/443)
- [x] pynvjitlink (https://github.com/rapidsai/pynvjitlink/pull/102)
- [x] rapids-cmake (https://github.com/rapidsai/rapids-cmake/pull/684)
- [x] raft (https://github.com/rapidsai/raft/pull/2417)
- [x] rmm (https://github.com/rapidsai/rmm/pull/1659)
- [x] ucx-py (https://github.com/rapidsai/ucx-py/pull/1065)
- [x] ucxx (https://github.com/rapidsai/ucxx/pull/272)
- [x] wholegraph (https://github.com/rapidsai/wholegraph/pull/209)
- [x] private repos

### after all libraries are updated
- [x] `pypi-wheel-scripts` (https://github.com/rapidsai/pypi-wheel-scripts/pull/16)
- [x] `ci-imgs` (https://github.com/rapidsai/ci-imgs/pull/173)
- [x] dask-build-environment (https://github.com/rapidsai/dask-build-environment/pull/97)
- [x] update install docs? (https://github.com/rapidsai/docs/blob/01dbbcb67c199893b839651f5dca01356c825068/install/install.md?plain=1#L101) (https://github.com/rapidsai/docs/pull/534)
- [x] Update release selector: need to disable Python 3.9 on nightlies https://github.com/rapidsai/docs/blob/01dbbcb67c199893b839651f5dca01356c825068/_includes/selector.html (https://github.com/rapidsai/docs/pull/534)

jakirkham commented 1 month ago

Some quick data points:

NumPy 2.1.0 drops Python 3.9 (note 2.0.0 has 3.9 support)
- https://github.com/numpy/numpy/issues/26242
Pandas next release drops Python 3.9 (cites SPEC 0)
- https://github.com/pandas-dev/pandas/pull/58238
CuPy 14 plans to drop Python 3.9 (cites SPEC 0)
- https://github.com/cupy/cupy/issues/8269
Numba still supports Python 3.9
scikit-learn still supports Python 3.9
scikit-image still supports Python 3.9
NetworkX 3.3 dropped Python 3.9 (came out Oct 2023)
- https://github.com/networkx/networkx/pull/7028
Bokeh 3.5 dropped Python 3.9
- https://github.com/bokeh/bokeh/pull/13634
Dask is discussing dropping Python 3.9
- https://github.com/dask/community/issues/373
- XGBoost next release drops Python 3.9
- https://github.com/dmlc/xgboost/pull/10434

jameslamb commented 1 month ago

I'm +1 on doing this in this timeframe.

DLFW builds are using Python 3.10 (DLFW support matrix), as that's the stable python3 that ships with Ubuntu 22.04 (Python version across Linux distros), so this change shouldn't be disruptive for those builds.

And nemo-curator is on python >= 3.10 (code link), so they shouldn't be affected either.

rjzamora commented 1 month ago

We are exploring dropping Python 3.9 in RAPIDS 24.10.

Just curious to know what the ETA is for dropping Python 3.9?

I believe we will need to remain pinned to dask 2024.7.1 until then. In the meantime, we cannot catch breaking changes in dask/distributed.

jameslamb commented 4 weeks ago

The plan is to do this "within 24.10", but we can't be more precise than that.

I believe we will need to remain pinned to dask 2024.7.1 until then. In the meantime, we cannot catch breaking changes in dask/distributed.

For Python 3.9, yes, but I think rapids-dask-dependency could take on a dependency on newer dask / distributed for Python 3.10+

Like this:

# TODO: drop the python_version specifiers and just use latest versions built from GitHub once
#              RAPIDS drops Python 3.9 support
dependencies = [
    "dask @ git+https://github.com/dask/dask.git@main ; python_version >= '3.10'",
    "distributed @ git+https://github.com/dask/distributed.git@main ; python_version >= '3.10'",
    "dask-expr @ git+https://github.com/dask/dask-expr.git@main ; python_version >= '3.10'",
    "dask==2024.7.1 ; python_version < '3.10'",
    "distributed==2024.7.1 ; python_version < '3.10'",
    "dask-expr ; python_version < '3.10'",
]

That'd allow projects to start testing against the latest development version of Dask throughout the 24.10 release cycle.

rjzamora commented 4 weeks ago

That'd allow projects to start testing against the latest development version of Dask throughout the 24.10 release cycle.

Okay, thanks for the response @jameslamb! This may be our best option -- I'll look into it.

jakirkham commented 4 weeks ago

For rapids-dask-dependency, likely Conda will work ok as-is since the solver will pick packages that are Python 3.9 (or 3.10+) compatible. Though please let us know if not

bdice commented 4 weeks ago

Dropping 3.9 builds should now be safe, and would also unblock the work of updating our dask pinning.

This PR does that: https://github.com/rapidsai/shared-workflows/pull/235

This change in shared-workflows is also safe to merge before we do the cleanup in each repo to remove dependencies, configs, etc. referencing Python 3.9. We're just building/testing fewer packages.

jakirkham commented 4 weeks ago

Closing the loop on Dask, we dropped Python 3.9 from CI, so no special casing was needed (or used) there

Just a normal unpinning of Dask as part of RAPIDS post-release cleanup

jakirkham commented 2 weeks ago

Thanks James! 🙏

Have shepherded a good chunk of these through

Of the remainder:

RAFT - Had a (flaky?) test failure. Trying rerunning to see if it clears. Fixed on restart. Now merged!
cuGraph-GNN & Wholegraph - Have active discussions (though seems like they are close to resolution)
Draft PRs - A few are marked as draft. Maybe we can take them out of draft?
Others - Did not get to some or saw others were already looking

Please let me know if there are others you need help following up on

jameslamb commented 2 weeks ago

Thanks John!

A few are marked as draft. Maybe we can take them out of draft?

Yep, my approach has been to open these PRs in draft first, to not bother people with notifications for review until I see CI (or at least, most CI) passing.

Happy to see all of those that were still in draft as of yesterday are looking ok, so I've opened them all up for review.

sisodia1701 commented 2 weeks ago

Discussed in today's build infra sync, this is almost done. Once complete, the team will start working on Add support for Python 3.12 https://github.com/rapidsai/build-planning/issues/40

jameslamb commented 2 weeks ago

Did some more GitHub searches, found a few more things that were missed.

All that's left here is those PRs and dask-build-environment: https://github.com/rapidsai/dask-build-environment/pull/97.

This is close enough to done that we can move forward with Python 3.12 (#40)

jameslamb commented 1 week ago

Did another set of GitHub searches, and I think this work is now complete! Thanks for the help everyone 😁

jakirkham commented 1 week ago

Thanks James! 🙏

rapidsai / build-planning

Drop support for Python 3.9 #88

When should we drop Python 3.9?