Update RAPIDS Python packages to use rapids-build-backend

vyasr commented 5 months ago

rapids-build-backend is a wrapper around standard backends like scikit-build-core and setuptools that handles some of the standard issues that we face for RAPIDS packages (CUDA versioning, alpha versions, etc). Substituting it into existing RAPIDS Python packages should be fairly painless, but will require some careful testing to verify that nothing is broken.

The backend need not be updated in lockstep across all of RAPIDS for this to work, and merging PRs in any order should be generally safe. However, since there are cases like unified devcontainers where underlying build commands may need an update, it would be best to try and test at least a couple of core packages together to verify that everything works as expected.

### Changes to `rapids-build-backend` and `rapids-dependency-file-generator`
- [x] Automatically write `GIT_COMMIT` file (https://github.com/rapidsai/rapids-build-backend/pull/30)
- [x] Add `disable-cuda` option (https://github.com/rapidsai/rapids-build-backend/pull/32)
- [x] Warn if `dependencies.yaml` is missing (https://github.com/rapidsai/rapids-build-backend/pull/33)
- [x] Release version 0.2.0 (https://github.com/rapidsai/rapids-build-backend/pull/34)
- [x] Prefix `-C` options with `rapidsai.` (https://github.com/rapidsai/rapids-build-backend/pull/37)
- [x] support projects with non-trivial `setup.py` (https://github.com/rapidsai/rapids-build-backend/issues/39)
- [x] set up publishing to public PyPI (https://github.com/rapidsai/rapids-build-backend/pull/47)

### Migrations
- [x] cucim (https://github.com/rapidsai/cucim/pull/736)
- [x] cudf (https://github.com/rapidsai/cudf/pull/15245)
- [x] cugraph (https://github.com/rapidsai/cugraph/pull/4393)
- [x] cugraph-ops (https://github.com/rapidsai/cugraph-ops/pull/652)
- [x] cuml (https://github.com/rapidsai/cuml/pull/5804)
- [x] cuopt (deferred until after RAPIDS 24.08, https://github.com/rapidsai/cuopt/issues/1879)
- [x] cuspatial (https://github.com/rapidsai/cuspatial/pull/1393)
- [x] cuvs (https://github.com/rapidsai/cuvs/pull/145)
- [x] cuxfilter (https://github.com/rapidsai/cuxfilter/pull/600)
- [x] dask-cuda (https://github.com/rapidsai/dask-cuda/pull/1343)
- [x] kvikio (https://github.com/rapidsai/kvikio/pull/389)
- [x] pynvjitlink (https://github.com/rapidsai/pynvjitlink/pull/88)
- [x] raft (https://github.com/rapidsai/raft/pull/2331, https://github.com/rapidsai/raft/pull/2344, https://github.com/rapidsai/raft/pull/2342)
- [x] rmm (https://github.com/rapidsai/rmm/pull/1502)
- [x] ucx-wheels (https://github.com/rapidsai/ucx-wheels/pull/9)
- [x] ucx-py (https://github.com/rapidsai/ucx-py/pull/1044, https://github.com/rapidsai/ucx-py/pull/1048)
- [x] ucxx (https://github.com/rapidsai/ucxx/pull/234)
- [x] wholegraph (https://github.com/rapidsai/wholegraph/pull/181)

### Other
- [x] Hook to verify alpha spec in `dependencies.yaml` (https://github.com/rapidsai/pre-commit-hooks/pull/28)
- [x] Any necessary `rapidsai/devcontainer` adjustments (https://github.com/rapidsai/devcontainers/pull/325)
- [x] replace "commit-file" with "commit-files" (or just remove that configuration) in projects that were updated before https://github.com/rapidsai/rapids-build-backend/pull/30
- [x] remove `.gitattributes` files left behind from when projects used `versioneer` (https://github.com/rapidsai/build-planning/issues/31#issuecomment-2176853319)

### dependencies.yaml updates (for devcontainers / DLFW)
- [x] devcontainers (https://github.com/rapidsai/devcontainers/pull/365)
- [x] cucim (not needed... only CUDA-suffixed dependency is cupy-cuda{nn}x, and we want to keep that suffixed)
- [x] cudf (https://github.com/rapidsai/cudf/pull/16183)
- [x] cugraph ( https://github.com/rapidsai/cugraph/pull/4552)
- [x] cugraph-ops (not needed... no CUDA-suffixed dependencies)
- [x] cugraph-gnn (https://github.com/rapidsai/cugraph-gnn/pull/16)
- [x] cuml (https://github.com/rapidsai/cuml/pull/5974)
- [x] cuspatial (https://github.com/rapidsai/cuspatial/pull/1414)
- [x] cuvs (https://github.com/rapidsai/cuvs/pull/247)
- [x] cuxfilter (https://github.com/rapidsai/cuxfilter/pull/609)
- [x] dask-cuda (https://github.com/rapidsai/dask-cuda/pull/1364, follow-up in https://github.com/rapidsai/dask-cuda/pull/1367)
- [x] kvikio ( https://github.com/rapidsai/kvikio/pull/411)
- [x] pynvjitlink (not needed... no CUDA-suffixed dependencies)
- [x] raft (https://github.com/rapidsai/raft/pull/2388)
- [x] rmm (https://github.com/rapidsai/rmm/pull/1627)
- [x] ucx-wheels (not needed ... no CUDA-suffixed dependencies, no dependencies.yaml)
- [x] ucx-py (https://github.com/rapidsai/ucx-py/pull/1057)
- [x] ucxx (https://github.com/rapidsai/ucxx/pull/251)
- [x] wholegraph (not needed ... no CUDA-suffixed dependencies, no dependencies.yaml)
- [x] forward-merge all these changes into branch-24.10
- [x] DLFW 24.08 build (RAPIDS/dlfw!158 in private repo)

mmccarty commented 3 months ago

Can we get a task list?

jakirkham commented 3 months ago

This was originally proposed for RAPIDS v24.06.00. However in discussion in our build-infra meeting today, we decided it is probably too late for this to go into v24.06.00. So moving this to v24.08.00

jameslamb commented 3 months ago

Can we get a task list?

I've updated the list of migrations with what I think is a complete set of projects we want to roll this out to. So we can track what and how much work remains to complete this.

Included any project in the rapidsai organization for which at least one of the following is true:

publishes wheels with CUDA suffix
has dependencies with CUDA suffix
set __git_commit__ in Python module-level constant
uses a build script to add alpha specs to RAPIDS dependencies

Some that don't have a dependencies.yaml but where I still think we want rapids-build-backend to handle CUDA suffixes for the project name and populating __git_commit__:

jakirkham commented 3 months ago

Ticked some boxes that looked done. Though please recheck and update if I've missed something

jameslamb commented 3 months ago

Added a few more repos that were previously missed to the task list (kvikio, ucx-py).

And 2 tasks capturing "make sure devcontainers adjust" and "make sure DLFW adjusts". Those can turn into something more detailed once I try some unified devcontainers builds of these projects (hopefully tomorrow) and when we (hopefully soon) test out a DLFW build targeting RAPIDS 24.08.

vyasr commented 3 months ago

As always, feel free to ping if you run into issues getting this working with devcontainers.

mmccarty commented 3 months ago

Thank you for the excellent use of task lists here!

jameslamb commented 2 months ago

From https://github.com/rapidsai/kvikio/pull/369#discussion_r1644861520 ... there are still some projects with a .gitattributes file like this:

dask_cuda/_version.py export-subst

From the git docs (link), that tells git to look for some special placeholders in that file and replace them with git information. That was added a few years ago to support versioneer, and is no longer necessary across RAPIDS.

Removing that as part of this, since "update git commit information checked into the package" is on of rapids-build-backend's responsibilities.

Identified these from GitHub search (link), ignoring the older projects still using versioneer

[x] cucim (https://github.com/rapidsai/cucim/pull/740)
[x] cugraph-ops (https://github.com/rapidsai/cugraph-ops/pull/657)
[x] cuxfilter (https://github.com/rapidsai/cuxfilter/pull/603)
[x] dask-cuda (https://github.com/rapidsai/dask-cuda/pull/1350)
[x] kvikio (https://github.com/rapidsai/kvikio/pull/369)

jameslamb commented 1 month ago

Any necessary DLFW adjustments

Trying to put the requirements into writing here. When reading this, recall that DLFW builds use the rapids-build-utils tools from https://github.com/rapidsai/devcontainers.

Requirements for DLFW, using `cudf` as an example

project name + wheel name should be "cudf" (no suffix)
it should have build and runtime dependencies on "rmm" (no suffix)
its runtime dependencies should differ based on CUDA version:
- CUDA 11 only: cubinlinker and ptxcompiler
- CUDA 12 only: pynvjitlink
dependencies in wheel metadata all must be satisfiable by one of the following:
- already have wheels built in the DLFW context
- can be installed from a package repository

To meet those requirements, it's necessary for rapids-build-backend to generate a list of dependencies which is both CUDA-major-version-specific and which does not have suffixes like -cu12.

Proposed Approach

Breaking dependencies.yaml dependency groups with CUDA suffixes up like this:

specific:
  - output_types: [requirements, pyproject]
    matrices:
      - matrix:
           cuda: "12.*"
           cuda_suffixed: "true"
         packages:
           - "rmm-cu12>=24.8.*"
      - matrix:
           cuda: "12.*"
           cuda_suffixed: "false"
         packages:
           - "rmm>=24.8.*"

That'd allow selecting the suffixed or unsuffixed lists in different contexts.

This relies on the changes in https://github.com/rapidsai/devcontainers/pull/365, and is being tested in https://github.com/rapidsai/cudf/pull/16183.

If that testing goes well, I'll propose similar changes across all RAPIDS repos using rapids-build-backend.

Build Contexts

This briefly describes 3 different build contexts where wheels with suffix-less names are produced, and how they'll take advantage of the changes in dependencies.yaml.

In all 3 contexts, I think we'll want to build wheels like this:

--no-build-isolation
--no-deps
-C rapidsai.disable-cuda=true
-C rapidsai.matrix-entry="cuda={cuda_ver};cuda_suffixed=false"

Individual-repo devcontainers

example: https://github.com/rapidsai/cudf/tree/branch-24.08/.devcontainer

Wheel metadata (click me)

For a wheel produced by running `build-all && pip show cudf`: ```text name: cudf ... Requires: ..., rmm-cu12, ... ```

How those dependencies are satisfied (click me)

* installed from package repositories (e.g. PyPI)

How wheels are built/ installed (click me)

```shell pip install --no-build-isolation --no-deps --editable {path_to_source_code} ``` ([code link](https://github.com/rapidsai/devcontainers/blob/623a7bf62cf6560063f257f9aa278870bc139f79/features/src/rapids-build-utils/opt/rapids-build-utils/bin/tmpl/python.build.editable.tmpl.sh#L18))

"unified" RAPIDS devcontainer

Wheel metadata (click me)

For a wheel produced by running `build-all && pip show cudf`: ```text name: cudf ... Requires: ..., rmm, ... ```

How those dependencies are satisfied (click me)

* RAPIDS things (e.g. `rmm`): built from source in the same container * all others: installed from package repositories (e.g. PyPI)

How wheels are built/ installed (click me)

```shell pip install --no-build-isolation --no-deps --editable {path_to_source_code} ``` ([code link](https://github.com/rapidsai/devcontainers/blob/623a7bf62cf6560063f257f9aa278870bc139f79/features/src/rapids-build-utils/opt/rapids-build-utils/bin/tmpl/python.build.editable.tmpl.sh#L18))

DLFW

Wheel metadata (click me)

```text name: cudf ... Requires: ..., rmm, ... ```

How those dependencies are satisfied (click me)

* RAPIDS things (e.g. `rmm`): built from source in the same container * all others: installed from package repositories (e.g. PyPI)

How wheels are built/ installed (click me)

```shell # build pip wheel --no-deps --no-build-isolation {path_to_source_code} # where ~/.out/rapids/run/python/* includes wheels for: # * all RAPIDS libraries + some of their dependencies we build from source # * all other runtime dependencies (downloaded with 'pip download' from package repositories) pip install --ignore-installed ~/.out/rapids/run/python/*.whl; ``` *([building code re-uses devcontainers rapids-build-utils](https://github.com/rapidsai/devcontainers/blob/branch-24.08/features/src/rapids-build-utils/opt/rapids-build-utils/bin/tmpl/python.build.wheel.tmpl.sh))* *(other code is private)*

sisodia1701 commented 1 month ago

Waiting for the tests to pass on the RAPIDS 24.10 build. Things related to rapids build backend are looking good, hence closing this issue.

rapidsai / build-planning