Closed vyasr closed 1 month ago
Can we get a task list?
This was originally proposed for RAPIDS v24.06.00
. However in discussion in our build-infra meeting today, we decided it is probably too late for this to go into v24.06.00
. So moving this to v24.08.00
Can we get a task list?
I've updated the list of migrations with what I think is a complete set of projects we want to roll this out to. So we can track what and how much work remains to complete this.
Included any project in the rapidsai
organization for which at least one of the following is true:
__git_commit__
in Python module-level constantSome that don't have a dependencies.yaml
but where I still think we want rapids-build-backend
to handle CUDA suffixes for the project name and populating __git_commit__
:
Ticked some boxes that looked done. Though please recheck and update if I've missed something
Added a few more repos that were previously missed to the task list (kvikio
, ucx-py
).
And 2 tasks capturing "make sure devcontainers adjust" and "make sure DLFW adjusts". Those can turn into something more detailed once I try some unified devcontainers builds of these projects (hopefully tomorrow) and when we (hopefully soon) test out a DLFW build targeting RAPIDS 24.08.
As always, feel free to ping if you run into issues getting this working with devcontainers.
Thank you for the excellent use of task lists here!
From https://github.com/rapidsai/kvikio/pull/369#discussion_r1644861520 ... there are still some projects with a .gitattributes
file like this:
dask_cuda/_version.py export-subst
From the git
docs (link), that tells git
to look for some special placeholders in that file and replace them with git information. That was added a few years ago to support versioneer
, and is no longer necessary across RAPIDS.
Removing that as part of this, since "update git commit information checked into the package" is on of rapids-build-backend
's responsibilities.
Identified these from GitHub search (link), ignoring the older projects still using versioneer
cucim
(https://github.com/rapidsai/cucim/pull/740)cugraph-ops
(https://github.com/rapidsai/cugraph-ops/pull/657)cuxfilter
(https://github.com/rapidsai/cuxfilter/pull/603)dask-cuda
(https://github.com/rapidsai/dask-cuda/pull/1350)kvikio
(https://github.com/rapidsai/kvikio/pull/369)Any necessary DLFW adjustments
Trying to put the requirements into writing here. When reading this, recall that DLFW builds use the rapids-build-utils
tools from https://github.com/rapidsai/devcontainers.
cudf
as an example"cudf"
(no suffix)"rmm"
(no suffix)cubinlinker
and ptxcompiler
pynvjitlink
To meet those requirements, it's necessary for rapids-build-backend
to generate a list of dependencies which is both CUDA-major-version-specific and which does not have suffixes like -cu12
.
Breaking dependencies.yaml
dependency groups with CUDA suffixes up like this:
specific:
- output_types: [requirements, pyproject]
matrices:
- matrix:
cuda: "12.*"
cuda_suffixed: "true"
packages:
- "rmm-cu12>=24.8.*"
- matrix:
cuda: "12.*"
cuda_suffixed: "false"
packages:
- "rmm>=24.8.*"
That'd allow selecting the suffixed or unsuffixed lists in different contexts.
This relies on the changes in https://github.com/rapidsai/devcontainers/pull/365, and is being tested in https://github.com/rapidsai/cudf/pull/16183.
If that testing goes well, I'll propose similar changes across all RAPIDS repos using rapids-build-backend
.
This briefly describes 3 different build contexts where wheels with suffix-less names are produced, and how they'll take advantage of the changes in dependencies.yaml
.
In all 3 contexts, I think we'll want to build wheels like this:
--no-build-isolation
--no-deps
-C rapidsai.disable-cuda=true
-C rapidsai.matrix-entry="cuda={cuda_ver};cuda_suffixed=false"
example: https://github.com/rapidsai/cudf/tree/branch-24.08/.devcontainer
Waiting for the tests to pass on the RAPIDS 24.10 build. Things related to rapids build backend are looking good, hence closing this issue.
rapids-build-backend
is a wrapper around standard backends like scikit-build-core and setuptools that handles some of the standard issues that we face for RAPIDS packages (CUDA versioning, alpha versions, etc). Substituting it into existing RAPIDS Python packages should be fairly painless, but will require some careful testing to verify that nothing is broken.The backend need not be updated in lockstep across all of RAPIDS for this to work, and merging PRs in any order should be generally safe. However, since there are cases like unified devcontainers where underlying build commands may need an update, it would be best to try and test at least a couple of core packages together to verify that everything works as expected.