Open jameslamb opened 6 months ago
There are some things we'd need to improve in how we handle CI artifacts for packages without a direct CUDA dependency, so that they can be tested on CUDA runners with either CUDA 11 or CUDA 12. cuGraph's CI scripts describe this:
I expect the set of packages to be changed here to be a strict subset of those in #43, with the exception being packages like dask-cudf that have a transitive CUDA dependency via one of their dependencies (because as you noted, wheels need to maintain that dependency so that the dependency tree is CUDA aware even if the current package is not).
Description
Created from @bdice's suggestion at https://github.com/rapidsai/cudf/pull/15245#discussion_r1617517646.
Some RAPIDS libraries do not have a direct CUDA dependency, but we're still doing multiple conda builds (one per CUDA major version) for them.
Those projects' conda packages should only be built once, and be free from unnecessarily-declared CUDA dependencies.
See "Notes" for a concrete example.
Benefits of this work
Acceptance Criteria
For every RAPIDS library that doesn't have a CUDA dependency, the following should be true for their conda packages:
custreamz
should go from 6 conda packages (2 CUDA versions x 3 Python versions) to 3 (3 Python versions)Approach
Look through the RAPIDS projects for libraries meeting these criteria:
cuda{version}_
in their build strings on https://anaconda.org/rapidsai-nightlyAdd them to a task list here.
For each of those, as described in https://github.com/rapidsai/cudf/pull/15245#discussion_r1617517646, modify them as follows:
cuda-version
orcudatoolkit
shared-workflows
CI matrixNotes
Related to #43, which describes changing the workflows for pure-Python packages to only build against one Python version.
Example:
custreamz
For example, let's consider
custreamz
.Look at these packages on the
rapidsai-nightly
channel: https://anaconda.org/rapidsai-nightly/custreamz/files?version=24.08.00a66.Those
cuda11_*
andcuda12_*
packages differ only by their dependency oncuda-version
cuda11_*
:cuda-version >=11,<12.0a0
cuda12_*
:cuda-version >=12,<13.0a0
But
custreamz
shouldn't need acuda-version
dependency... it only contains Python code and only interacts withcudf
andcudf_kafka
via their Python APIs.The set of changes to address this issue for
custreamz
should be, roughly:cuda-version
dependency (conda/recipes/custreamz/meta.yaml)cuda
component frombuild.string
(conda/recipes/custreamz/meta.yam)ci/build_python.sh
(code link) andconda-python-build
uses in.github/workflows/
(e.g. .github/workflows/pr.yaml) such thatcustreamz
conda package is only built onceWhat about wheels?
This issue is about conda packages only.
Wheels have a different set of concerns, namely that they use suffixed package names to convey CUDA major version support, and that suffix affects everything in the dependency tree.
For example,
dask-cudf
depends oncudf
, and so we publish wheels with names likedask-cudf-cu11
(depending oncudf-cu12
) anddask-cudf-cu12
(depending ondask-cudf-cu12
).