rapidsai / devcontainers

18 stars 26 forks source link

Explicitly select for CUDA-suffixed dependencies in `rapids-make-pip-dependencies` #365

Closed jameslamb closed 2 months ago

jameslamb commented 2 months ago

Contributes to https://github.com/rapidsai/build-planning/issues/31.

In https://github.com/rapidsai/cudf/pull/16183 (and likely most other RAPIDS projects soon), I'm proposing decomposing lists of RAPIDS dependencies in dependencies.yaml into cuda_suffixed=true and cuda_suffixed=false groups, like this:

specific:
  - output_types: [requirements, pyproject]
    matrices:
      - matrix:
           cuda: "12.*"
           cuda_suffixed: "true"
         packages:
           - "rmm-cu12>=24.8.*"
      - matrix:
           cuda: "12.*"
           cuda_suffixed: "false"
         packages:
           - "rmm>=24.8.*"

This proposes always selecting those cuda_suffixed=true lists when rapids-make-pip-dependencies is called.

That should guarantee that that always generates a list of project names that can really be installed from PyPI, pypi.nvidia.com, and pypi.anaconda.org/rapidsai-wheels-nightly.

Notes for Reviewers

This should be totally safe to do even without that cuda_suffixed convention being rolled out to all the repos. rapids-dependency-file-generator does not care if you use matrix selectors that don't match anything.

Try yourself, e.g. in rmm:

rapids-dependency-file-generator \
  --output requirements \
  --file-key all \
  --matrix "cuda=12.2;nonsense=yes"

But don't we want unsuffixed dependencies for DLFW builds or unified devcontainers?

Yes, in those context we want to produce wheels with unsuffixed names (e.g. cudf) and for those wheels to depend on unsuffixed versions of everything else built from source there (e.g. rmm)... but this change doesn't affect that.

Both the suffixed and unsuffixed versions of every project with an entry in manifest.yaml is already being filtered out of the list of requirements generated by rapids-make-pip-dependencies.

https://github.com/rapidsai/devcontainers/blob/331dd88c45a2df33adda936e5fb467587ddc1e89/features/src/rapids-build-utils/opt/rapids-build-utils/bin/make-pip-dependencies.sh#L152-L160

https://github.com/rapidsai/devcontainers/blob/331dd88c45a2df33adda936e5fb467587ddc1e89/features/src/rapids-build-utils/opt/rapids-build-utils/bin/make-pip-dependencies.sh#L163

https://github.com/rapidsai/devcontainers/blob/331dd88c45a2df33adda936e5fb467587ddc1e89/features/src/rapids-build-utils/opt/rapids-build-utils/bin/make-pip-dependencies.sh#L167

This only affects the code used to generate a virtual env that builds are performed in.