rapidsai / dependency-file-generator

https://pypi.org/project/rapids-dependency-file-generator/
Apache License 2.0
15 stars 13 forks source link

feat: Add support for prepending conda channels via the CLI. #67

Closed bdice closed 6 months ago

bdice commented 6 months ago

This adds an option --prepend-channels "my_channel;my_other_channel" to the rapids-dependency-file-generator CLI. This will allow us to use rapids-dependency-file-generator with local channels containing PR artifacts fetched in CI workflows. If we combine this feature with some small changes in dependencies.yaml, we will be able to generate the entire CI environment in one pass rather than generating an environment with test dependencies and then installing the packages (e.g. libcuml and libcuml-tests) in a separate step.

This option is modeled after conda config --prepend channels new_channel (reference).

See original proposal in https://github.com/rapidsai/cuml/pull/5781#issuecomment-1958498274 (this PR is slightly improved from that proposal).

This is a partial solution for https://github.com/rapidsai/build-planning/issues/22.

bdice commented 6 months ago

To solve https://github.com/rapidsai/build-planning/issues/22, I'm thinking we'll do this kind of thing in each RAPIDS repo CI test script:

rapids-logger "Downloading artifacts from previous jobs"
CPP_CHANNEL=$(rapids-download-conda-from-s3 cpp)
PYTHON_CHANNEL=$(rapids-download-conda-from-s3 python)

rapids-dependency-file-generator \
  --output conda \
  --file_key test_python \
  --matrix "cuda=${RAPIDS_CUDA_VERSION%.*};arch=$(arch);py=${RAPIDS_PY_VERSION}" \
  --prepend-channels "${CPP_CHANNEL};${PYTHON_CHANNEL}" | tee env.yaml

rapids-mamba-retry env create --force -f env.yaml -n test

We would modify the test_cpp file key to include a dependency list containing libcuml and libcuml-tests. We would also modify the test_python file key to include a dependency list containing libcuml and cuml. Then this environment would be equivalent to the conda-merge output I was using in https://github.com/rapidsai/cuml/pull/5781.

GPUtester commented 6 months ago

:tada: This PR is included in version 1.9.0 :tada:

The release is available on:

Your semantic-release bot :package::rocket: