search

rapidsai / build-planning

Tracking for RAPIDS-wide build tasks
https://github.com/rapidsai
0 stars 4 forks source link

Update package metadata to allow safe coexistence of nightly and release conda channels #24

Open vyasr opened 8 months ago

vyasr commented 8 months ago

Currently RAPIDS conda packages pin other RAPIDS packages in recipes using version constraints that are effectively of the form YY.MM.*. In conda, the trailing .* allows nightlies. That makes using the rapidsai and rapidsai-nightly channels in the same conda install/env creation command potentially problematic, and could lead to situations where user install commands result in environments that are technically invalid. This is especially likely to be problematic for rapids-dask-dependency given the high rate of dask changes and the fact that we track the main branch until just before releases, causing potential problems around release time.

With pip packages, our use of nightly packages is in some sense more controlled. pip will only use nightlies if passed the --pre flag on the command line or if the version constraint explicitly includes dev versions, e.g. via a constraint of the form >=YY.MM.00a0. To accommodate this, we set the versions of our packages inside our package build scripts, directly modifying pyproject.toml before invoking pip wheel. Our final releases do not have constraints specified in this way. That behavior affords users some degree of protection. Although users could still break things by manually specifying --pre, the default behavior is safe and it's fair to say users are on their own if they use --pre with release channels. Therefore, mixing the nightly and stable pip indexes is in this sense relatively safer than mixing the nightly and stable conda channels.

We should consider rewriting dependencies in our conda packages to specify constraints in a way that only allows nightly packages to be installed when building nightlies. This could easily be accomplished by using an environment variable that is read in the meta.yaml recipe, by parsing the VERSION file to determine whether the current version corresponds to a nightly build, or by any number of other similar strategies.

bdice commented 7 months ago

We should consider rewriting dependencies in our conda packages to specify constraints in a way that only allows nightly packages to be installed when building nightlies.

This could negatively impact hotfixes. Recently I believe we used rmm packages from the 24.02 release while building the 24.02.2 hotfix for cudf.