jameslamb
commented
6 months ago I'm +1 on this idea. From my experience so far these things are so tightly coupled that it'd be useful to develop and release them together from the same repo.
PR merges in gha-tools trigger a release, which then triggers a rebuild of ci-imgs in order to embed the latest version of the tools.
Adding a link to the specific mechanism, since I happened to have just been looking at this recently.
# Install gha-tools
RUN wget https://github.com/rapidsai/gha-tools/releases/latest/download/tools.tar.gz -O - \
| tar -xz -C /usr/local/bin
This task is a follow-up to #48. Many of the reasons are the same. The gha-tools repo defines a number of bash scripts that we use in our CI scripts throughout RAPIDS. These tools are automatically installed in the images: https://github.com/rapidsai/ci-imgs/blob/main/ci-conda.Dockerfile#L103. Some of these tools rely on environment variables set in the images, and some developments between the repos must be coordinated. For example, the addition of
rapids-configure-sccacheinvolved the simultaneous removal of these variables from CI images to verify that the tool was actually setting the right variables. Like with miniforge-cuda, PR merges in gha-tools trigger a release, which then triggers a rebuild of ci-imgs in order to embed the latest version of the tools. There is a lot of extra process here that we could elide by simply moving the tools into the ci-imgs repository. We could also make it easier to test changes; if the two are in the same repository, then a change to a tool would automatically trigger an image build with the new tools and we would only need to add a parameter to our shared workflows to enable rerunning a workflow from another repo (e.g. cudf) with the latest images.