Add support for CI for workflows in repositories that don't follow the traditional RAPIDS branching model

[vyasr]

RAPIDS has a growing number of packages that do not follow the traditional release-branch model of Git branches that most RAPIDS repositories have used historically (development on branch-* branches, merging to main upon release). Instead, many new repositories (pynvjitlink, rapids-build-backend, ucx-wheels) are following the more common industry practice of simply developing directly on a main(/dev/etc) and making releases using Git tags to indicate the release version. While RAPIDS as a whole may eventually move to such a model, in the interim this leads to significant fragmentation, particularly when it comes to our CI practices and our usage of common images and shared-workflows. Since our workflows are "versioned" by RAPIDS branches and all RAPIDS repositories are tied to that, projects that do not follow this versioning scheme are still partially tied to the RAPIDS branching scheme by virtue of reusing the same shared workflows, and occasionally in other similar small ways as well.

We should consider solutions for this problem. One of the most obvious would be maintaining an additional branch in the shared-workflows repository for repositories that do not follow the same branching model. That would allow us to still standardize things that we need to maintain in a centralized location (such as our CI build matrix). Another thing that we might need (this is more administrative; there are no technical barriers here) is to decouple the matrix of container images more from our shared workflows since there are an increasing number of cases where we need CI images for build combinations not supported by our shared workflows (e.g. Python 3.12 images for pynvjitlink).

[bdice]

projects that do not follow this versioning scheme are still partially tied to the RAPIDS branching scheme

I'm generally okay with "decoupled" packages treating the shared-workflows branch as a plain version number, and not forcing them to stay in sync with the RAPIDS release cycle. If pynvjitlink, rapids-build-backend, etc. needs a newer workflow version, they can just update the RAPIDS version in .github/workflows/*.yaml as needed. I don't think this will present any major issues, aside from the need to update the build/test matrix on occasion.

decouple the matrix of container images more from our shared workflows

I'm pretty happy with "decoupled" projects maintaining their own matrix in the same way that pynvjitlink is doing here: https://github.com/rapidsai/pynvjitlink/blob/main/.github/actions/compute-matrix/action.yaml That doesn't seem like too much effort for the customization required.

[bdice]

Other options:

• Create a shared-workflows@main branch that stays in sync with the latest RAPIDS branch (e.g. forward-merge from branch-24.06 into main automatically), and use that in decoupled repos
• Create a shared-workflows@main branch and use it across all of RAPIDS, but pin to the RAPIDS branch at code freeze. This is somewhat like what we do with rapids-dask-dependency for pinning/unpinning. It would be extra work in the short term but would help us to start migrating all of RAPIDS to a main branch model.

[vyasr]

Create a shared-workflows@main branch

Some version of this is basically what I was proposing, yeah.