GitHub workflows improvements

NVIDIA / NeMo-Curator

Scalable data pre processing and curation toolkit for LLMs

Apache License 2.0

589 stars 79 forks source link

GitHub workflows improvements #259

Open sarahyurick opened 1 month ago

sarahyurick commented 1 month ago

There are a couple of GitHub Actions I want to add to NeMo Curator:

[x] GPU CI (already have https://github.com/NVIDIA/NeMo-Curator/pull/253 open for this)
[ ] Auto label PRs with the gpuci label if the PR is (1) created by a user with write access and (2) modifying at least 1 Python file
[ ] Automate Changelog build
[x] Automate PyPI releases
[ ] Display suggested style changes when the pre-commit CI fails (e.g., with Ruff)

After the first one is merged, I can open separate PRs to add the others.

sarahyurick commented 1 month ago

Note from Oliver about 2: " This file didn’t work because using secrets.GITHUB_TOKEN is not permitted to trigger other workflows. GH disallows that to prevent spam. You would need to use your personal access token (PAT). However, since we’re in a public environment, you would need to limit that to a protected branch (via environments). It’s still doable (from branch main, react to all issue change events), but a bit more tricky. As long as the PAT is not exposed to all branches, this is fine. "

sarahyurick commented 1 month ago

Additional tasks outside of GitHub:

[ ] Set up a nightly or weekly job that runs these same PyTests with RAPIDS Nightly Builds to identify any upstream breakages
[x] Releasebot on Slack
[ ] CI alerts on Slack

sarahyurick commented 1 month ago

NeMo files of interest:

sarahyurick commented 1 month ago

Suggestion from @praateekmahajan: parametrizing the GPU tests with commonly used clients.

get_client(cluster_type="gpu", protocol="ucx")
get_client(cluster_type="gpu", protocol="tcp")
Maybe even set_torch_to_use_rmm/enable_spilling (we shouldn't try all the permutations, just the ones where we expect different behavior)

Depending on how many GPUs the node has we could we even try a multi-GPU setup.

sarahyurick commented 1 week ago

Regarding the gpuCI label:

Always have it run (without having to re-add the label) for Nvidians/RAPIDS members
For non-verified users we could use /okay to test comments