The only yamls that should be in the main workflows folder are the ones with the on tag. The goal of this is to have a structure that is easier to follow.
ci-third-party.yml: handles third-party elements (i.e. airflow)
We can then have other folders to keep everything else. I.e go, python etc. This should help CI look cleaner and keep to the DRY principle, and make it more "natural" to keep our CI consistent across everything. (Based on code here: https://github.com/m4rs-mt/ILGPU/pull/1046)
Mage / Tox Changes
All tox commands should go through mage to make CI consistent. It also removed the problem of needing to be in a different folder for it to work. Everything should be runnable from the root of the repo.
Mage Update / Verify
These are targets that Kubernetes has. Verify is equivalent to running CI locally in full. Update is everything like docs, linting, builds, being done for you in one bundled command.
Lookout CI Additions
We should:
Lint and build Lookout in CI
Cache lookout builds
Copy the build instead of re-building inside the image for the "prepare" job in CI.
dave-gantenbein
commented
6 months ago
This ticket will track potential things to cleanup in our CI.
A lot of this can be based on what is present in https://github.com/G-Research/fasttrackml/tree/main/.github/workflows
New Structure
The only yamls that should be in the main
workflowsfolder are the ones with theontag. The goal of this is to have a structure that is easier to follow.I think we should have a https://github.com/armadaproject/armada/blob/master/.github/workflows/ci.yml for each "type" of CI job:
We can then have other folders to keep everything else. I.e
go,pythonetc. This should help CI look cleaner and keep to the DRY principle, and make it more "natural" to keep our CI consistent across everything. (Based on code here: https://github.com/m4rs-mt/ILGPU/pull/1046)Mage / Tox Changes
All tox commands should go through mage to make CI consistent. It also removed the problem of needing to be in a different folder for it to work. Everything should be runnable from the root of the repo.
Mage Update / Verify
These are targets that Kubernetes has. Verify is equivalent to running CI locally in full. Update is everything like docs, linting, builds, being done for you in one bundled command.
Lookout CI Additions
We should: