Open Eta0 opened 1 year ago
Alright so after some false starts, a little research, and a chat with @ChandonPierre, I'm going to try registering an auto-scaling solution inside a virtual cluster. Falling back to punting with a large dedicated virtual server for this repository.
Let's spin up some more VMs, I Think they can simply be scaled up.
Sounds like a plan. punting with vms 🏈
ml-containers
uses a self-hosted GitHub Actions runner to build container images through CI. It is currently only capable of handling one job at a time, sequentially. As a consequence, complex builds with many variations such asml-containers/torch
are taking up to 7 hours per commit to finish their CI.Very heavy commits slow down development, as it makes iteratively fixing bugs in a CI deployment impractical.
Either dedicating more resources to keep runners available or implementing some form of autoscaling like with actions-runner-controller may improve on the situation.