dduportal
commented
1 year ago Hi @basil thanks for this excellent suggestion. With @olblak , we played a bit with (https://github.com/nestybox/sysbox) for that purpose (Docker in containers) in the past with nice results but only in AKS: in AWS EKS it was a nightmare.
We won't be able to work on this request because we lack time to spend on this on the upcoming 3 months, but that could be a great improvement before end of year if we can make it work as expected.
Regarding the VM agents, the team is currently working (for billing reasons initially) in 2 phases that could help:
- Using inbound agent mode for Azure VMs (notable faster startup of the agents) as part of #3481 (to allow moving agents to another more reliable network in Azure)
- Using DigitalOcean droplet as ephemeral VMs, which are way faster than Azure VM to start in https://github.com/jenkins-infra/helpdesk/issues/3576
We could experiment (and get some help) in the 2 OSUSL VMs we currently have as we planned to run a k3s custom cluster: that could be a great place to experiment with these technologies once migrated to a recent Ubuntu version
GitHub Actions runs builds on VMs with Docker preinstalled. Libraries like Testcontainers therefore work out of the box without any additional configuration.
The agent experience on https://ci.jenkins.io is not as great. Users have to choose between fast container agents which don't have Docker or slow VM agents which do have Docker.
Users should not have to choose between features and speed; rather, they should get both by default.
The most promising option for implementing this appears to be a combination of Firecracker and Kata Containers. A Kubernetes cluster configured to use Kata Containers with the Firecracker hypervisor would allow CI jobs to run in fast and lightweight VM agents with Docker preinstalled.