Use of Kubernetes based ephemeral runners

[mtupitsyn]

Abstract

Currently we have single, statically configured, self hosted runner, baked by a VM in GCP. For better scalability, resiliency and security, we need to support ephemeral runners. To minimize workflow execution time and optimize costs, solution must also provide auto-scaling.

Problem Statement

GitHub documentation refers to 2 opensource projects with auto-scaling support:

• actions-runner-controller/actions-runner-controller - creates self-hosted runners as containers in Kubernetes. Multiple container-as-a-service vendors supported, including GKE.
• philips-labs/terraform-aws-github-runner - operates AWS deployments using Terraform. Does not work for us, as we are not using AWS.

We can also come up with home-grown solution for auto-scaling.

This proposal is to choose 1st option from the list above, as option 2 is not suitable for us, and home-grown solution will require a lot of development and collaboration efforts.

Advantages of this approach:

• Containers come up quickly. Even if current size of runners pool is insufficient, Kubernetes will start new pods in seconds (as oppose to minutes when we use runners based on VMs).
• We have additional overview and control via Kubernetes control plane. In case of VM's, we have to rely on GCP tooling.
• Standard Kubernetes monitoring solutions (for ex. Prometehus + Grafana). In case of VM's, we have to rely on GCP tooling.

Disadvantages/Caveats:

• Possible complications with builds which invoke docker/podman. Essentially, for such builds we'll need to run docker-in-docker, i.e. connect to host node docker daemon via local socket. Depending on CRI implementation GKE is using, we may not be able to support all combinations. For example, builds using docker-compose may not be able to connect to host containerd.
• Possible complications with networking - will we be able to connect to vpc-interconnect?
• Pricing? Initially, GKE cluster is going to be more expensive, then a single VM. Though with amount of workflow runs increasing, container-based runs are going to be less expensive, then VM based.

Internal References

• CASMINST-3308

External References

• https://github.blog/changelog/2021-09-20-github-actions-ephemeral-self-hosted-runners-new-webhooks-for-auto-scaling/
• https://docs.github.com/en/actions/hosting-your-own-runners/autoscaling-with-self-hosted-runners

Proposed Solution(s)

• Create Kubernetes cluster in GKE
• Deploy actions-runner-controller/actions-runner-controller and connect it to our GitHub org

Impact of Action/Inaction

Static self-hosted runner is going to be a bottleneck when amount of Github workflow executions arises.

Further Information

• GKE Docs
• GKE Pricing
• GCP Compute Engine pricing

Suggested Reviewers

• [x] @nieuwsma
• [x] @rkleinman-hpe
• [ ] @denniswalker
• [ ] @gbaker-hpe

Comment Period

Comment period for this proposal shall close on November 10, 2021.

[rkleinman-hpe]

@denniswalker should probably also be a reviewer on this given the budget implications.

[nieuwsma]

@mtupitsyn : Possible complications with builds which invoke docker/podman. Essentially, for such builds we'll need to run docker-in-docker, i.e. connect to host node docker daemon via local socket. Depending on CRI implementation GKE is using, we may not be able to support all combinations. For example, builds using docker-compose may not be able to connect to host containerd.

The HMS builds use a LOT of docker in docker and docker-compose. (Im not sure how we will replace things like this as the docker desktop licensing goes away (maybe podman (although it doesnt support volumes 👎🏻 )). I digress. HMS needs the ability to do 'docker' based builds, very heavily. Its a 'must have' for our 40 builds.

• How would you see our builds working?
• Do you think its just 'complicated' or will we now know if its possible until we try it?
• How can we have a build strategy that keeps my builds away from this resource if its not compatible?

[mtupitsyn]

@nieuwsma I can confirm that docker-in-docker works, I had been using this configuration for quite a while. It has some issues with multi-tenancy (imagine containers from different tenants accessing the same instance of docker daemon on the host), but otherwise ok. Podman is supposed to eliminate exactly these types of issues with multi-tenancy.

[jsollom-hpe]

I support this proposal.