gVisor is currently not enabled

[fungiboletus]

gVisor is a Linux sandbox to run software containers. We used it in the past to run untrusted code and reduce risks, and we would like to keep using it.

gVisor is compatible with Kubernetes and containerd, or Docker. However, it is currently, and unfortunately, not supporting Kubernetes and Docker together. This is unsupported by cri-dockerd, the Docker kubernetes Container Runtime Interface.

Before SIM-PIPE started to rely on Kubernetes, we configured Docker to use gVisor. Now that SIM-PIPE uses Kubernetes with Docker, we cannot use gVisor.

Explored solutions:

Use of containerd instead of Docker in Kubernetes

We could use containerd instead of Docker. containerd provides nerdctl, a command line tool compatible with Docker. That would make the user experience relatively straightforward for our users.

It may make the user experience a bit more challenging, as users may expect Docker and classic Docker debugging strategies. One benefit is that users will not get confused between potential multiple docker instances on their machine.

Also, containerd doesn't expose as much metadata about the containers as Docker. In practice, we can extract fewer metrics using cAdvisor.

Use of gVisor as a default docker runtime

If we set gVisor (and runsc) as the default runtime in the docker daemon configuration, all Kubernetes containers will use gVisor. Unfortunately, many containers we use do not work with gVisor, so we need to have the possibility to run some containers on gvisor and some not.

Hot-swap the runc and runsc binaries

We could spin up the Kubernetes cluster without gVisor, and once everything is up and running, we could sneakily swap the runc and runsc binaries in /usr/bin to switch the default docker containers and the gvisor one. That kinda works. It's an ugly hack. However, some Kubernetes containers may restart or spawn anytime and not appreciate gVisor. This hack sounds like a bad solution.

Use of gVisor as a default docker runtime on a worker node

We could have a Kubernetes worker node with gVisor as the docker default runtime. This node would have a taint preventing containers from being deployed on itself, but we could have the SIM-PIPE dry runs containers run exclusively on this node. This method has yet to be tested and may not work.

Wait for the support of alternative runtimes in cri-dockerd, or write a pull request

The discussion is here: https://github.com/Mirantis/cri-dockerd/issues/111

[aleenathomas]

@fungiboletus Could you take this up ?

[fungiboletus]

I'm afraid I don't have a solution on this one and I would rather wait to see how it evolves.