Prepare to use CRI instead of built-in dockershim

[rata]

Kubernetes uses the Container Runtime Interface, created by Kubernetes, to communicate with the container runtime. Docker doesn't provide a way to interact with the CRI and support is built into the kubelet to do that. It is called dockershim

Kubernetes will add a warning about dockershim being deprecated in Kubernetes 1.20: https://github.com/kubernetes/enhancements/pull/1985. However, as you can see in that KEP, there is no migration guide created yet and is not clear if dockershim will be maintainer by others out of the Kubelet or not. Of course, the CRI interface is alpha too :joy:. The target is not really close nor too far away: Kubernetes 1.21 (https://github.com/kubernetes/enhancements/pull/1985/files#diff-8368b5d11d71e49cc6ca0e6289548c45R143-R146)

There are several options (like use containerd that supports the CRI as a plugin), but we should take into account:

1. The CRI interface is alpha (will be moved to beta soon) and as it isn't user visible, breaking changes can happen even in Beta IIUC.
2. We will need to choose if we want docker or other container runtime and how to update it. Docker is shipped with Flatcar, so it is automatically updated too.

More info regarding 1:

• It seems in k8s 1.17 the CRI interface was changed from v1alpha1 to v1alpha2 to add support for shared PID namespace. The kubelet in 1.17 started serving only v1alpha2, so an old container runtime probably didn't work. Also, containerd was updated to use v1alpha2 but probably didn't work with a kubelet running kubernetes < 1.17 (using CRI v1alpha1). Will try to find the links to specific commits if needed, this is something we saw with @mauriciovasquezbernal and @alban (please correct me! :)) on a completely unrelated task.
• The CRI will probably see similar breaking changes when user namespace support is added to Kubernetes
• This means that upgrading/downgrading the Kubelet is not so easy when those changes happen, IF WE ARE NOT MISSING SOMETHING, as coordination between Kubernetes and the container runtime versions is needed (probably until the interface is GA).
• Today this is not an issue because dockershim is built in the Kubelet, so if Kubernetes changes the interface, the change is built-in and it just works
• Other things technically possible to consider (but seems ugly) is to use a container runtime to run another container runtime and use the latter to communicate with Kubernetes and manage containers on the host. In those cases, we can upgrade kubelet and container runtime in lockstep, but seems really meta and error-prone.
• Other projects, like CRI-O release versions for each Kubernetes release and kubernetes and CRI-O need to be upgraded in lockstep.
• It will be very nice if container runtimes or the kubelet support more than one interface for some releases, so there is no flag day when upgrading. Maybe is something we can contribute to some projects or something that the community decides to do before removing dockershim :crossed_fingers:

More info regarding 2:

• I'm not sure if Flatcar plans to support other container runtime or if we can install and upgrade (like security upgrades) the container runtime in some way (torcx?)

EDIT: Just to be clear, I guess this is an issue until the CRI is GA (or tools supports more than one interface). My guess is that once CRI is GA, this won't be an issue anymore. Therefore, another way to look into this is: see how to smartly workaround the problems that arise until the interface is stable and all is solved :)

[invidian]

containerd is close to be usable on Flatcar (https://github.com/flatcar-linux/Flatcar/issues/201) and it provides CNI, though not out of the box (needs to be configured). We could use it instead of Docker. I tried it on Ubuntu and it worked fine.

[rata]

@invidian does that solve/helps with the upgrade/downgrade part? I don't know enough about torcx :)

[rata]

CRI change might take longer: https://github.com/kubernetes/kubernetes/pull/96325#issuecomment-724899472. Seems kubelet will continue to support one interface, runtimes might support more than one if they want (that would be SUPER helpful for us :)).

This change also was merged for 1.20: https://github.com/kubernetes/kubernetes/commit/9fcede9d5b37fccfe3269bd93559c551e47de5ff