support for Kubernetes RuntimeClass API

[ahmetb]

https://gvisor.dev/docs/user_guide/kubernetes/ says:

You can also setup Kubernetes nodes to run pods in gvisor using the containerd CRI runtime and the gvisor-containerd-shim. You can use either the io.kubernetes.cri.untrusted-workload annotation or RuntimeClass to run Pods with runsc. You can find instructions here.

I prefer using the RuntimeClass CRD, but I can't seem to find anything about RuntimeClass in this repo. Is it documented elsewhere?

[ianlewis]

It's here since the right terminology is runtime handler for CRI (RuntimeClass is an object in k8s that specifies the handler used by CRI). https://github.com/google/gvisor-containerd-shim/blob/master/docs/runtime-handler-shim-v2-quickstart.md

[ianlewis]

If you set up containerd for your kubelet per the above doc, and then create a RuntimeClass with handler of 'runsc' then specifying that RuntimeClass in your pod should work. See the minikube PR https://github.com/kubernetes/minikube/pull/4494

Particularly the RuntimeClass https://github.com/kubernetes/minikube/pull/4494/files#diff-4c2f5e9c15fde5dbe6c7512ae08d1d02

and containerd config https://github.com/kubernetes/minikube/pull/4494/files#diff-17f6d9e2e28e4fb05d88672eed46bd9b

[ianlewis]

Hopefully this PR makes it clearer. https://github.com/google/gvisor-containerd-shim/pull/33

See: https://github.com/ianlewis/gvisor-containerd-shim/blob/runtimeclass-doc/docs/runtime-handler-shim-v2-quickstart.md

[MichelMan]

Hello,

I'm trying to setup Kubernetes (v1.14.3) + containerd (v1.2.6)+ runsc. Trust the documentation (https://github.com/ianlewis/gvisor-containerd-shim/blob/runtimeclass-doc/docs/runtime-handler-shim-v2-quickstart.md), i already installed and configured containerd-shim-runsc-v1 and critctl on a specific node on my cluster k8s. It works ...

... but I'm having some issues when trying to run runsc in a pod k8s after created and deployed the RuntimeClass on this specific node (targets this node with nodeselector).

The events when I'm starting the pod:

Events:
  Type     Reason                  Age                     From                  Message
  ----     ------                  ----                    ----                  -------
  Normal   Scheduled               8m54s                   default-scheduler     Successfully assigned default/nginx-gvisor to p2kubenode3
  Warning  FailedCreatePodSandBox  3m30s (x26 over 8m54s)  kubelet, p2kubenode3  Failed create pod sandbox: rpc error: code = Unknown desc = RuntimeHandler "runsc" not supported

Note: p2kubenode3 is the node where i installed containerd-shim-runsc-v1 and critctl .

I don't know why it said that "runsc" not supported....

gVisor Runtime Class:

apiVersion: node.k8s.io/v1beta1
kind: RuntimeClass
metadata:
  name: gvisor
handler: runsc

Pod with the gVisor Runtime Class:

apiVersion: v1
kind: Pod
metadata:
  name: nginx-gvisor
spec:
  runtimeClassName: gvisor
  containers:
  - name: nginx
    image: nginx
  nodeSelector:
    gvisor: "enabled"

[ianlewis]

@MichelMan It seems like that should work, when you use crictl what happens? Are you sure you are specifying the --runtime flag when running the sandbox with crictl? Do you see the runsc processes running?

[MichelMan]

@ianlewis thanks for your reply.

After installed crictl and created the nginx Sandbox in gVisor:

$:~/gvisor-containerd-shim$ sudo crictl pods
POD ID              CREATED             STATE               NAME                NAMESPACE           ATTEMPT
c335758128f12       3 minutes ago       Ready               nginx-sandbox       default             1
$:~/gvisor-containerd-shim$ sudo crictl inspectp c335758128f12 | grep "runsc"
    "runtimeHandler": "runsc",
    "runtimeType": "io.containerd.runsc.v1",

The runsc processes seems running:

$:~/gvisor-containerd-shim$ ps -A | grep runsc
 7590 ?        00:00:00 runsc

After created the nginx Container in the Sandbox:

$:~/gvisor-containerd-shim$ sudo crictl ps
CONTAINER ID        IMAGE               CREATED             STATE               NAME                ATTEMPT             POD ID
98c2f958ecc43       nginx               20 seconds ago      Running             nginx               0                   c335758128f12
$:~/gvisor-containerd-shim$ sudo crictl inspect 98c2f958ecc43 | grep runsc
    "runtimeType": "io.containerd.runsc.v1"

Then, I verified that nginx is running in gVisor:

$:~/gvisor-containerd-shim$ sudo crictl exec 98c2f958ecc43 dmesg
[    0.000000] Starting gVisor...
[    0.235354] Creating cloned children...
[    0.666510] Feeding the init monster...
[    1.038358] Rewriting operating system in Javascript...
[    1.162553] Consulting tar man page...
[    1.653117] Daemonizing children...
[    2.114803] Generating random numbers by fair dice roll...
[    2.291879] Moving files to filing cabinet...
[    2.691147] Segmenting fault lines...
[    3.185163] Forking spaghetti code...
[    3.619175] Reading process obituaries...
[    3.874116] Ready!

Maybe i have to install and configure runsc on all nodes on my cluster k8s? Not only on a specific node in oder to make runtimeClass work.

[ianlewis]

@MichelMan You need to install it on all the nodes if you don't use a node selector or node affinity on the Pod. But since you are using the node selector it should work.

Currently nodeSelector should work on the Pod but nodeSelector isn't supported on the RuntimeClass until Kubernetes 1.16+

[ianlewis]

I'm closing this since I added docs in #33

@MichelMan Please create another bug if the node selector continues to be an issue.