Example crictl run/runp fail on a machine with a running k8s CP

[RonBarkan]

What happened:

On a Linux system with a successfully running single node Kubernetes control plane, with containerd, I am using the example run/runp commands here and here, and I am getting the following errors:

$ sudo crictl -r unix:///run/containerd/containerd.sock runp /tmp/nginx-pod.json 
E1122 19:08:31.584796 3158795 remote_runtime.go:193] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create containerd task: failed to create shim task: OCI runtime create failed: 
runc create failed: expected cgroupsPath to be of format \"slice:prefix:name\" for systemd cgroups, got \"/k8s.io/e5a83c8255cf21db9fa18c1999cb571db2139e87ed0c592324e851117eefc9f6\" instead: unknown"

and

$ sudo crictl -r unix:///run/containerd/containerd.sock run /tmp/container.json /tmp/nginx-pod.json
E1122 19:12:17.887097 3159492 remote_runtime.go:193] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create containerd task: failed to create shim task: OCI runtime create failed: 
runc create failed: expected cgroupsPath to be of format \"slice:prefix:name\" for systemd cgroups, got \"/k8s.io/7f31c4319bc73ca556da493fee2f7c28abef514e0103e7277f766556da9c0d8f\" instead: unknown"

What you expected to happen:

The examples to work.

How to reproduce it (as minimally and precisely as possible):

Installed containerd version 1.6.12 through apt. crictl is v1.28.0.

The config.toml was generated using:

containerd config default | sed "s/SystemdCgroup *= *false/SystemdCgroup = true/" | sudo tee /etc/containerd/config.toml

Which means it uses SystemdCgroups = true.

Anything else we need to know?:

Cilium with kube-proxy is installed on the healthy Kubernetes control plane.

In case this is important:

sudo cat /var/lib/kubelet/config.yaml | grep cgroup
cgroupDriver: systemd

Environment:

• Container runtime or hardware configuration:
  • containerd 1.6.12
  • crictl v1.28.0
  • kubelet (presumably not relevant): v1.29.6
• OS (e.g: cat /etc/os-release): Ubuntu rodete
• Kernel (e.g. uname -a): 6.9.10-1rodete5-amd64
• Others:

[kannon92]

Reading this I don’t think this is a bug with crictl but with containerd. Your version is pretty old so I’d maybe ask containerd on this one.