search

canonical / k8s-snap

Canonical Kubernetes is an opinionated and CNCF conformant Kubernetes operated by Snaps and Charms, which come together to bring simplified operations and an enhanced security posture on any infrastructure.
GNU General Public License v3.0
20 stars 6 forks source link

failed to initialize top level QOS containers: root container [kubepods] doesn't exist #483

Open slapcat opened 3 weeks ago

slapcat commented 3 weeks ago

Summary

Kubelet fails to start after bootstrap completes. The system journal shows the following error from the snap.k8s.kubelet.service unit:

kubelet.go:1547] "Failed to start ContainerManager" err="failed to initialize top level QOS containers: root container [kubepods] doesn't exist"

What Should Happen Instead?

Kubelet is configured correctly to start on the system.

Reproduction Steps

  1. sudo snap install k8s --channel=latest/edge --classic
  2. sudo k8s bootstrap
  3. sudo systemctl status snap.k8s.kubelet.service

System information

# cat /etc/os-release 
PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy

Can you suggest a fix?

This issue might be related to the upstream bug here: https://github.com/cri-o/cri-o/issues/7701

Adding these lines to the bottom of /var/snap/k8s/common/args/kubelet and restarting snap.k8s.kubelet.service fixes the issue:

--cgroups-per-qos=false
--enforce-node-allocatable=""

Are you interested in contributing with a fix?

No response

neoaggelos commented 3 weeks ago

Hi @slapcat, thank you for bringing this up.

See https://github.com/canonical/microk8s/issues/4361#issuecomment-1902584389 for the related issue in MicroK8s. Though these args can help to work around the bug, in MicroK8s we went the route of adjusting the kubelet service so that the required cgroup controllers are properly delegated, see https://github.com/canonical/microk8s/pull/4503

We will need to adjust that patch for k8s-snap as well.