CoreDNS containers hang in Container Creating state - no cpu controller access

[anthonyhartin]

The usernetes install script fails on the condition of waiting for coreDNS containers to reach ready state:

• kubectl apply -f /home/fedora/usernetes/manifests/coredns.yaml serviceaccount/coredns created clusterrole.rbac.authorization.k8s.io/system:coredns created clusterrolebinding.rbac.authorization.k8s.io/system:coredns created configmap/coredns created deployment.apps/coredns created service/kube-dns created
• set +x [INFO] Waiting for CoreDNS pods to be available
• sleep 3
• kubectl -n kube-system wait --for=condition=ready pod -l k8s-app=kube-dns timed out waiting for the condition on pods/coredns-697b4969d7-6925r timed out waiting for the condition on pods/coredns-697b4969d7-zg2r2

kubectl get pods -A

NAMESPACE NAME READY STATUS RESTARTS AGE kube-system coredns-697b4969d7-6925r 0/1 ContainerCreating 0 4m9s kube-system coredns-697b4969d7-zg2r2 0/1 ContainerCreating 0 4m9s

kubectl describe pod coredns-697b4969d7-6925r -n kube-system

Events: Type Reason Age From Message

---

Warning FailedScheduling 4m42s default-scheduler 0/1 nodes are available: 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate. Normal Scheduled 4m41s default-scheduler Successfully assigned kube-system/coredns-697b4969d7-6925r to un-worker.novalocal Warning FailedCreatePodSandBox 13s (x20 over 4m30s) kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create containerd task: failed to create shim task: OCI runtime create failed: the requested cgroup controller cpu is not available: unknown

So we see that the problem occurs because the user has no access to the cpu controller.

I followed the instructions on confirming cgroups v2 available and delegating control

ls -lah /sys/fs/cgroup/cgroup.controllers

-r--r--r--. 1 root root 0 Mar 31 05:49 /sys/fs/cgroup/cgroup.controllers

grep cgroup /proc/filesystems

nodev cgroup nodev cgroup2

sudo cat /etc/systemd/system/user@.service.d/delegate.conf

[Service] Delegate=yes

sudo grubby --info=ALL | grep args

args="ro no_timer_check net.ifnames=0 console=tty1 console=ttyS0,115200n8 systemd.unified_cgroup_hierarchy=1" args="ro no_timer_check net.ifnames=0 console=tty1 console=ttyS0,115200n8 systemd.unified_cgroup_hierarchy=1" args="ro no_timer_check net.ifnames=0 console=tty1 console=ttyS0,115200n8 systemd.unified_cgroup_hierarchy=1" args="ro no_timer_check net.ifnames=0 console=tty1 console=ttyS0,115200n8 systemd.unified_cgroup_hierarchy=1"

After reboot, cpu and io control is not granted to the user

cat /sys/fs/cgroup/user.slice/user-$(id -u).slice/user@$(id -u).service/cgroup.subtree_control

memory pids

There are other people who having difficulty with delegating cpu control in other contexts. For instance this post describes difficulties and possible work arounds: https://unix.stackexchange.com/questions/624428/cgroups-v2-cgroup-controllers-not-delegated-to-non-privileged-users-on-centos-s

I have tried the work arounds and different configuration described, but nothing seems to work. I have tried operating systems CentOS 8, Fedora 33 and Fedora 35,, so far.

[lukasheinrich]

hi @AkihiroSuda - do you have any guidance on this. If so that'd be much appreciated!

[AkihiroSuda]

Does it work if you change the runtime from crun to runc?

https://github.com/rootless-containers/usernetes/blob/191445c1b1f55a3064bb90d0559fc5fedc354070/boot/containerd.sh#L25

[AkihiroSuda]

The CI uses Fedora 35. Vagrantfile can be found here: https://github.com/rootless-containers/usernetes/blob/master/Vagrantfile

[lukasheinrich]

Thanks @AkihiroSuda ! @anthonyhartin - were you able to try this?

[anthonyhartin]

Hi @lukasheinrich @AkihiroSuda,

I tried changing crun to runc and it didn't help. The upstream problem which needs to be fixed before anything else, is that the described procedure for granting the user control of the cpu and the io does not work for me. Once cgroups v2 is present and configured for user delegation, and the boot time arguments added, the reboot doesn't actually grant user control.

I found at least one other instance online where someone else also couldn't get user delegation to work. The workarounds provided didn't do anything for me. I've tried on several different operating systems including fc35.

cheers.

[anthonyhartin]

Just an update, The initial part of the installation - delegation of cpu and io to the user - it works on my laptop, but not on our openstack virtual machines. So it may be the case that bare metal is ok, but virtual machines, not. I'm not sure whether it is just our virtual machines, or virtual machines in general.. I will try and track down further error messages to see if I can clarify the situation

[AkihiroSuda]

Maybe you need DefaultCPUAccounting=yes https://www.freedesktop.org/software/systemd/man/systemd-system.conf.html

[AkihiroSuda]

it works on my laptop, but not on our openstack virtual machines

systemctl --user show might be useful to check the differences across your laptop and VM

[anthonyhartin]

OK, I set the CPUAccounting parameter and ran systemctl --user show on the VM and LAPTOP. Output attached below. Just comparing the accounting parameters, they are both the same: DefaultCPUAccounting=yes DefaultBlockIOAccounting=no DefaultMemoryAccounting=yes DefaultTasksAccounting=yes

but it's still not working on the VM. I don't yet see anything else obvious, but I will keep looking into it.

VM_systemctl_user.txt LAPTOP_systemctl_user.txt

[AkihiroSuda]

Let me close this, as the architecture was changed in "Generation 2": https://github.com/rootless-containers/usernetes/releases/tag/gen2-v20230906.0