Problem creating cluser using Kind with Podman on Rocky Linux 8

[jwoods-millennial]

General Problem: When trying to create a cluster as per the Quick Start documentation, I have the following issue:

$ kind create cluster
enabling experimental podman provider
ERROR: failed to create cluster: running kind with rootless provider requires setting systemd property "Delegate=yes", see https://kind.sigs.k8s.io/docs/user/rootless/

I have followed the documentation at the link, and setup my system like the following:

$ podman info | grep cgroup
  cgroupControllers:
  cgroupManager: systemd
  cgroupVersion: v2

$ cat /etc/systemd/system/user@.service.d/delegate.conf 
[Service]
Delegate=yes

$ cat /etc/modules-load.d/iptables.conf
ip6_tables
ip6table_nat
ip_tables
iptable_nat

I have run the commands sudo update-grub (sudo grub2-mkconfig -o /etc/grub2.cfg on my system), sudo systemctl daemon-reload, and systemctl --user restart podman but they have not helped. I started explicitly podman as well, but the error persists:

$ KIND_EXPERIMENTAL_PROVIDER=podman kind create cluster
using podman due to KIND_EXPERIMENTAL_PROVIDER
enabling experimental podman provider
ERROR: failed to create cluster: running kind with rootless provider requires setting systemd property "Delegate=yes", see https://kind.sigs.k8s.io/docs/user/rootless/

Kind Version = kind v0.23.0 go1.21.10 linux/amd64 Podman Version = Version: 4.9.4-rhel OS/Arch: linux/amd64 OS / Kernel = Rocky Linux 8 / 6.9.3-1.el8.elrepo.x86_64

If anyone has any insight into what might be causing the issue or how to get around it, or if additional information would be helpful, please let me know. Thank you!

[BenTheElder]

https://kind.sigs.k8s.io/docs/user/rootless/

sudo update-grub

updating grub will only take affect on rebooting, but you appear to be on cgroup v2 already

systemctl --user restart podman

podman is usually daemonless which is why we don't mention this for podman

However rootless podman also documents: https://kind.sigs.k8s.io/docs/user/rootless/#:~:text=On%20some%20distributions,kind%20create%20cluster

did you try systemd-run?

[jwoods-millennial]

Sorry forgot to mention that, but yes I did try that systemd-run as well and get the same error.

[BenTheElder]

kind is detecting if the environment is correct, this error means we don't see the correct cgroup controllers available: https://github.com/kubernetes-sigs/kind/blob/b39bf4d29a842a56819e6f5fa23181a8a06f2b9e/pkg/cluster/internal/create/create.go#L252-L253

Can you try rocky linux 9? It's possible this has been fixed in the distro. Rootless containers is relatively more recently stabilized.

[jwoods-millennial]

Unfortunately upgrading is not an option for me. I was able to gain access to a remote cluster so I don't necessarily need this up and running, but it is concerning that the error message being provided here doesn't provide a full solution. I also tried to manually set certain permissions in my cgroup like so:

echo +cpuset | sudo tee /sys/fs/cgroup/cgroup.subtree_control
echo +cpu | sudo tee /sys/fs/cgroup/cgroup.subtree_control
echo +memory | sudo tee /sys/fs/cgroup/cgroup.subtree_control
echo +pids | sudo tee /sys/fs/cgroup/cgroup.subtree_control

but the CPU kept failing. That portion seems to be more of a Rocky OS problem so I wouldn't expect that to be figured out here.

[BenTheElder]

The error message is just a hint, the docs for rootless are here: https://kind.sigs.k8s.io/docs/user/rootless/