podman: Failed to add conmon to cgroupfs sandbox cgroup

[djarbz]

Maintainer: @oskarirauta Environment: OpenWRT@23.05 x86 on Sophos SG-105r1

Description:

I installed Podman per the wiki and I am unable to get any container to run.
Unfortunately, Cgroups is beyond my scope of knowledge to properly troubleshoot.

root@OpenWrt:/srv/.podman/scripts# podman run hello-world
WARN[0000] Failed to add conmon to cgroupfs sandbox cgroup: creating cgroup path /libpod_parent/conmon: write /sys/fs/cgroup/cgroup.subtree_control: invalid argument
Error: OCI runtime error: crun: writing file `/sys/fs/cgroup/cgroup.subtree_control`: Invalid argument

Podman Info

```yaml host: arch: amd64 buildahVersion: 1.33.2 cgroupControllers: - cpuset - cpu - io - memory - pids - rdma cgroupManager: cgroupfs cgroupVersion: v2 conmon: package: Unknown path: /usr/bin/conmon version: 'conmon version 2.1.8, commit: ' cpuUtilization: idlePercent: 99.24 systemPercent: 0.3 userPercent: 0.46 cpus: 2 databaseBackend: sqlite distribution: distribution: openwrt version: 23.05.0 eventLogger: none freeLocks: 2046 hostname: OpenWrt idMappings: gidmap: null uidmap: null kernel: 5.15.134 linkmode: dynamic logDriver: k8s-file memFree: 1687359488 memTotal: 1983795200 networkBackend: netavark networkBackendInfo: backend: netavark dns: package: Unknown path: /usr/lib/podman/aardvark-dns version: aardvark-dns 1.9.0 package: Unknown path: /usr/lib/podman/netavark version: netavark 1.9.0 ociRuntime: name: crun package: Unknown path: /usr/bin/crun version: "crun version 1.12\ncommit: \nrundir: /run/crun\nspec: 1.0.0\n+SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL" os: linux pasta: executable: "" package: "" version: "" remoteSocket: exists: true path: /run/podman/podman.sock security: apparmorEnabled: false capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT rootless: false seccompEnabled: true seccompProfilePath: /usr/share/containers/seccomp.json selinuxEnabled: false serviceIsRemote: false slirp4netns: executable: /usr/bin/slirp4netns package: Unknown version: |- slirp4netns version 1.2.2 commit: 0ee2d87523e906518d34a6b423271e4826f71faf libslirp: 4.7.0 SLIRP_CONFIG_VERSION_MAX: 4 libseccomp: 2.5.4 swapFree: 0 swapTotal: 0 uptime: 5h 29m 19.00s (Approximately 0.21 days) variant: "" plugins: authorization: null log: - k8s-file - none - passthrough network: - bridge - macvlan - ipvlan volume: - local registries: search: - docker.io - registry.fedoraproject.org - registry.access.redhat.com store: configFile: /etc/containers/storage.conf containerStore: number: 2 paused: 0 running: 0 stopped: 2 graphDriverName: overlay graphOptions: overlay.imagestore: /srv/.podman/images overlay.mountopt: nodev graphRoot: /srv/.podman/storage graphRootAllocated: 59237908480 graphRootUsed: 905797632 graphStatus: Backing Filesystem: extfs Native Overlay Diff: "true" Supports d_type: "true" Supports shifting: "false" Supports volatile: "true" Using metacopy: "false" imageCopyTmpDir: /var/tmp imageStore: number: 4 runRoot: /run/containers/storage transientStore: false volumePath: /srv/.podman/storage/volumes version: APIVersion: 4.8.0 Built: 1706521279 BuiltTime: Mon Jan 29 09:41:19 2024 GitCommit: "" GoVersion: go1.21.5 Os: linux OsArch: linux/amd64 Version: 4.8.0 ```

root@OpenWrt:/srv/.podman/scripts# ls /sys/fs/cgroup/
cgroup.controllers      cgroup.stat             cpuset.cpus.effective   memory.stat
cgroup.max.depth        cgroup.subtree_control  cpuset.mems.effective   services
cgroup.max.descendants  cgroup.threads          io.stat
cgroup.procs            cpu.stat                libpod_parent

[brada4]

Try runc instead, no telling where you got your both containers from. Also check if same config starts on ubuntu or so.

[djarbz]

What do you mean by both my containers? It's literally the hello world container to test if containers are working.

Anyways, I did install runc and here is the result of the test.

root@OpenWrt:~# podman run --rm --runtime=runc hello-world:latest
WARN[0000] Failed to add conmon to cgroupfs sandbox cgroup: creating cgroup path /libpod_parent/conmon: write /sys/fs/cgroup/cgroup.subtree_control: invalid argument

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

root@OpenWrt:~# podman run --rm --runtime=crun hello-world:latest
WARN[0000] Failed to add conmon to cgroupfs sandbox cgroup: creating cgroup path /libpod_parent/conmon: write /sys/fs/cgroup/cgroup.subtree_control: invalid argument
Error: OCI runtime error: crun: writing file `/sys/fs/cgroup/cgroup.subtree_control`: Invalid argument

So it looks like runc "works", but not fully.

[brada4]

The subtree_control node adds/removes restriction controllers, can you show what is in that file? Hard to tell if one was intended to be added or removed from short fault message.

[djarbz]

root@OpenWrt:~# cat /sys/fs/cgroup/cgroup.subtree_control
cpu io memory pids rdma

[brada4]

Try to enable all available controllers:

cat ...../cgroup.controllers > ......./subtree_control

[djarbz]

Hmm...

root@OpenWrt:/sys/fs/cgroup# cat cgroup.controllers > cgroup.subtree_control
cat: write error: Invalid argument

root@OpenWrt:~# cat /sys/fs/cgroup/cgroup.controllers
cpuset cpu io memory pids rdma

Only adds cpuset if it had worked.

[brada4]

try echo "+cpuset" > subtree_control

It seems some kernel option is missing which is essential for crun while not important for runc. I tried docker/hello-world , and it just clones full resource controller tree, and debian does not offer a way to enable/disable them on the spot. Cant help with guesswork (or youconfigure runtime to use runc)

[djarbz]

No luck.

root@OpenWrt:~# echo  "+cpuset" > /sys/fs/cgroup/cgroup.subtree_control
ash: write error: Invalid argument

I'll try reaching out to the Podman team and see if they have any thoughts.