<cgroupfs-mount> systemd mount missing, lxc container will not start

[iamhermes]

Environment: (amd64)

Description: cgroupfs-mount scripts seems not updated anymore is it possible to include the patch to mount systemd? lxc can not run latest debian and ubuntu images

https://github.com/tianon/cgroupfs-mount/pull/16/commits/794b82755846de2ec94f20bea79772378809d8f8 cgroupfs-mount

root@OpenWrt:~# lxc-start debian_buster -F
Failed to mount cgroup at /sys/fs/cgroup/systemd: Operation not permitted
[!!!!!!] Failed to mount API filesystems.
Exiting PID 1...

temporary fix:

mkdir -p /sys/fs/cgroup/systemd
mount -t cgroup -o none,name=systemd systemd /sys/fs/cgroup/systemd

[brada4]

You need to tame appetite of guest systemd, no "update" can fix guest contents. https://wiki.debian.org/LXC/CGroupV2

[tyuxar]

Environment: (amd64)

Description: cgroupfs-mount scripts seems not updated anymore is it possible to include the patch to mount systemd? lxc can not run latest debian and ubuntu images

tianon/cgroupfs-mount@794b827 cgroupfs-mount

Thanks for the links, I had precisely the same problem with the same error message on 22.03.2, and it fixed my sole Ubuntu container.

The icing on the cake: I changed nor the host, neither the container - including package updates. It just ceased to work after a shutdown/maintenance. Nice, deterministic IT.

[twisteroidambassador]

I have found that on OpenWRT 23.05.3, without installing cgroupfs-mount, a Debian Bookworm container can start, but with cgroupfs-mount installed the container fails to start. Maybe this package is no longer required.

Without cgroupfs-mount, the output of lxc-checkconfig is this:

LXC version 5.0.3

--- Namespaces ---
Namespaces: enabled
Utsname namespace: enabled
Ipc namespace: enabled
Pid namespace: enabled
User namespace: enabled
newuidmap is not installed
newgidmap is not installed
Network namespace: enabled

--- Control groups ---
Cgroups: enabled
Cgroup namespace: enabled
Cgroup v1 mount points: 
Cgroup v2 mount points: 
 - /sys/fs/cgroup
Cgroup device: missing
Cgroup sched: enabled
Cgroup cpu account: enabled
Cgroup memory controller: enabled
Cgroup cpuset: enabled

--- Misc ---
Veth pair device: enabled, loaded
Macvlan: enabled, not loaded
Vlan: enabled, not loaded
Bridges: enabled, not loaded
Advanced netfilter: enabled, loaded
CONFIG_IP_NF_TARGET_MASQUERADE: missing
CONFIG_IP6_NF_TARGET_MASQUERADE: missing
CONFIG_NETFILTER_XT_TARGET_CHECKSUM: enabled, not loaded
CONFIG_NETFILTER_XT_MATCH_COMMENT: enabled, not loaded
FUSE (for use with lxcfs): enabled, not loaded

--- Checkpoint/Restore ---
checkpoint restore: missing
CONFIG_FHANDLE: enabled
CONFIG_EVENTFD: enabled
CONFIG_EPOLL: enabled
CONFIG_UNIX_DIAG: missing
CONFIG_INET_DIAG: enabled
CONFIG_PACKET_DIAG: missing
CONFIG_NETLINK_DIAG: enabled
File capabilities: enabled

Note : Before booting a new kernel, you can check its configuration
usage : CONFIG=/path/to/config /usr/bin/lxc-checkconfig

When starting a Debian Bookworm container, this error message is printed twice, but otherwise the container start successfully:

lxc-start: debian2: ../src/lxc/cgroups/cgfsng.c: __cgfsng_delegate_controllers: 3341 Invalid argument - Could not enable "+cpuset +cpu +io +memory +pids +rdma" controllers in the unified cgroup 7

With cgroupfs-mount installed, the output of lxc-checkconfig is:

LXC version 5.0.3        

--- Namespaces ---    
Namespaces: enabled     
Utsname namespace: enabled           
Ipc namespace: enabled               
Pid namespace: enabled                
User namespace: enabled
newuidmap is not installed
newgidmap is not installed 
Network namespace: enabled       

--- Control groups ---
Cgroups: enabled
Cgroup namespace: enabled        
Cgroup v1 mount points:     
 - /sys/fs/cgroup/cpuset 
 - /sys/fs/cgroup/cpu       
 - /sys/fs/cgroup/cpuacct          
 - /sys/fs/cgroup/blkio                
 - /sys/fs/cgroup/memory                
 - /sys/fs/cgroup/net_cls                                                                          
 - /sys/fs/cgroup/pids                                                                             
 - /sys/fs/cgroup/rdma                        
Cgroup v2 mount points: 
Cgroup v1 systemd controller: missing
Cgroup v1 freezer controller: missing
Cgroup v1 clone_children flag: enabled
Cgroup device: missing 
Cgroup sched: enabled
Cgroup cpu account: enabled
Cgroup memory controller: enabled
Cgroup cpuset: enabled     

--- Misc ---    
Veth pair device: enabled, loaded
Macvlan: enabled, not loaded
Vlan: enabled, not loaded
Bridges: enabled, not loaded
Advanced netfilter: enabled, loaded
CONFIG_IP_NF_TARGET_MASQUERADE: missing
CONFIG_IP6_NF_TARGET_MASQUERADE: missing
CONFIG_NETFILTER_XT_TARGET_CHECKSUM: enabled, not loaded                                           
CONFIG_NETFILTER_XT_MATCH_COMMENT: enabled, not loaded                                             
FUSE (for use with lxcfs): enabled, not loaded

--- Checkpoint/Restore ---           
checkpoint restore: missing          
CONFIG_FHANDLE: enabled               
CONFIG_EVENTFD: enabled
CONFIG_EPOLL: enabled
CONFIG_UNIX_DIAG: missing  
CONFIG_INET_DIAG: enabled        
CONFIG_PACKET_DIAG: missing
CONFIG_NETLINK_DIAG: enabled                     
File capabilities: enabled

Note : Before booting a new kernel, you can check its configuration                                
usage : CONFIG=/path/to/config /usr/bin/lxc-checkconfig                                            

And now a Debian Bookworm container prints the following error message and exits:

Failed to mount cgroup at /sys/fs/cgroup/systemd: Operation not permitted
[!!!!!!] Failed to mount API filesystems.
Exiting PID 1...

[kvic-z]

@twisteroidambassador Excellent finding.

Uninstalling 'cgroupfs-mount' resolves a similar and related issue in guest systemd for me.

Running LXC on OpenWrt host (Banana Pi BPI-R4) with guest ArchLinux for ARM.