Failure to start systemd containers with cgroup v1 with dockerd

[cybrknght]

Rancher Desktop Version

0.7.0

Rancher Desktop K8s Version

1.21.7

What operating system are you using?

macOS

Operating System / Build Version

MacOS BigSur 11.6.2

What CPU architecture are you using?

x64

Windows User Only

No response

Actual Behavior

When I run a container, centos7 in this case, the container fails to run systemd.

Steps to Reproduce

Following the instructions at https://hub.docker.com/_/centos I've created a Centos Systemd ready container with the following Dockerfile:

FROM centos:7
ENV container docker
RUN (cd /lib/systemd/system/sysinit.target.wants/; for i in *; do [ $i == \
systemd-tmpfiles-setup.service ] || rm -f $i; done); \
rm -f /lib/systemd/system/multi-user.target.wants/*;\
rm -f /etc/systemd/system/*.wants/*;\
rm -f /lib/systemd/system/local-fs.target.wants/*; \
rm -f /lib/systemd/system/sockets.target.wants/*udev*; \
rm -f /lib/systemd/system/sockets.target.wants/*initctl*; \
rm -f /lib/systemd/system/basic.target.wants/*;\
rm -f /lib/systemd/system/anaconda.target.wants/*;
VOLUME [ "/sys/fs/cgroup" ]
CMD ["/usr/sbin/init"]

Build the dockerfile:

docker build --rm -t local/c7-systemd .

Run the container:

 docker run --rm  -ti --privileged -v /sys/fs/cgroup:/sys/fs/cgroup local/c7-systemd

Result

When I run the container, I get the following error:

$ docker run --rm  -ti --privileged -v /sys/fs/cgroup:/sys/fs/cgroup local/c7-systemd
Failed to insert module 'autofs4'
Failed to mount cgroup at /sys/fs/cgroup/systemd: Operation not permitted
systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN)
Detected virtualization docker.
Detected architecture arm64.

Welcome to CentOS Linux 7 (AltArch)!

Set hostname to <6870484659ce>.
Initializing machine ID from random generator.
Cannot determine cgroup we are running in: No such file or directory
Failed to allocate manager object: No such file or directory
[!!!!!!] Failed to allocate manager object, freezing.

Expected Behavior

The container should run and start with systemd running:

$ docker run --rm  -ti --privileged -v /sys/fs/cgroup:/sys/fs/cgroup local/c7-systemd
Failed to insert module 'autofs4'
systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN)
Detected virtualization docker.
Detected architecture x86-64.

Welcome to CentOS Linux 7 (Core)!

Set hostname to <3c882f4c07f9>.
Initializing machine ID from random generator.
[  OK  ] Reached target Paths.
[  OK  ] Reached target Swap.
[  OK  ] Reached target Local File Systems.
[  OK  ] Created slice Root Slice.
[  OK  ] Listening on Delayed Shutdown Socket.
[  OK  ] Listening on Journal Socket.
[  OK  ] Created slice System Slice.
[  OK  ] Reached target Slices.
         Starting Create Volatile Files and Directories...
         Starting Journal Service...
[  OK  ] Started Create Volatile Files and Directories.
[ INFO ] Update UTMP about System Boot/Shutdown is not active.
[DEPEND] Dependency failed for Update UTMP about System Runlevel Changes.
Job systemd-update-utmp-runlevel.service/start failed with result 'dependency'.
[  OK  ] Started Journal Service.
[  OK  ] Reached target System Initialization.
[  OK  ] Started Daily Cleanup of Temporary Directories.
[  OK  ] Reached target Timers.
[  OK  ] Listening on D-Bus System Message Bus Socket.
[  OK  ] Reached target Sockets.
[  OK  ] Reached target Basic System.
[  OK  ] Reached target Multi-User System.

Additional Information

Seems to be related to this issue https://github.com/docker/for-mac/issues/6073

[gunamata]

I could reproduce the issue with Rancher Desktop 1.5.1, I see a different error message though

>docker run -ti -v /sys/fs/cgroup:/sys/fs/cgroup:ro -p 80:80 local/c7-systemd-httpd
Failed to mount tmpfs at /run: Operation not permitted
Failed to mount cgroup at /sys/fs/cgroup/systemd: Operation not permitted
[!!!!!!] Failed to mount API filesystems, freezing.

[medoni]

Any update on this? Still failing on RD 1.15.1 when creating KinD cluster.

[jandubois]

Any update on this? Still failing on RD 1.15.1 when creating KinD cluster.

Not sure what this has to do with KinD, but that has been working for a couple releases already:

$ kind create cluster
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.31.0) 🖼
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Thanks for using kind! 😊

There may be still issues on Windows, unrelated to cgroup, but this is a macOS issue...

[jandubois]

As for the original issue in this report, I think this will not work as long as Rancher Desktop uses an Alpine VM that uses OpenRC. Mounting the cgroup from it into a container running systemd is not going to work.

[jandubois]

I just tried running the sample in a docker-rootful VM (based on Ubuntu) hosted by Lima on macOS, and it still fails similarly. Maybe centos7 is too old now; idk:

$ docker run --rm  -ti --privileged -v /sys/fs/cgroup:/sys/fs/cgroup local/c7-systemd
Failed to mount cgroup at /sys/fs/cgroup/systemd: Operation not permitted
systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN)
Detected virtualization docker.
Detected architecture x86-64.

Welcome to CentOS Linux 7 (Core)!

Set hostname to <a1503cfcac12>.
Initializing machine ID from random generator.
Cannot determine cgroup we are running in: No such file or directory
Failed to allocate manager object: No such file or directory
[!!!!!!] Failed to allocate manager object, freezing.

It is unlikely that we will spend any effort on figuring this out, so I will close this issue as unsupported.