net_cls usage confuses systemctl status

[codicodi]

Issue report

Operating system: Arch Linux

App version: 2021.6

Issue description

After updating my system to systemd v250 systemctl status started showing net_cls tree instead of systemd's own cgroup tree (see https://github.com/systemd/systemd/issues/21945). systemd developers claim this is not a bug, as usage of cgroup v1 controllers on systems using unified cgroup hierarchy (AFAIK most modern distributions) is not supported.

I've worked around this issue locally by creating an override for mullvad-daemon.service:

[Service]
Environment=TALPID_NET_CLS_MOUNT_DIR='/run/mullvad-exclude'

Mounting the controller in a nonstandard directory seems to prevent the confusion from systemd's side (at least for now).

While Arch Linux is not a supported system, the problem is likely to occur on supported systems as well, as soon as they start using systemd >=250.

[pinkisemils]

Thank you for reporting this, we're somewhat aware of the issue but it's good to know that we'll have to fix this in some upcoming release. I don't think mounting the cgroup v1 controller in a non-standard directory is a nice fix for this, as there will probably be other users of the v1 cgroup for a while and we don't want to interfere with those.

[jessicarod7]

Confirming that this now affects a supported OS, and specifically breaks running containers with systemd (see containers/podman#18551).

OS: Fedora 38 (GNOMe 44.1) Kernel: 6.3.5-200.fc38.x86_64 systemd: 253.4 App version: 2023.3

[manueldeljesus]

I have also found this issue. I have tried the solution proposed in this thread, which is similar to the one proposed here, mounting net_cls in another directory and edit the Mullvad service files to point there, but it does not seem to work. The only way to have systemd working properly is to completely uninstall or disable Mullvad.

In my case, it is precluding me from launching Podman services.

OS: Fedora 39 (Gnome 45.1) Kernel: Linux 6.6.2-201.fc39.x86_64 systemd: 254.7-1.fc39 App version: 2023.5

[icyqar]

Same issue here

Podman containers scheduled to start on boot via systemd don't work(even with client closed). Wouldn't start despite many reboots and many different option combinations tried in the client app

Had to uninstall the client and use OpenVPN config files instead and then systemd Podman services worked as expected.

No fix coming from Podman team since this is Mullvad's app that results in an "incorrect system setup" https://github.com/containers/podman/issues/18103#issuecomment-1582158201

Host: uname -a Linux archlinux 6.7.3-arch1-1 #1 SMP PREEMPT_DYNAMIC Thu, 01 Feb 2024 10:30:35 +0000 x86_64 GNU/Linux

# Systemd service unit file for the Mullvad VPN daemon

[Unit]
Description=Mullvad VPN daemon
Before=network-online.target
After=mullvad-early-boot-blocking.service NetworkManager.service systemd-resolved.service

StartLimitBurst=5
StartLimitIntervalSec=20
RequiresMountsFor=/opt/Mullvad\x20VPN/resources/

[Service]
Restart=always
RestartSec=1
ExecStart=/usr/bin/mullvad-daemon -v --disable-stdout-timestamps
Environment="MULLVAD_RESOURCE_DIR=/opt/Mullvad VPN/resources/"
Environment="TALPID_NET_CLS_MOUNT_DIR=/opt/net-cls-v1/"

[Install]
WantedBy=multi-user.target

[tiritto]

Same issue as above. Hours of trying to figure out what's wrong just to find out that Mullvad was the culprit. To make the matter worse, the Mullvad client wasn't even running when the issue happened, making it even harder to connect the dots. As soon as I removed Mullvad and rebooted my OS, my problem instantly disappeared.

[OverkillGuy]

Bump: Running a podman container as systemd job (in my user slice), but the container keeps crashing within a second of start (despite podman run locally working fine outside systemd) which I traced to the above bug report in podman, and down to Mullvad.

I love you Mullvad, but this is making my computer worse because of what (based on podman devs) is a "broken app setup".

Keen to hear more about this, or a workaround for the podman-in-systemd issue.

[dawidpotocki]

I lost 2 days trying to debug the issue with crashing Podman services.

Seeing that it's unfixed for over 2 years is very disappointing.

This combined with the removal of port forwarding and me recently sometimes experiencing kbps speeds in a certain region is really making me reconsider my VPN provider.

[e3dio]

Can confirm Mullvad client breaks SystemD, this needs to be fixed https://github.com/containers/podman/issues/18103#issuecomment-1563395872

[atagen]

TALPID_NET_CLS_MOUNT_DIR workaround not working here either.

kind of an insane chase to track it down to this issue too, so I'm just gonna go ahead and say: failed systemd user service podman quadlet sd_notify conmon handover new pid seo spam rootless new main PID does not belong to service, refusing.