Unable to start rootless container under Podman when in WSL

[Pvlerick]

Although #719 has been fixed in #2370 - I checked it using versions of Youki before and after under Ubuntu; the issue still remains when running in WSL.

podman run -it --rm --runtime youkidev alpine:latest
Error: ERROR libcontainer::process::container_intermediate_process: failed to add task to cgroup pid=Pid(43606) err=V1(WrappedIo(CreateDir { err: Os { code: 13, kind: PermissionDenied, message: "Permission denied" }, path: "/sys/fs/cgroup/cpuacct/3652f57e7e8a876e538ca3d186cad1bfb0cdb8554171cab8a39d7c3c58c29110" })) init=true: OCI permission denied

youkidev being defined in /usr/share/containers/containers.conf as pointing to the latest build of youki from master

./youki info
Version           0.3.0
Commit            97488350
Kernel-Release    5.15.90.1-microsoft-standard-WSL2
Kernel-Version    #1 SMP Fri Jan 27 02:56:13 UTC 2023
Architecture      x86_64
Operating System  Debian GNU/Linux 11 (bullseye)
Cores             8
Total Memory      15859
Cgroup setup      hybrid
Cgroup mounts
Capabilities
CAP_BPF           available
CAP_PERFMON       available
CAP_CHECKPOINT_RESTORE available

I would be interested to work on this myself, but I'm ramping up on Rust right now. Any pointers to where to start appreciated, otherwise I'll just dig and dig more.

[YJDoc2]

Hey @Pvlerick ,thanks for reporting this. Can I ask you to check running youki 0.2.0 or older version, and see what is the error message? I think under wsl, we do not have systemd, so the changes in #2370 would not have any relation with this. If in youki 0.2.0 the error message is the same, then the wsl setup uses cgroups v2 manager and not systemd cgroups manager. If that is the case, then the cgroups v2 manager would be a good place to start looking.

[Pvlerick]

Using 0.2.0:

./youki info
Version           0.2.0
Commit            ed75057
Kernel-Release    5.15.90.1-microsoft-standard-WSL2
Kernel-Version    #1 SMP Fri Jan 27 02:56:13 UTC 2023
Architecture      x86_64
Operating System  Debian GNU/Linux 11 (bullseye)
Cores             8
Total Memory      15859
Cgroup setup      hybrid
Cgroup mounts
Capabilities
CAP_BPF           available
CAP_PERFMON       available
CAP_CHECKPOINT_RESTORE available

podman run -it --rm --runtime /tmp/youki-0.2.0/youki_0_2_0_linux/youki-0.2.0/youki alpine:latest
Error: IO error

               Caused by:
                             Permission denied (os error 13)
                                                            ERRO[0000] Error removing container 80e6b0528249d12fa0ebbc16d3cf56a0a0d3e6c421270ec4611e455cc7d4782b from runtime after creation failed
Error: IO error

Caused by:
    Permission denied (os error 13)
Error:     Permission denied (os error 13): OCI permission denied

I'm running systemd in WSL:

systemctl --version
systemd 247 (247.3-7+deb11u4)
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +ZSTD +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=unified

If that gives you any idea, let me know :-)

In any event, I'll start looking around cgroups manager.

[utam0k]

@Pvlerick May I ask you to use the latest version(v0.3.0)?

[Pvlerick]

@utam0k that's what I did initially https://github.com/containers/youki/issues/2484#issue-1965453915

I probably didn't explain myself very well; I tried under WSL with 0.3.0, 0.2.0 and the latest from master and I have similar errors. I tested under Ubuntu "bare metal" (no vm) and it works like a charm since 0.3.0.

[Pvlerick]

Turns out it's cgroup manager V1 that is used in the end:

youki[13853]: error in executing command: failed to receive. "waiting for intermediate process". BrokenChannel
                                         Caused by:
                                             channel connection broken
...
youki[13310]: cgroup manager V1 will be used
youki[13853]: failed to run container process
youki[13855]: failed to run intermediate process
youki[13855]: failed to add task to cgroup

Hacking in the code to force it to use V2 or systemd gets the same error, so I think there might be something other at play here. The plot thickens! :-)

[YJDoc2]

Hey @Pvlerick Thanks for the debugging! There is also https://github.com/containers/youki/issues/2528 which is related to a bug in cgroups v1 creation. Not sure if that would be the root cause, but can you try this PR branch https://github.com/containers/youki/pull/2532 which aims to fix that? If it does, then we might need to do something similar with cgroups v2

On the other hand, I'm not even sure if for rootless, we can use v1 or v2 directly... As we delegate the cgroup management to systemd when running rootless (as non-root, we don't have perms for manipulating cgroups) we need systemd to handle it for us...

[Pvlerick]

Under Ubuntu (not in WSL), it's indeed systemd that is used. I'll checkout that branch and see what happens, thanks for the pointers.

[Pvlerick]

Same error with https://github.com/containers/youki/pull/2532, unfortunately.

[YJDoc2]

@Pvlerick Hey, sorry I couldn't be of much help here, did you find out anything? I currently don't have access to WSL, so cannot re-produce this locally, but for rootless, the systemd cgroups manager must be used.

[Pvlerick]

@YJDoc2 no problem, I still have this issue on my radar but I don't have time to work on it right now. I'll eventually get to it, don't worry :-)