Unable to start rootless container under Podman when Youki used as runtime

[ghost]

When I try to start a rootless podman container with Youki used as runtime, it tries to create a /run/youki directory, which, as not running as root, fails with permission denied.

Youki version: 0.0.2 (commit: 0.0.2-0-73dc75c)

% podman create --runtime /home/ondra/.cargo/bin/youki --name fedora fedora
128fed7131b2a59030b9691108977dd0b4fc4c1fcc6b064c6f4c16d615637b96

% podman start fedora        
Error: failed to create directory /run/youki

Caused by:
    Permission denied (os error 13)
ERRO[0000] Error removing container 128fed7131b2a59030b9691108977dd0b4fc4c1fcc6b064c6f4c16d615637b96 from runtime after creation failed 
Error: unable to start container "128fed7131b2a59030b9691108977dd0b4fc4c1fcc6b064c6f4c16d615637b96":     Permission denied (os error 13): OCI permission denied

[utam0k]

@ondra05 Thanks for your report. But, I couldn't reproduce this error in my environment.

$ podman create --runtime /home/utam0k/ghq/github.com/utam0k/youki/youki --name fedora fedora
WARN[0000] "/" is not a shared mount, this could cause issues or missing mounts with rootless containers
Resolved "fedora" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull registry.fedoraproject.org/fedora:latest...
Getting image source signatures
Copying blob c6183d119aa8 done
Copying config e417cd49a8 done
Writing manifest to image destination
Storing signatures
e60cafa77762fe924e97d1d9e9ce2cdf75315517ed511b1eb53e0058ba0bfbe0

Please tell me the result of youki info?

[ghost]

Sure thing!

Version           0.0.2
Commit            73dc75c
Kernel-Release    5.16.8-200.fc35.x86_64
Kernel-Version    #1 SMP PREEMPT Tue Feb 8 20:58:59 UTC 2022
Architecture      x86_64
Operating System  Fedora Linux 35 (Workstation Edition)
Cores             8
Total Memory      15699
Cgroup setup      unified
Cgroup mounts
  unified         /sys/fs/cgroup
[WARN crates/libcgroups/src/v2/util.rs:41] 2022-02-20T10:19:40.888719702+01:00 Controller misc is not yet implemented.
CGroup v2 controllers
  cpu             attached
  cpuset          attached
  hugetlb         attached
  io              attached
  memory          attached
  pids            attached
  device          attached
Namespaces        enabled
  mount           enabled
  uts             enabled
  ipc             enabled
  user            enabled
  pid             enabled
  network         enabled
  cgroup          enabled

Well, the issue happnens when I tried to start the container, creation of the container worked just fine.

[riyad]

I have the same issue with Podman 3.4.2 and Youki 0.0.3 on Ubuntu 21.10:

$ podman run --runtime youki --rm -it ubuntu
Error: failed to create directory /run/youki

                                            Caused by:
                                                          Permission denied (os error 13)
                                                                                         ERRO[0000] Error removing container 98fd9fee35115f1f92adff042897f827271ca4bcf6e607c6d4666a4860caef65 from runtime after creation failed
Error:     Permission denied (os error 13): OCI permission denied

$ youki info
Version           0.0.3
Commit            541bf4a
Kernel-Release    5.13.0-39-generic
Kernel-Version    #44-Ubuntu SMP Thu Mar 24 15:35:05 UTC 2022
Architecture      x86_64
Operating System  Ubuntu 21.10
Cores             4
Total Memory      15767
Cgroup setup      unified
Cgroup mounts
  unified         /sys/fs/cgroup
[WARN libcgroups/src/v2/util.rs:41] 2022-04-14T16:09:49.230724310+02:00 Controller rdma is not yet implemented.
[WARN libcgroups/src/v2/util.rs:41] 2022-04-14T16:09:49.230868203+02:00 Controller misc is not yet implemented.
CGroup v2 controllers
  cpu             attached
  cpuset          attached
  hugetlb         attached
  io              attached
  memory          attached
  pids            attached
  device          attached
Namespaces        enabled
  mount           enabled
  uts             enabled
  ipc             enabled
  user            enabled
  pid             enabled
  network         enabled
  cgroup          enabled

[stefins]

I'm having the same issue with Ubuntu 20.04.1 !

[orimanabu]

Hi,

I encountered the same issue on Fedora 36 with podman v4.1.0. It seems that mkdir /etc/youki got EACCESS.

$ podman --runtime=$(pwd)/youki run --rm --name test hello-world 
[DEBUG crates/youki/src/main.rs:92] 2022-05-17T17:36:43.548188063+09:00 started by user 0 with ArgsOs { inner: ["/home/ori/devel/src/github.com/containers/youki/work/youki", "delete", "--force", "8d38990d6a0ded9fb3797bb996a7b5ec43a9b411a65fec108034a1e83c387c64"] }
Error: failed to create directory /run/youki

Caused by:
    Permission denied (os error 13)
ERRO[0000] Removing container 8d38990d6a0ded9fb3797bb996a7b5ec43a9b411a65fec108034a1e83c387c64 from runtime after creation failed 
Error: /home/ori/devel/src/github.com/containers/youki/work/youki:     Permission denied (os error 13): OCI permission denied

$ ./youki info
[DEBUG crates/youki/src/main.rs:92] 2022-05-17T17:36:07.126714051+09:00 started by user 1000 with ArgsOs { inner: ["./youki", "info"] }
Version           0.0.3
Commit            05ce5c6
Kernel-Release    5.17.7-300.fc36.x86_64
Kernel-Version    #1 SMP PREEMPT Thu May 12 14:56:44 UTC 2022
Architecture      x86_64
Operating System  Fedora Linux 36 (Workstation Edition)
Cores             12
Total Memory      64043
Cgroup setup      unified
Cgroup mounts
  unified         /sys/fs/cgroup
[WARN crates/libcgroups/src/v2/util.rs:41] 2022-05-17T17:36:07.156835792+09:00 Controller misc is not yet implemented.
CGroup v2 controllers
  cpu             attached
  cpuset          attached
  hugetlb         attached
  io              attached
  memory          attached
  pids            attached
  device          attached
Namespaces        enabled
  mount           enabled
  uts             enabled
  ipc             enabled
  user            enabled
  pid             enabled
  network         enabled
  cgroup          enabled

[riyad]

Hmm .. I tried some "dbg!-ing" and found out that the check for rootless_required() in determine_root_path() behaves weirdly.

When running e.g. ./youki info it returns true (UID is 1000):

$ ./youki info
[DEBUG crates/youki/src/main.rs:92] 2022-05-18T23:21:42.556071673+02:00 started by user 1000 with ArgsOs { inner: ["./youki", "info"] }
[crates/youki/src/main.rs:133] getuid().as_raw() = 1000
[crates/youki/src/main.rs:135] root_path = None
[crates/youki/src/main.rs:143] rootless_required() = true
[crates/youki/src/main.rs:151] Path::new(&path).join("youki") = "/run/user/1000/youki"
Version           0.0.3
Commit            05ce5c6
Kernel-Release    5.15.0-30-generic
Kernel-Version    #31-Ubuntu SMP Thu May 5 10:00:34 UTC 2022
Architecture      x86_64
Operating System  Ubuntu 22.04 LTS
Cores             4
Total Memory      15767
Cgroup setup      unified
Cgroup mounts
  unified         /sys/fs/cgroup
[WARN crates/libcgroups/src/v2/util.rs:41] 2022-05-18T23:21:42.621769976+02:00 Controller rdma is not yet implemented.
[WARN crates/libcgroups/src/v2/util.rs:41] 2022-05-18T23:21:42.621803261+02:00 Controller misc is not yet implemented.
CGroup v2 controllers
  cpu             attached
  cpuset          attached
  hugetlb         attached
  io              attached
  memory          attached
  pids            attached
  device          attached
Namespaces        enabled
  mount           enabled
  uts             enabled
  ipc             enabled
  user            enabled
  pid             enabled
  network         enabled
  cgroup          enabled

But when running it though podman (as a non-root user) it returns false (UID is 0, but should be 1000) :raised_eyebrow: :

$ podman run --runtime $PWD/youki --rm hello-world
[DEBUG crates/youki/src/main.rs:92] 2022-05-18T23:22:05.152662158+02:00 started by user 0 with ArgsOs { inner: ["/home/riyad/src/youki/youki", "delete", "--force", "cc560217ade908d275148f4b162e35e6ac668106460a232853afe0fca505f504"] }
[crates/youki/src/main.rs:133] getuid().as_raw() = 0
[crates/youki/src/main.rs:135] root_path = None
[crates/youki/src/main.rs:143] rootless_required() = false
[crates/youki/src/main.rs:144] get_default_not_rootless_path() = "/run/youki"
Error: failed to create directory /run/youki

Caused by:
    Permission denied (os error 13)
time="2022-05-18T23:22:05+02:00" level=error msg="Error removing container cc560217ade908d275148f4b162e35e6ac668106460a232853afe0fca505f504 from runtime after creation failed"
Error:     Permission denied (os error 13): OCI permission denied

[utam0k]

Thanks @stefins @orimanabu @riyad!

[utam0k]

Hi, @Furisto Do you have time to check this because you assigned it by yourself. But if you don't have time, please let me. I'll check.

[Furisto]

Hey, sorry for the late reply. I will check this.

[utam0k]

@Furisto No problem. Thanks ❤️

[terinjokes]

I wrapped youki in a scrip that set the environment variable YOUKI_USE_ROOTLESS=true, which allowed this to progress farther. It then complained "rootless container requires valid user namespace definition", which I seemed to have resolved by passing --userns=auto to podman run.

This results in another failure, where youki attempts to connect to the system dbus (instead of the session dbus). It looks like it has the same effective uid check as determine_root_path:

https://github.com/containers/youki/blob/a72a33b3e946a33d83d555f210e89401ef57cef6/crates/libcgroups/src/common.rs#L230

[Furisto]

The behavior I am seeing is weird. Podman is not started as root, but youki seems to be. If I create a rootless container with youki without podman it works and I am seeing the correct (unprivileged) uid, so this does not seem to be a bug with the detection of the user in youki.

The config.json that podman provides to youki does not contain a user namespace either, which is further evidence that podman is not trying to create a rootless container. Youki also has no problems creating the /run/youki directory when started standalone, only when started by podman. Maybe podman has a security policy that prevents it? Will look further into this.

[orimanabu]

It seems to me that checking nix::unistd::geteuid().is_root() only is not enough to determine rootless. Maybe we should consult /proc/self/uid_map also?

[utam0k]

@orimanabu Thanks. @Furisto WDYT? If you don't have time, please tell me, and I can take over

[orimanabu]

My previous comment was just a guess, but I did some more research.

When in rootless mode, podman creates user namespace in the very early phase. (userns is created in PersistentPreRunE of rootCmd.) So When youki is invoked by podman, youki runs as root, uid mapped in the userns.

runc and crun seem to check if /proc/self/uid_map contains "4294967295" whether they run in rootless. Should youki do the same, maybe in rootless_required() ?

[utam0k]

@orimanabu Perfect 💯 I think your investigation is right.

[YJDoc2]

@ondra05 this took a long time, but with current main, this should be resolved. Can you verify once, otherwise I'll close this in few days, Thanks!

[YJDoc2]

Going ahead and closing this. Thank you everyone!