containers / podman

Podman: A tool for managing OCI containers and pods.
https://podman.io
Apache License 2.0
23.64k stars 2.41k forks source link

podman run with pod and uidmap: mount mqueue not permitted #8970

Closed matpen closed 3 years ago

matpen commented 3 years ago

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

I am filing this issue to re-open the discussion on https://github.com/containers/podman/issues/6843. I just tested with the latest versions of the involved software, and it does not seem to work.

# The host is Ubuntu 18.04 on a physical laptop
matteo@matteo-laptop:~$ cat /etc/issue
Ubuntu 18.04.4 LTS

# Kernel is the latest available through HWE
matteo@matteo-laptop:~$ uname -r
5.4.0-60-generic

# Using the latest crun
matteo@matteo-laptop:~$ crun --version
crun version 0.16
commit: 91ec195708efb8eed1699d59cd0369d639d8a7a8
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL

# Using the latest podman
matteo@matteo-laptop:~$ podman --version
podman version 2.2.1

# Create a pod
matteo@matteo-laptop:~$ sudo podman pod create \
    --runtime crun \
    --name test_pod

# Create a uid-mapped container and join the pod
matteo@matteo-laptop:~$ sudo podman run \
    --rm \
    --privileged \
    --runtime crun \
    --pod test_pod \
    --uidmap 0:1000:1000 \
    --uidmap 1000:100000:1 \
    "docker.io/library/ubuntu:18.04" whoami
Error: mount 'mqueue' to '/dev/mqueue': Operation not permitted: OCI runtime permission denied error

Note that:

  1. I am using kernel 5.4, so this does not apply;
  2. I am using crun 0.16, so this does not apply.
rhatdan commented 3 years ago

Could this be apparmor blocking it?

matpen commented 3 years ago

@rhatdan Thank you for the suggestion. I tend to exclude this hypothesis based on the following tests:

  1. While running the commands described above, I see no AppArmor entries neither in the /var/log/kern.log nor in the kernel ring buffer via dmesg;
  2. I tested the same in a VirtualBox VM where I disabled AppArmor, and got the same result
matteo@ubuntu-vbox:~$ sudo systemctl stop apparmor
matteo@ubuntu-vbox:~$ sudo systemctl disable apparmor
Synchronizing state of apparmor.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install disable apparmor
matteo@ubuntu-vbox:~$ sudo reboot

matteo@ubuntu-vbox:~$ sudo service apparmor status
● apparmor.service - AppArmor initialization
   Loaded: loaded (/lib/systemd/system/apparmor.service; disabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:apparmor(7)
           http://wiki.apparmor.net/
matteo@ubuntu-vbox:~$ sudo aa-status
apparmor module is loaded.
0 profiles are loaded.
0 profiles are in enforce mode.
0 profiles are in complain mode.
0 processes have profiles defined.
0 processes are in enforce mode.
0 processes are in complain mode.
0 processes are unconfined but have a profile defined.

matteo@matteo-vbox:~$ sudo podman run \
    --rm \
    --privileged \
    --runtime crun \
    --pod test_pod \
    --uidmap 0:1000:1000 \
    --uidmap 1000:100000:1 \
    "docker.io/library/ubuntu:18.04" whoami
Error: mount 'mqueue' to '/dev/mqueue': Operation not permitted: OCI permission denied
rhatdan commented 3 years ago

Does this work if you don't deal with usernamespace?

rhatdan commented 3 years ago
 sudo podman run \
    --rm \
    --privileged \
    --runtime crun \
    --pod test_pod \
    "docker.io/library/ubuntu:18.04" whoami
rhatdan commented 3 years ago

Ok looking a little deeper, I think the issue is doing this within a pod and using the user namespace. Sorry I was not paying attention to your command. Basically what is happening, is /dev/mqueue is being shared between the infra container of the pod and the container you are starting. Since the container you are starting is in a different user namespace, the /dev/mqueue in the pod, is not allowed to mount. I would guess that it is owned by real root.

To make this work, I believe we have to fix podman pod create to allow you to create a pod within a user namespace.

rhatdan commented 3 years ago

@giuseppe Agree?

matpen commented 3 years ago

Does this work if you don't deal with usernamespace?

Yes, that command works as expected.

Basically what is happening, is /dev/mqueue is being shared between the infra container of the pod and the container you are starting. Since the container you are starting is in a different user namespace, the /dev/mqueue in the pod, is not allowed to mount. I would guess that it is owned by real root.

This sounds plausible based on my research in the Internet and (limited) understanding of containers.

matteo@matteo-laptop:~$ sudo ls -ld /dev/mqueue
drwxrwxrwt 2 root root 40 Jan 15 08:28 /dev/mqueue

To make this work, I believe we have to fix podman pod create to allow you to create a pod within a user namespace.

Does this help at all?

giuseppe commented 3 years ago

Error: mount 'mqueue' to '/dev/mqueue': Operation not permitted: OCI runtime permission denied error

this error doesn't look like coming from crun.

Could you try moving --runtime crun (or setting crun as the default) before the run command? --runtime is a global flag to Podman, not for the run command:

$ sudo podman --runtime crun pod create --name test_pod
$ sudo podman --runtime crun run  --rm   --privileged --pod test_pod  --uidmap 0:1000:1000 --uidmap 1000:100000:1 "docker.io/library/ubuntu:18.04" whoami
matpen commented 3 years ago

Could you try moving --runtime crun (or setting crun as the default) before the run command?

Just tried that... unfortunately I get the same error message.

giuseppe commented 3 years ago

could you check why it fails to use crun?

The error message Error: mount 'mqueue' to '/dev/mqueue': Operation not permitted: OCI permission denied is coming from runc, not crun

matpen commented 3 years ago

Ok so this is weird:

  1. I get a different error message if I omit the --runtime crun (or maybe the same error, just formatted in a different way):

    matteo@matteo-laptop:~$ sudo podman pod create --name test_pod2
    fd137ec997b3cf901417cbf51f1979661fa5e396ab890705c9e1cc3d9d479265
    matteo@matteo-laptop:~$ sudo podman run --rm --privileged --pod test_pod2 --uidmap 0:1000:1000 --uidmap 1000:100000:1 "docker.io/library/ubuntu:18.04" whoami
    Error: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"rootfs_linux.go:58: mounting \\\"sysfs\\\" to rootfs \\\"/var/lib/containers/storage/overlay/b25a11d2a576f9625b039ce170b914e656668c16746f80a50dff5c54fa8b2574/merged\\\" at \\\"/sys\\\" caused \\\"operation not permitted\\\"\"": OCI runtime permission denied error
  2. When run with --log-level debug it looks like it is considering both runtimes, and finally using runc:

    ...
    DEBU[0000] using runtime "/usr/sbin/runc"               
    DEBU[0000] using runtime "/usr/local/bin/crun"     
    ...
    DEBU[0000] running conmon: /usr/bin/conmon args=... -r /usr/sbin/runc -b /var/lib/containers/storage/...
Full log here ``` matteo@matteo-laptop:~$ sudo podman --log-level debug run --rm --privileged --pod test_pod2 --uidmap 0:1000:1000 --uidmap 1000:100000:1 "docker.io/library/ubuntu:18.04" whoami DEBU[0000] using conmon: "/usr/bin/conmon" DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db DEBU[0000] Using graph driver overlay DEBU[0000] Using graph root /var/lib/containers/storage DEBU[0000] Using run root /var/run/containers/storage DEBU[0000] Using static dir /var/lib/containers/storage/libpod DEBU[0000] Using tmp dir /var/run/libpod DEBU[0000] Using volume path /var/lib/containers/storage/volumes DEBU[0000] Set libpod namespace to "" DEBU[0000] [graphdriver] trying provided driver "overlay" DEBU[0000] cached value indicated that overlay is supported DEBU[0000] cached value indicated that metacopy is not being used DEBU[0000] cached value indicated that native-diff is usable DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false DEBU[0000] Initializing event backend journald DEBU[0000] using runtime "/usr/sbin/runc" DEBU[0000] using runtime "/usr/local/bin/crun" INFO[0000] Found CNI network podman (type=bridge) at /etc/cni/net.d/87-podman-bridge.conflist DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]docker.io/library/ubuntu:18.04" DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]@d27b9ffc56677946e64c1dc85413006d8f27946eeb9505140b094bade0bfb0cc" DEBU[0000] exporting opaque data as blob "sha256:d27b9ffc56677946e64c1dc85413006d8f27946eeb9505140b094bade0bfb0cc" DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]@d27b9ffc56677946e64c1dc85413006d8f27946eeb9505140b094bade0bfb0cc" DEBU[0000] exporting opaque data as blob "sha256:d27b9ffc56677946e64c1dc85413006d8f27946eeb9505140b094bade0bfb0cc" DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]@d27b9ffc56677946e64c1dc85413006d8f27946eeb9505140b094bade0bfb0cc" WARNING: The same type, major and minor should not be used for multiple devices. WARNING: The same type, major and minor should not be used for multiple devices. WARNING: The same type, major and minor should not be used for multiple devices. WARNING: The same type, major and minor should not be used for multiple devices. WARNING: The same type, major and minor should not be used for multiple devices. WARNING: The same type, major and minor should not be used for multiple devices. DEBU[0000] using container 47756306bc4566505d1377f980b0019df785b5789ca73d850389402313de1e47 netmode DEBU[0000] using container 47756306bc4566505d1377f980b0019df785b5789ca73d850389402313de1e47 utsmode DEBU[0000] Using container 47756306bc4566505d1377f980b0019df785b5789ca73d850389402313de1e47 ipcmode DEBU[0000] adding container to pod test_pod2 DEBU[0000] created OCI spec and options for new container DEBU[0000] Allocated lock 29 for container c199dbef39fb5afb7c4fe439060e478e80817520d6ce44738f7aa55f303fe82c DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]@d27b9ffc56677946e64c1dc85413006d8f27946eeb9505140b094bade0bfb0cc" DEBU[0000] exporting opaque data as blob "sha256:d27b9ffc56677946e64c1dc85413006d8f27946eeb9505140b094bade0bfb0cc" DEBU[0000] created container "c199dbef39fb5afb7c4fe439060e478e80817520d6ce44738f7aa55f303fe82c" DEBU[0000] container "c199dbef39fb5afb7c4fe439060e478e80817520d6ce44738f7aa55f303fe82c" has work directory "/var/lib/containers/storage/overlay-containers/c199dbef39fb5afb7c4fe439060e478e80817520d6ce44738f7aa55f303fe82c/userdata" DEBU[0000] container "c199dbef39fb5afb7c4fe439060e478e80817520d6ce44738f7aa55f303fe82c" has run directory "/var/run/containers/storage/overlay-containers/c199dbef39fb5afb7c4fe439060e478e80817520d6ce44738f7aa55f303fe82c/userdata" DEBU[0000] New container created "c199dbef39fb5afb7c4fe439060e478e80817520d6ce44738f7aa55f303fe82c" DEBU[0000] container "c199dbef39fb5afb7c4fe439060e478e80817520d6ce44738f7aa55f303fe82c" has CgroupParent "machine.slice/machine-libpod_pod_fd137ec997b3cf901417cbf51f1979661fa5e396ab890705c9e1cc3d9d479265.slice/libpod-c199dbef39fb5afb7c4fe439060e478e80817520d6ce44738f7aa55f303fe82c.scope" DEBU[0000] Not attaching to stdin DEBU[0000] overlay: mount_data=lowerdir=/var/lib/containers/storage/overlay/l/6VGL52LX467RJQ2IV75X54DNME:/var/lib/containers/storage/overlay/l/6VGL52LX467RJQ2IV75X54DNME/../diff1:/var/lib/containers/storage/overlay/l/3ZUTSUBZH5DCTDQAHZ4TMT7CJY:/var/lib/containers/storage/overlay/l/MU44Y6BVEGYHO4TEFOQ77P7PH7:/var/lib/containers/storage/overlay/l/QL7BMVI4PRIM5X223CCASWOU2V:/var/lib/containers/storage/overlay/l/FQ3GDQGN4FBXWO5QYL4VUKUEAO,upperdir=/var/lib/containers/storage/overlay/35bb441ca66ecba59db58093816775333738aa84fdc9267c4ca14e935905b08b/diff,workdir=/var/lib/containers/storage/overlay/35bb441ca66ecba59db58093816775333738aa84fdc9267c4ca14e935905b08b/work DEBU[0000] mounted container "c199dbef39fb5afb7c4fe439060e478e80817520d6ce44738f7aa55f303fe82c" at "/var/lib/containers/storage/overlay/35bb441ca66ecba59db58093816775333738aa84fdc9267c4ca14e935905b08b/merged" DEBU[0000] Created root filesystem for container c199dbef39fb5afb7c4fe439060e478e80817520d6ce44738f7aa55f303fe82c at /var/lib/containers/storage/overlay/35bb441ca66ecba59db58093816775333738aa84fdc9267c4ca14e935905b08b/merged DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode secret DEBU[0000] Setting CGroups for container c199dbef39fb5afb7c4fe439060e478e80817520d6ce44738f7aa55f303fe82c to machine-libpod_pod_fd137ec997b3cf901417cbf51f1979661fa5e396ab890705c9e1cc3d9d479265.slice:libpod:c199dbef39fb5afb7c4fe439060e478e80817520d6ce44738f7aa55f303fe82c DEBU[0000] reading hooks from /usr/share/containers/oci/hooks.d DEBU[0000] reading hooks from /etc/containers/oci/hooks.d DEBU[0000] Created OCI spec for container c199dbef39fb5afb7c4fe439060e478e80817520d6ce44738f7aa55f303fe82c at /var/lib/containers/storage/overlay-containers/c199dbef39fb5afb7c4fe439060e478e80817520d6ce44738f7aa55f303fe82c/userdata/config.json DEBU[0000] /usr/bin/conmon messages will be logged to syslog DEBU[0000] running conmon: /usr/bin/conmon args="[--api-version 1 -s -c c199dbef39fb5afb7c4fe439060e478e80817520d6ce44738f7aa55f303fe82c -u c199dbef39fb5afb7c4fe439060e478e80817520d6ce44738f7aa55f303fe82c -r /usr/sbin/runc -b /var/lib/containers/storage/overlay-containers/c199dbef39fb5afb7c4fe439060e478e80817520d6ce44738f7aa55f303fe82c/userdata -p /var/run/containers/storage/overlay-containers/c199dbef39fb5afb7c4fe439060e478e80817520d6ce44738f7aa55f303fe82c/userdata/pidfile -l k8s-file:/var/lib/containers/storage/overlay-containers/c199dbef39fb5afb7c4fe439060e478e80817520d6ce44738f7aa55f303fe82c/userdata/ctr.log --exit-dir /var/run/libpod/exits --socket-dir-path /var/run/libpod/socket --log-level debug --syslog --conmon-pidfile /var/run/containers/storage/overlay-containers/c199dbef39fb5afb7c4fe439060e478e80817520d6ce44738f7aa55f303fe82c/userdata/conmon.pid --exit-command /usr/bin/podman2 --exit-command-arg --root --exit-command-arg /var/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /var/run/containers/storage --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /var/run/libpod --exit-command-arg --runtime --exit-command-arg runc --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg --rm --exit-command-arg c199dbef39fb5afb7c4fe439060e478e80817520d6ce44738f7aa55f303fe82c]" INFO[0000] Running conmon under slice machine-libpod_pod_fd137ec997b3cf901417cbf51f1979661fa5e396ab890705c9e1cc3d9d479265.slice and unitName libpod-conmon-c199dbef39fb5afb7c4fe439060e478e80817520d6ce44738f7aa55f303fe82c.scope DEBU[0000] Received: -1 DEBU[0000] Cleaning up container c199dbef39fb5afb7c4fe439060e478e80817520d6ce44738f7aa55f303fe82c DEBU[0000] unmounted container "c199dbef39fb5afb7c4fe439060e478e80817520d6ce44738f7aa55f303fe82c" DEBU[0000] Cleaning up container c199dbef39fb5afb7c4fe439060e478e80817520d6ce44738f7aa55f303fe82c DEBU[0000] Container c199dbef39fb5afb7c4fe439060e478e80817520d6ce44738f7aa55f303fe82c storage is already unmounted, skipping... DEBU[0000] Container c199dbef39fb5afb7c4fe439060e478e80817520d6ce44738f7aa55f303fe82c storage is already unmounted, skipping... DEBU[0000] ExitCode msg: "time=\"2021-01-19t22:51:16+01:00\" level=warning msg=\"exit status 1\"\ntime=\"2021-01-19t22:51:16+01:00\" level=error msg=\"container_linux.go:349: starting container process caused \\\"process_linux.go:449: container init caused \\\\\\\"rootfs_linux.go:58: mounting \\\\\\\\\\\\\\\"sysfs\\\\\\\\\\\\\\\" to rootfs \\\\\\\\\\\\\\\"/var/lib/containers/storage/overlay/35bb441ca66ecba59db58093816775333738aa84fdc9267c4ca14e935905b08b/merged\\\\\\\\\\\\\\\" at \\\\\\\\\\\\\\\"/sys\\\\\\\\\\\\\\\" caused \\\\\\\\\\\\\\\"operation not permitted\\\\\\\\\\\\\\\"\\\\\\\"\\\"\"\ncontainer_linux.go:349: starting container process caused \"process_linux.go:449: container init caused \\\"rootfs_linux.go:58: mounting \\\\\\\"sysfs\\\\\\\" to rootfs \\\\\\\"/var/lib/containers/storage/overlay/35bb441ca66ecba59db58093816775333738aa84fdc9267c4ca14e935905b08b/merged\\\\\\\" at \\\\\\\"/sys\\\\\\\" caused \\\\\\\"operation not permitted\\\\\\\"\\\"\": oci runtime permission denied error" ERRO[0000] time="2021-01-19T22:51:16+01:00" level=warning msg="exit status 1" time="2021-01-19T22:51:16+01:00" level=error msg="container_linux.go:349: starting container process caused \"process_linux.go:449: container init caused \\\"rootfs_linux.go:58: mounting \\\\\\\"sysfs\\\\\\\" to rootfs \\\\\\\"/var/lib/containers/storage/overlay/35bb441ca66ecba59db58093816775333738aa84fdc9267c4ca14e935905b08b/merged\\\\\\\" at \\\\\\\"/sys\\\\\\\" caused \\\\\\\"operation not permitted\\\\\\\"\\\"\"" container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"rootfs_linux.go:58: mounting \\\"sysfs\\\" to rootfs \\\"/var/lib/containers/storage/overlay/35bb441ca66ecba59db58093816775333738aa84fdc9267c4ca14e935905b08b/merged\\\" at \\\"/sys\\\" caused \\\"operation not permitted\\\"\"": OCI runtime permission denied error ```
  1. I searched through the configuration files, but do not seem to find any value that could interfere with the runtime specified on command line (which IIUC should have precedence anyway):
Example of storage.conf ``` matteo@matteo-laptop:~$ cat /etc/containers/storage.conf # storage.conf is the configuration file for all tools # that share the containers/storage libraries # See man 5 containers-storage.conf for more information # The "container storage" table contains all of the server options. [storage] # Default Storage Driver driver = "overlay" # Temporary storage location runroot = "/var/run/containers/storage" # Primary read-write location of container storage graphroot = "/var/lib/containers/storage" [storage.options] # AdditionalImageStores is used to pass paths to additional read-only image stores # Must be comma separated list. additionalimagestores = [ ] # Size is used to set a maximum size of the container image. Only supported by # certain container storage drivers (currently overlay, zfs, vfs, btrfs) size = "" # OverrideKernelCheck tells the driver to ignore kernel checks based on kernel version override_kernel_check = "true" ```
giuseppe commented 3 years ago

what is the --debug output when you use --runtime crun?

You can override the runtime settings in containers.conf, not in storage.conf

matpen commented 3 years ago

what is the --debug output when you use --runtime crun?

Similar to the above, but in this case it looks like it ends up using crun:

...
DEBU[0000] using runtime "/usr/sbin/runc"               
DEBU[0000] using runtime "/usr/local/bin/crun"          
...
DEBU[0000] running conmon: /usr/bin/conmon args=... -r /usr/local/bin/crun -b /var/lib/containers/storage/...
Full log here ``` matteo@matteo-laptop:~$ sudo podman --runtime crun pod create --name test_pod 22f8f797edf8dfd26a7879fcf5628369fd8b03aef0b43698186c70397d1dd1d5 matteo@matteo-laptop:~$ sudo podman --log-level debug --runtime crun run --rm --privileged --pod test_pod --uidmap 0:1000:1000 --uidmap 1000:100000:1 "docker.io/library/ubuntu:18.04" whoami DEBU[0000] using conmon: "/usr/bin/conmon" DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db DEBU[0000] Using graph driver overlay DEBU[0000] Using graph root /var/lib/containers/storage DEBU[0000] Using run root /var/run/containers/storage DEBU[0000] Using static dir /var/lib/containers/storage/libpod DEBU[0000] Using tmp dir /var/run/libpod DEBU[0000] Using volume path /var/lib/containers/storage/volumes DEBU[0000] Set libpod namespace to "" DEBU[0000] [graphdriver] trying provided driver "overlay" DEBU[0000] cached value indicated that overlay is supported DEBU[0000] cached value indicated that metacopy is not being used DEBU[0000] cached value indicated that native-diff is usable DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false DEBU[0000] Initializing event backend journald DEBU[0000] using runtime "/usr/sbin/runc" DEBU[0000] using runtime "/usr/local/bin/crun" INFO[0000] Found CNI network podman (type=bridge) at /etc/cni/net.d/87-podman-bridge.conflist DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]docker.io/library/ubuntu:18.04" DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]@d27b9ffc56677946e64c1dc85413006d8f27946eeb9505140b094bade0bfb0cc" DEBU[0000] exporting opaque data as blob "sha256:d27b9ffc56677946e64c1dc85413006d8f27946eeb9505140b094bade0bfb0cc" DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]@d27b9ffc56677946e64c1dc85413006d8f27946eeb9505140b094bade0bfb0cc" DEBU[0000] exporting opaque data as blob "sha256:d27b9ffc56677946e64c1dc85413006d8f27946eeb9505140b094bade0bfb0cc" DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]@d27b9ffc56677946e64c1dc85413006d8f27946eeb9505140b094bade0bfb0cc" WARNING: The same type, major and minor should not be used for multiple devices. WARNING: The same type, major and minor should not be used for multiple devices. WARNING: The same type, major and minor should not be used for multiple devices. WARNING: The same type, major and minor should not be used for multiple devices. WARNING: The same type, major and minor should not be used for multiple devices. WARNING: The same type, major and minor should not be used for multiple devices. DEBU[0000] using container 16811f515a90bd7806bf2022265928f805538581fc4d7c1176dc653e11937e6a netmode DEBU[0000] using container 16811f515a90bd7806bf2022265928f805538581fc4d7c1176dc653e11937e6a utsmode DEBU[0000] Using container 16811f515a90bd7806bf2022265928f805538581fc4d7c1176dc653e11937e6a ipcmode DEBU[0000] adding container to pod test_pod DEBU[0000] created OCI spec and options for new container DEBU[0000] Allocated lock 27 for container 36468bf601c1eb3003654a69b7e6ceb76eccd42617b7b1f9d859dcbe7ecb9ae2 DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]@d27b9ffc56677946e64c1dc85413006d8f27946eeb9505140b094bade0bfb0cc" DEBU[0000] exporting opaque data as blob "sha256:d27b9ffc56677946e64c1dc85413006d8f27946eeb9505140b094bade0bfb0cc" DEBU[0000] created container "36468bf601c1eb3003654a69b7e6ceb76eccd42617b7b1f9d859dcbe7ecb9ae2" DEBU[0000] container "36468bf601c1eb3003654a69b7e6ceb76eccd42617b7b1f9d859dcbe7ecb9ae2" has work directory "/var/lib/containers/storage/overlay-containers/36468bf601c1eb3003654a69b7e6ceb76eccd42617b7b1f9d859dcbe7ecb9ae2/userdata" DEBU[0000] container "36468bf601c1eb3003654a69b7e6ceb76eccd42617b7b1f9d859dcbe7ecb9ae2" has run directory "/var/run/containers/storage/overlay-containers/36468bf601c1eb3003654a69b7e6ceb76eccd42617b7b1f9d859dcbe7ecb9ae2/userdata" DEBU[0000] New container created "36468bf601c1eb3003654a69b7e6ceb76eccd42617b7b1f9d859dcbe7ecb9ae2" DEBU[0000] container "36468bf601c1eb3003654a69b7e6ceb76eccd42617b7b1f9d859dcbe7ecb9ae2" has CgroupParent "machine.slice/machine-libpod_pod_22f8f797edf8dfd26a7879fcf5628369fd8b03aef0b43698186c70397d1dd1d5.slice/libpod-36468bf601c1eb3003654a69b7e6ceb76eccd42617b7b1f9d859dcbe7ecb9ae2.scope" DEBU[0000] Not attaching to stdin DEBU[0000] Strongconnecting node 16811f515a90bd7806bf2022265928f805538581fc4d7c1176dc653e11937e6a DEBU[0000] Pushed 16811f515a90bd7806bf2022265928f805538581fc4d7c1176dc653e11937e6a onto stack DEBU[0000] Finishing node 16811f515a90bd7806bf2022265928f805538581fc4d7c1176dc653e11937e6a. Popped 16811f515a90bd7806bf2022265928f805538581fc4d7c1176dc653e11937e6a off stack DEBU[0000] overlay: mount_data=lowerdir=/var/lib/containers/storage/overlay/l/RZWOGGJLGOLPPVHM7ASX5UZW2H,upperdir=/var/lib/containers/storage/overlay/99b6932d6fdc6d735490d259e6f2d5b64886f229deea68b5d0812bbf522287cf/diff,workdir=/var/lib/containers/storage/overlay/99b6932d6fdc6d735490d259e6f2d5b64886f229deea68b5d0812bbf522287cf/work DEBU[0000] mounted container "16811f515a90bd7806bf2022265928f805538581fc4d7c1176dc653e11937e6a" at "/var/lib/containers/storage/overlay/99b6932d6fdc6d735490d259e6f2d5b64886f229deea68b5d0812bbf522287cf/merged" DEBU[0000] Created root filesystem for container 16811f515a90bd7806bf2022265928f805538581fc4d7c1176dc653e11937e6a at /var/lib/containers/storage/overlay/99b6932d6fdc6d735490d259e6f2d5b64886f229deea68b5d0812bbf522287cf/merged DEBU[0000] Made network namespace at /var/run/netns/cni-d820224f-e47d-99f9-fb98-fc6ac455146d for container 16811f515a90bd7806bf2022265928f805538581fc4d7c1176dc653e11937e6a INFO[0000] Got pod network &{Name:22f8f797edf8-infra Namespace:22f8f797edf8-infra ID:16811f515a90bd7806bf2022265928f805538581fc4d7c1176dc653e11937e6a NetNS:/var/run/netns/cni-d820224f-e47d-99f9-fb98-fc6ac455146d Networks:[] RuntimeConfig:map[podman:{IP: PortMappings:[] Bandwidth: IpRanges:[]}]} INFO[0000] About to add CNI network cni-loopback (type=loopback) INFO[0000] Got pod network &{Name:22f8f797edf8-infra Namespace:22f8f797edf8-infra ID:16811f515a90bd7806bf2022265928f805538581fc4d7c1176dc653e11937e6a NetNS:/var/run/netns/cni-d820224f-e47d-99f9-fb98-fc6ac455146d Networks:[] RuntimeConfig:map[podman:{IP: PortMappings:[] Bandwidth: IpRanges:[]}]} INFO[0000] About to add CNI network podman (type=bridge) DEBU[0000] [0] CNI result: Interfaces:[{Name:cni-podman0 Mac:62:57:91:b6:95:06 Sandbox:} {Name:vethff7e2ebe Mac:4a:eb:32:b3:19:a2 Sandbox:} {Name:eth0 Mac:56:89:c6:05:85:58 Sandbox:/var/run/netns/cni-d820224f-e47d-99f9-fb98-fc6ac455146d}], IP:[{Version:4 Interface:0xc4207b6ee8 Address:{IP:10.88.179.176 Mask:ffff0000} Gateway:10.88.0.1}], Routes:[{Dst:{IP:0.0.0.0 Mask:00000000} GW:}], DNS:{Nameservers:[] Domain: Search:[] Options:[]} INFO[0000] No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4] INFO[0000] IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844] DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode secret WARN[0000] User mount overriding libpod mount at "/dev/shm" DEBU[0000] Setting CGroups for container 16811f515a90bd7806bf2022265928f805538581fc4d7c1176dc653e11937e6a to machine-libpod_pod_22f8f797edf8dfd26a7879fcf5628369fd8b03aef0b43698186c70397d1dd1d5.slice:libpod:16811f515a90bd7806bf2022265928f805538581fc4d7c1176dc653e11937e6a DEBU[0000] reading hooks from /usr/share/containers/oci/hooks.d DEBU[0000] reading hooks from /etc/containers/oci/hooks.d DEBU[0000] Created OCI spec for container 16811f515a90bd7806bf2022265928f805538581fc4d7c1176dc653e11937e6a at /var/lib/containers/storage/overlay-containers/16811f515a90bd7806bf2022265928f805538581fc4d7c1176dc653e11937e6a/userdata/config.json DEBU[0000] /usr/bin/conmon messages will be logged to syslog DEBU[0000] running conmon: /usr/bin/conmon args="[--api-version 1 -s -c 16811f515a90bd7806bf2022265928f805538581fc4d7c1176dc653e11937e6a -u 16811f515a90bd7806bf2022265928f805538581fc4d7c1176dc653e11937e6a -r /usr/local/bin/crun -b /var/lib/containers/storage/overlay-containers/16811f515a90bd7806bf2022265928f805538581fc4d7c1176dc653e11937e6a/userdata -p /var/run/containers/storage/overlay-containers/16811f515a90bd7806bf2022265928f805538581fc4d7c1176dc653e11937e6a/userdata/pidfile -l k8s-file:/var/lib/containers/storage/overlay-containers/16811f515a90bd7806bf2022265928f805538581fc4d7c1176dc653e11937e6a/userdata/ctr.log --exit-dir /var/run/libpod/exits --socket-dir-path /var/run/libpod/socket --log-level debug --syslog --conmon-pidfile /var/run/containers/storage/overlay-containers/16811f515a90bd7806bf2022265928f805538581fc4d7c1176dc653e11937e6a/userdata/conmon.pid]" INFO[0000] Running conmon under slice machine-libpod_pod_22f8f797edf8dfd26a7879fcf5628369fd8b03aef0b43698186c70397d1dd1d5.slice and unitName libpod-conmon-16811f515a90bd7806bf2022265928f805538581fc4d7c1176dc653e11937e6a.scope DEBU[0000] Received: 12502 INFO[0000] Got Conmon PID as 12473 DEBU[0000] Created container 16811f515a90bd7806bf2022265928f805538581fc4d7c1176dc653e11937e6a in OCI runtime DEBU[0000] Starting container 16811f515a90bd7806bf2022265928f805538581fc4d7c1176dc653e11937e6a with command [/pause] DEBU[0000] Started container 16811f515a90bd7806bf2022265928f805538581fc4d7c1176dc653e11937e6a DEBU[0000] overlay: mount_data=lowerdir=/var/lib/containers/storage/overlay/l/6VGL52LX467RJQ2IV75X54DNME:/var/lib/containers/storage/overlay/l/6VGL52LX467RJQ2IV75X54DNME/../diff1:/var/lib/containers/storage/overlay/l/3ZUTSUBZH5DCTDQAHZ4TMT7CJY:/var/lib/containers/storage/overlay/l/MU44Y6BVEGYHO4TEFOQ77P7PH7:/var/lib/containers/storage/overlay/l/QL7BMVI4PRIM5X223CCASWOU2V:/var/lib/containers/storage/overlay/l/FQ3GDQGN4FBXWO5QYL4VUKUEAO,upperdir=/var/lib/containers/storage/overlay/365e0639159c4a319184a52656f304be5e526d773694bfcedd465cc2a4aa55e2/diff,workdir=/var/lib/containers/storage/overlay/365e0639159c4a319184a52656f304be5e526d773694bfcedd465cc2a4aa55e2/work DEBU[0000] mounted container "36468bf601c1eb3003654a69b7e6ceb76eccd42617b7b1f9d859dcbe7ecb9ae2" at "/var/lib/containers/storage/overlay/365e0639159c4a319184a52656f304be5e526d773694bfcedd465cc2a4aa55e2/merged" DEBU[0000] Created root filesystem for container 36468bf601c1eb3003654a69b7e6ceb76eccd42617b7b1f9d859dcbe7ecb9ae2 at /var/lib/containers/storage/overlay/365e0639159c4a319184a52656f304be5e526d773694bfcedd465cc2a4aa55e2/merged DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode secret DEBU[0000] Setting CGroups for container 36468bf601c1eb3003654a69b7e6ceb76eccd42617b7b1f9d859dcbe7ecb9ae2 to machine-libpod_pod_22f8f797edf8dfd26a7879fcf5628369fd8b03aef0b43698186c70397d1dd1d5.slice:libpod:36468bf601c1eb3003654a69b7e6ceb76eccd42617b7b1f9d859dcbe7ecb9ae2 DEBU[0000] reading hooks from /usr/share/containers/oci/hooks.d DEBU[0000] reading hooks from /etc/containers/oci/hooks.d DEBU[0000] Created OCI spec for container 36468bf601c1eb3003654a69b7e6ceb76eccd42617b7b1f9d859dcbe7ecb9ae2 at /var/lib/containers/storage/overlay-containers/36468bf601c1eb3003654a69b7e6ceb76eccd42617b7b1f9d859dcbe7ecb9ae2/userdata/config.json DEBU[0000] /usr/bin/conmon messages will be logged to syslog DEBU[0000] running conmon: /usr/bin/conmon args="[--api-version 1 -s -c 36468bf601c1eb3003654a69b7e6ceb76eccd42617b7b1f9d859dcbe7ecb9ae2 -u 36468bf601c1eb3003654a69b7e6ceb76eccd42617b7b1f9d859dcbe7ecb9ae2 -r /usr/local/bin/crun -b /var/lib/containers/storage/overlay-containers/36468bf601c1eb3003654a69b7e6ceb76eccd42617b7b1f9d859dcbe7ecb9ae2/userdata -p /var/run/containers/storage/overlay-containers/36468bf601c1eb3003654a69b7e6ceb76eccd42617b7b1f9d859dcbe7ecb9ae2/userdata/pidfile -l k8s-file:/var/lib/containers/storage/overlay-containers/36468bf601c1eb3003654a69b7e6ceb76eccd42617b7b1f9d859dcbe7ecb9ae2/userdata/ctr.log --exit-dir /var/run/libpod/exits --socket-dir-path /var/run/libpod/socket --log-level debug --syslog --conmon-pidfile /var/run/containers/storage/overlay-containers/36468bf601c1eb3003654a69b7e6ceb76eccd42617b7b1f9d859dcbe7ecb9ae2/userdata/conmon.pid --exit-command /usr/bin/podman2 --exit-command-arg --root --exit-command-arg /var/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /var/run/containers/storage --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /var/run/libpod --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg --rm --exit-command-arg 36468bf601c1eb3003654a69b7e6ceb76eccd42617b7b1f9d859dcbe7ecb9ae2]" INFO[0000] Running conmon under slice machine-libpod_pod_22f8f797edf8dfd26a7879fcf5628369fd8b03aef0b43698186c70397d1dd1d5.slice and unitName libpod-conmon-36468bf601c1eb3003654a69b7e6ceb76eccd42617b7b1f9d859dcbe7ecb9ae2.scope DEBU[0000] Received: -1 DEBU[0000] Cleaning up container 36468bf601c1eb3003654a69b7e6ceb76eccd42617b7b1f9d859dcbe7ecb9ae2 DEBU[0000] unmounted container "36468bf601c1eb3003654a69b7e6ceb76eccd42617b7b1f9d859dcbe7ecb9ae2" DEBU[0000] Cleaning up container 36468bf601c1eb3003654a69b7e6ceb76eccd42617b7b1f9d859dcbe7ecb9ae2 DEBU[0000] Container 36468bf601c1eb3003654a69b7e6ceb76eccd42617b7b1f9d859dcbe7ecb9ae2 storage is already unmounted, skipping... DEBU[0000] Container 36468bf601c1eb3003654a69b7e6ceb76eccd42617b7b1f9d859dcbe7ecb9ae2 storage is already unmounted, skipping... DEBU[0000] ExitCode msg: "mount `mqueue` to '/dev/mqueue': operation not permitted: oci runtime permission denied error" ERRO[0000] mount `mqueue` to '/dev/mqueue': Operation not permitted: OCI runtime permission denied error ```

You can override the runtime settings in containers.conf, not in storage.conf

True that, but I checked all paths listed in the documentation, and they dont exist on my system, e.g.:

matteo@matteo-laptop:~$ sudo cat /usr/share/containers/containers.conf
cat: /usr/share/containers/containers.conf: No such file or directory
matteo@matteo-laptop:~$ sudo cat /etc/containers/containers.conf
cat: /etc/containers/containers.conf: No such file or directory
matteo@matteo-laptop:~$ sudo cat /root/.config/containers/containers.conf
cat: /root/.config/containers/containers.conf: No such file or directory
matteo@matteo-laptop:~$ sudo cat /home/matteo/.config/containers/containers.conf
cat: /home/matteo/.config/containers/containers.conf: No such file or directory
rhatdan commented 3 years ago

If you podman package comes without containers.conf, then you have not install containers-common package.

rhatdan commented 3 years ago

Running the commands on Fedora 33 I am getting a similar error.

sh-5.0# sudo podman pod create \
>     --runtime crun \
>     --name test_pod
0d05b29a362d17035c0f6d0581c4f0539fa831aded0a43ab64ef5bce6c424d1a
sh-5.0# sudo podman run \
>     --rm \
>     --privileged \
>     --runtime crun \
>     --pod test_pod \
>     --uidmap 0:1000:1000 \
>     --uidmap 1000:100000:1 \
>     "docker.io/library/ubuntu:18.04" whoami
Error: error stat'ing file `/var/lib/containers/storage/overlay-containers/6afd0f476274af3c7b5f41aa080179f0e6169328bd4276a69ae5c941fe976ab1/userdata/shm`: Permission denied: OCI permission denied

The issue is the pod's infra container is running in the host user namespace, the container is running in a different user namespace, When the container starts some of the cross mounts between the containers are getting permission deneid. We have to fix pod creation to be able to use a user namespace, so the infra container will run in the same user namespace as the containers running in the pod.

matpen commented 3 years ago

If you podman package comes without containers.conf, then you have not install containers-common package.

Seems like containers-common is installed, but does not contain a containers.conf

Checked with dpkg ``` matteo@matteo-laptop:~$ dpkg -s containers-common Package: containers-common Status: install ok installed Priority: optional Section: devel Installed-Size: 23 Maintainer: Lokesh Mandvekar Architecture: all Source: skopeo Version: 0.1.40-1~ubuntu18.04~ppa1 Replaces: skopeo-containers Provides: skopeo-containers Depends: containers-golang, containers-image (>= 3.0.0) Conflicts: skopeo-containers Conffiles: /etc/containers/policy.json c01eb6950f03419e09d4fc88cb42ff6f /etc/containers/registries.d/default.yaml c827dbcb84f08f744d8cde1ec6ea6614 /etc/containers/storage.conf 6095f3d3a349349d637c2151feeac41d /etc/containers/registries.conf b846d68aaf20142c3591d3f714f62baf obsolete Description: Configuration files for working with image signatures. Homepage: https://github.com/projectatomic/skopeo matteo@matteo-laptop:~$ dpkg -L containers-common /. /etc /etc/containers /etc/containers/policy.json /etc/containers/registries.d /etc/containers/registries.d/default.yaml /etc/containers/storage.conf /usr /usr/share /usr/share/doc /usr/share/doc/containers-common /usr/share/doc/containers-common/changelog.Debian.gz /usr/share/doc/containers-common/copyright /usr/share/man /usr/share/man/man5 /usr/share/man/man5/containers-storage.conf.5.gz /var /var/lib /var/lib/atomic /var/lib/atomic/sigstore /etc/containers/registries.conf matteo@matteo-laptop:~$ podman info host: BuildahVersion: 1.11.3 CgroupVersion: v1 Conmon: package: 'conmon: /usr/bin/conmon' path: /usr/bin/conmon version: 'conmon version 2.0.3, commit: unknown' Distribution: distribution: ubuntu version: "18.04" MemFree: 6011133952 MemTotal: 16647417856 OCIRuntime: name: runc package: 'runc: /usr/sbin/runc' path: /usr/sbin/runc version: 'runc version spec: 1.0.1-dev' SwapFree: 17036734464 SwapTotal: 17036734464 arch: amd64 cpus: 8 eventlogger: journald hostname: matteo-laptop kernel: 5.4.0-62-generic os: linux rootless: false uptime: 1h 26m 45.99s (Approximately 0.04 days) registries: blocked: null insecure: null search: null store: ConfigFile: /etc/containers/storage.conf ContainerStore: number: 28 GraphDriverName: overlay GraphOptions: {} GraphRoot: /var/lib/containers/storage GraphStatus: Backing Filesystem: extfs Native Overlay Diff: "true" Supports d_type: "true" Using metacopy: "false" ImageStore: number: 163 RunRoot: /var/run/containers/storage VolumePath: /var/lib/containers/storage/volumes matteo@matteo-laptop:~$ podman --version podman version 1.6.2 ```
chengscott commented 3 years ago

Related PR #8393

giuseppe commented 3 years ago

The issue is the pod's infra container is running in the host user namespace, the container is running in a different user namespace, When the container starts some of the cross mounts between the containers are getting permission deneid. We have to fix pod creation to be able to use a user namespace, so the infra container will run in the same user namespace as the containers running in the pod.

crun supports this case and will use the correct namespaces inside of the container.

The issue I see here is that the owner of /var/lib/containers/storage/overlay-containers/6afd0f476274af3c7b5f41aa080179f0e6169328bd4276a69ae5c941fe976ab1/userdata/shm is not mapped in the new user namespace, you'd need to try with something like: --uidmap 0:0:1000:

# podman --runtime crun pod create    --name test_pod
# podman --runtime crun run  --rm  --pod test_pod --uidmap 0:0:1000   fedora whoami
root
matpen commented 3 years ago

Tried a few variations of the command with no luck:

matteo@matteo-laptop:~$ sudo podman --runtime crun pod create test_pod
matteo@matteo-laptop:~$ sudo podman --runtime crun run --rm --privileged --pod test_pod --uidmap 0:0:1000 "docker.io/library/fedora:latest" whoami
Error: mount `mqueue` to '/dev/mqueue': Operation not permitted: OCI runtime permission denied error

matteo@matteo-laptop:~$ sudo podman --runtime crun run --rm --privileged --pod test_pod --uidmap 0:0:1000 "docker.io/library/ubuntu:18.04" whoami
Error: mount `mqueue` to '/dev/mqueue': Operation not permitted: OCI runtime permission denied error

matteo@matteo-laptop:~$ sudo podman --runtime crun run --rm --privileged --pod test_pod --uidmap 0:0:1000 --uidmap 0:1000:1000 "docker.io/library/ubuntu:18.04" whoami
Error: setresuid(0): Invalid argument
writing file `/proc/9332/gid_map`: Invalid argument: OCI runtime error

matteo@matteo-laptop:~$ sudo podman --runtime crun run --rm --privileged --pod test_pod --uidmap 0:0:1000 --uidmap 0:1000:1000 --uidmap 1000:100000:1 "docker.io/library/ubuntu:18.04" whoami
Error: setresuid(0): Invalid argument
writing file `/proc/9332/gid_map`: Invalid argument: OCI runtime error

The owner of /var/lib/containers/storage/overlay-containers/.../userdata (and all parent folders) is root, but I dont see the shm folder inside of it.

matteo@matteo-laptop:~$ sudo ls -la /var/lib/containers/storage/overlay-containers/7421c667446e421d4c49c4f3e7e93ab441abb4b2b43b2c25662c8640e970ab3d/userdata
total 20
drwx------ 3 root root 4096 Feb 27 16:50 .
drwx------ 3 root root 4096 Feb 27 16:49 ..
drwxr-xr-x 2 root root 4096 Feb 27 16:49 artifacts
srwx------ 1 root root    0 Feb 27 16:50 attach
-rw-r--r-- 1 root root 7950 Feb 27 16:50 config.json
prw-r--r-- 1 root root    0 Feb 27 16:50 ctl
-rw------- 1 root root    0 Feb 27 16:50 ctr.log
prw-r--r-- 1 root root    0 Feb 27 16:50 winsz
github-actions[bot] commented 3 years ago

A friendly reminder that this issue had no activity for 30 days.

rhatdan commented 3 years ago

@matpen Any update on this issue?

matpen commented 3 years ago

No update from my side: it keeps not working, as per my last message. I am able to reproduce on multiple system, but not to figure out what might be wrong.

rhatdan commented 3 years ago

Well it seems @giuseppe believes it works. with crun.

giuseppe commented 3 years ago

--uidmap 0:0:1000 --uidmap 0:1000:1000 --uidmap 1000:100000:1 is not a valid configuration as the range 0:1000 is assigned multiple times.

I've tried:

$ sudo podman --runtime crun pod create --name test_pod
381785df3f388b89ebfefc2dc582f7a1c546cc9e74fddab74d095ae06ae6296b
$ sudo podman --runtime crun run  --rm  --pod test_pod --uidmap 0:0:10000 alpine whoami
root

With --privileged I see a different error:

$ sudo podman --runtime crun run  --rm  --pod test_pod --privileged --uidmap 0:0:10000 alpine whoami
Error: OCI runtime error: create device `/dev/tty`: No such device or address
matpen commented 3 years ago

--uidmap 0:0:1000 --uidmap 0:1000:1000 --uidmap 1000:100000:1 is not a valid configuration

True, I just tried a lot of combinations, including some wrong ones.

I've tried:

I do recognize that this works on your setup. I just tried yet another system, where I freshly installed crun for the test: same error.

At this stage, it appears obvious that it works for you and it does not work for me. Something must be different either in the fact that the host OS is different (in my tests, it is always Ubuntu), or in the way I install podman or crun.

I just hoped to get some pointers towards figuring out what the cause could be and help someone else who might find himself in the same situation. However, since we cannot reproduce it, it might just not be worth the effort: I will simply try a different approach for my use case.

giuseppe commented 3 years ago

At this stage, it appears obvious that it works for you and it does not work for me. Something must be different either in the fact that the host OS is different (in my tests, it is always Ubuntu), or in the way I install podman or crun.

crun alone is not enough, it also requires some newer features in the kernel, that should be available in Linux 5.3. I could be wrong about that, is there any newer kernel you can try?

giuseppe commented 3 years ago

another potential issue is that crun could be compiled without the support for the new mount API.

There is a check at build time, from the crun configure.ac file:

AC_MSG_CHECKING([for new mount API (fsconfig)])
AC_COMPILE_IFELSE(
    [AC_LANG_SOURCE([[
            #include <linux/mount.h>
            int cmd = FSCONFIG_CMD_CREATE;
        ]])],
        [AC_MSG_RESULT(yes)
         AC_DEFINE([HAVE_FSCONFIG_CMD_CREATE], 1, [Define if FSCONFIG_CMD_CREATE is available])],
        [AC_MSG_RESULT(no)])

Could you try building crun and see whether you get the following:

$ ./configure
$ grep HAVE_FSCONFIG_CMD_CREATE config.h
#define HAVE_FSCONFIG_CMD_CREATE 1
giuseppe commented 3 years ago

or you could check more easily with:

$ grep FSCONFIG_CMD_CREATE /usr/include/linux/mount.h 
    FSCONFIG_CMD_CREATE = 6,    /* Invoke superblock creation */
matpen commented 3 years ago

Thanks @giuseppe for not giving up on this.

crun alone is not enough, it also requires some newer features in the kernel, that should be available in Linux 5.3. I could be wrong about that, is there any newer kernel you can try?

As stated in the OP, the issue was first detected on 5.4.0-60-generic. Over time I upgraded the kernel a few times, and right now I am testing with a range of systems from 5.4.0-58-generic to 5.4.0-70-generic .

I do not feel confident to upgrade beyond what is officially provided by the OS vendor, but if you think that it can be useful, I will try a VM with a newer one. IIUC 5.10 should be latest stable at the time of this writing.

Could you try building crun and see whether you get the following:

Interesting. Both with kernel 5.4.0-65-generic and 5.4.0-70-generic I get

matteo@test1:/tmp/crun$ uname -a
Linux test1 5.4.0-65-generic #73~18.04.1-Ubuntu SMP Tue Jan 19 09:02:24 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
matteo@test1:/tmp/crun$ grep HAVE_FSCONFIG_CMD_CREATE config.h
/* #undef HAVE_FSCONFIG_CMD_CREATE */

or you could check more easily with:

Even more interesting. On my system, that file does not exist, but I run a search with locate and found the following:

matteo@matteo-laptop:~$ grep FSCONFIG_CMD_CREATE \
    /usr/include/sys/mount.h \
    /usr/include/x86_64-linux-gnu/sys/mount.h \
    /usr/src/linux-headers-5.3.0-62/include/linux/mount.h \
    /usr/src/linux-hwe-5.4-headers-5.4.0-58/include/linux/mount.h \
    /usr/src/linux-hwe-5.4-headers-5.4.0-58/include/uapi/linux/mount.h \
    /usr/src/linux-hwe-5.4-headers-5.4.0-70/include/linux/mount.h \
    /usr/src/linux-hwe-5.4-headers-5.4.0-70/include/uapi/linux/mount.h
/usr/src/linux-hwe-5.4-headers-5.4.0-58/include/uapi/linux/mount.h: FSCONFIG_CMD_CREATE = 6,    /* Invoke superblock creation */
/usr/src/linux-hwe-5.4-headers-5.4.0-70/include/uapi/linux/mount.h: FSCONFIG_CMD_CREATE = 6,    /* Invoke superblock creation */

Note how we get results only for the UAPI files (not sure about what this implies).

I am not familiar with the above results, but they do seem to hint towards a missing kernel feature.

giuseppe commented 3 years ago

The kernel is fine but the /usr/include/sys/mount.h header file is outdated and misses the definition for FSCONFIG_CMD_CREATE. So crun is built without the support for such features.

We could probably fix the issue in crun itself and define the missing symbols when they are missing.

giuseppe commented 3 years ago

could you try to build crun with the following patch?

diff --git a/src/libcrun/linux.c b/src/libcrun/linux.c
index 9e2264f..95bd9e7 100644
--- a/src/libcrun/linux.c
+++ b/src/libcrun/linux.c
@@ -76,6 +76,10 @@
 #  define MOVE_MOUNT_T_EMPTY_PATH 0x00000040
 #endif

+#ifndef HAVE_FSCONFIG_CMD_CREATE
+#  define FSCONFIG_CMD_CREATE 6
+#endif
+
 struct remount_s
 {
   struct remount_s *next;
@@ -543,7 +547,6 @@ open_mount_target (libcrun_container_t *container, const char *target_rel, libcr
 static int
 fsopen_mount (runtime_spec_schema_defs_mount *mount)
 {
-#ifdef HAVE_FSCONFIG_CMD_CREATE
   cleanup_close int fsfd = -1;
   int ret;

@@ -556,32 +559,15 @@ fsopen_mount (runtime_spec_schema_defs_mount *mount)
     return ret;

   return syscall_fsmount (fsfd, FSMOUNT_CLOEXEC, 0);
-#else
-  (void) mount;
-  (void) syscall_fsopen;
-  (void) syscall_fsconfig;
-  (void) syscall_fsmount;
-  errno = ENOSYS;
-  return -1;
-#endif
 }

 static int
 fs_move_mount_to (int fd, int dirfd, const char *name)
 {
-#ifdef HAVE_FSCONFIG_CMD_CREATE
   if (name)
     return syscall_move_mount (fd, "", dirfd, name, MOVE_MOUNT_F_EMPTY_PATH);

   return syscall_move_mount (fd, "", dirfd, "", MOVE_MOUNT_T_EMPTY_PATH | MOVE_MOUNT_F_EMPTY_PATH);
-#else
-  (void) fd;
-  (void) dirfd;
-  (void) name;
-  (void) syscall_move_mount;
-  errno = ENOSYS;
-  return -1;
-#endif
 }

 enum
matpen commented 3 years ago

The patch applies correctly against master

matteo@test1:/tmp/crun$ patch -p1 < /tmp/crun.patch
patching file src/libcrun/linux.c
Hunk #3 succeeded at 559 with fuzz 1.

but build fails with

matteo@test1:/tmp/crun$ ./autogen.sh
...
matteo@test1:/tmp/crun$ ./configure
...
matteo@test1:/tmp/crun$ make
...
src/libcrun/linux.c: In function 'fsopen_mount':
src/libcrun/linux.c:553:39: error: 'FSOPEN_CLOEXEC' undeclared (first use in this function); did you mean 'SOCK_CLOEXEC'?
   fsfd = syscall_fsopen (mount->type, FSOPEN_CLOEXEC);
                                       ^~~~~~~~~~~~~~
                                       SOCK_CLOEXEC
src/libcrun/linux.c:553:39: note: each undeclared identifier is reported only once for each function it appears in
src/libcrun/linux.c:561:33: error: 'FSMOUNT_CLOEXEC' undeclared (first use in this function); did you mean 'FSOPEN_CLOEXEC'?
   return syscall_fsmount (fsfd, FSMOUNT_CLOEXEC, 0);
                                 ^~~~~~~~~~~~~~~
                                 FSOPEN_CLOEXEC

Full log attached, it case it is useful.

giuseppe commented 3 years ago

thanks, some other missing symbols. Could you add:

#define FSOPEN_CLOEXEC      0x00000001
#define FSMOUNT_CLOEXEC     0x00000001

at the beginning of the linux.c file and try again?

matpen commented 3 years ago

With that, crun was built successfully. Here is the updated patch.

Unfortunately though, it does not seem to solve the problem:


matteo@test1:/tmp/crun$ crun --version
crun version 0.18.90-a72dd-dirty
commit: a72dd4d303f48e23f3a74a103216363167e4407f
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL

matteo@test1:/tmp/crun$ sudo podman --runtime crun pod create --name test_pod
7cc7e5b939ab7fa5e3b75b5929ef0b3bfcba063823115bc88f4000ffbd10196e

matteo@test1:/tmp/crun$ sudo podman --runtime crun run  --rm  --pod test_pod --privileged --uidmap 0:0:10000 "docker.io/library/ubuntu:18.04" whoami
Error: mount `mqueue` to `/dev/mqueue`: Operation not permitted: OCI runtime permission denied error
matpen commented 3 years ago

The kernel is fine but the /usr/include/sys/mount.h header file is outdated

Can you share thoughts about how this came to be? Anything I can try on my side?

giuseppe commented 3 years ago

where have you copied the crun binary?

Could you specify the full path to podman:

sudo podman --runtime /path/to/the/just/built/crun run  --rm  --pod test_pod --privileged --uidmap 0:0:10000 "docker.io/library/ubuntu:18.04" whoami
giuseppe commented 3 years ago

Can you share thoughts about how this came to be? Anything I can try on my side?

because it lacks definitions for symbols that are present in the current kernel, as you showed with the grep command above

matpen commented 3 years ago

where have you copied the crun binary?

I accepted the system default with a simple make install

matteo@test1:/tmp/crun$ which crun
/usr/local/bin/crun

Could you specify the full path to podman:

No luck

matteo@test1:/tmp/crun$ sudo podman --runtime /usr/local/bin/crun  pod create --name test_pod
a99dc2fdf3a7870ee219b80f7cb9d39c21d6499363a860a8669725ea8057da63

matteo@test1:/tmp/crun$ sudo podman --runtime /usr/local/bin/crun run  --rm  --pod test_pod --privileged --uidmap 0:0:10000 "docker.io/library/ubuntu:18.04" whoami
Error: mount `mqueue` to `/dev/mqueue`: Operation not permitted: OCI runtime permission denied error
Debug log seems to confirm that it is using the correct runtime ``` matteo@test1:/tmp/crun$ sudo podman --runtime /usr/local/bin/crun --log-level debug run --rm --pod test_pod --privileged --uidmap 0:0:10000 "docker.io/library/ubuntu:18.04" whoami DEBU[0000] using conmon: "/usr/bin/conmon" DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db DEBU[0000] Using graph driver overlay DEBU[0000] Using graph root /var/lib/containers/storage DEBU[0000] Using run root /var/run/containers/storage DEBU[0000] Using static dir /var/lib/containers/storage/libpod DEBU[0000] Using tmp dir /var/run/libpod DEBU[0000] Using volume path /var/lib/containers/storage/volumes DEBU[0000] Set libpod namespace to "" DEBU[0000] [graphdriver] trying provided driver "overlay" DEBU[0000] cached value indicated that overlay is supported DEBU[0000] cached value indicated that metacopy is not being used DEBU[0000] cached value indicated that native-diff is usable DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false DEBU[0000] Initializing event backend journald DEBU[0000] using runtime "/usr/sbin/runc" DEBU[0000] using runtime "/usr/local/bin/crun" DEBU[0000] using runtime "/usr/local/bin/crun" INFO[0000] Found CNI network podman (type=bridge) at /etc/cni/net.d/87-podman-bridge.conflist DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]docker.io/library/ubuntu:18.04" DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]@a2a15febcdf362f6115e801d37b5e60d6faaeedcb9896155e5fe9d754025be12" DEBU[0000] exporting opaque data as blob "sha256:a2a15febcdf362f6115e801d37b5e60d6faaeedcb9896155e5fe9d754025be12" DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]@a2a15febcdf362f6115e801d37b5e60d6faaeedcb9896155e5fe9d754025be12" DEBU[0000] exporting opaque data as blob "sha256:a2a15febcdf362f6115e801d37b5e60d6faaeedcb9896155e5fe9d754025be12" DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]@a2a15febcdf362f6115e801d37b5e60d6faaeedcb9896155e5fe9d754025be12" DEBU[0000] using container df869d5bd6eb9b6f3a92e27dfd59fe2f4dc63de3403681ac4fd3b87dac25e256 netmode DEBU[0000] using container df869d5bd6eb9b6f3a92e27dfd59fe2f4dc63de3403681ac4fd3b87dac25e256 utsmode DEBU[0000] Using container df869d5bd6eb9b6f3a92e27dfd59fe2f4dc63de3403681ac4fd3b87dac25e256 ipcmode DEBU[0000] adding container to pod test_pod DEBU[0000] created OCI spec and options for new container DEBU[0000] Allocated lock 57 for container d46051f8844b7d7e9f3f699a1dea69e8d56be342ce9e4fae42908791252ebf63 DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]@a2a15febcdf362f6115e801d37b5e60d6faaeedcb9896155e5fe9d754025be12" DEBU[0000] exporting opaque data as blob "sha256:a2a15febcdf362f6115e801d37b5e60d6faaeedcb9896155e5fe9d754025be12" DEBU[0000] created container "d46051f8844b7d7e9f3f699a1dea69e8d56be342ce9e4fae42908791252ebf63" DEBU[0000] container "d46051f8844b7d7e9f3f699a1dea69e8d56be342ce9e4fae42908791252ebf63" has work directory "/var/lib/containers/storage/overlay-containers/d46051f8844b7d7e9f3f699a1dea69e8d56be342ce9e4fae42908791252ebf63/userdata" DEBU[0000] container "d46051f8844b7d7e9f3f699a1dea69e8d56be342ce9e4fae42908791252ebf63" has run directory "/var/run/containers/storage/overlay-containers/d46051f8844b7d7e9f3f699a1dea69e8d56be342ce9e4fae42908791252ebf63/userdata" DEBU[0000] New container created "d46051f8844b7d7e9f3f699a1dea69e8d56be342ce9e4fae42908791252ebf63" DEBU[0000] container "d46051f8844b7d7e9f3f699a1dea69e8d56be342ce9e4fae42908791252ebf63" has CgroupParent "machine.slice/machine-libpod_pod_a99dc2fdf3a7870ee219b80f7cb9d39c21d6499363a860a8669725ea8057da63.slice/libpod-d46051f8844b7d7e9f3f699a1dea69e8d56be342ce9e4fae42908791252ebf63.scope" DEBU[0000] Not attaching to stdin DEBU[0000] overlay: mount_data=lowerdir=/var/lib/containers/storage/overlay/l/GEYEF4K5KKXJRK3TX326GZNZYN:/var/lib/containers/storage/overlay/l/GEYEF4K5KKXJRK3TX326GZNZYN/../diff1:/var/lib/containers/storage/overlay/l/EOQYERFLVY5LG3CQNE3QAJONJU:/var/lib/containers/storage/overlay/l/KM3LJZMRJ5XVIGJBLRNPW7Z7KN:/var/lib/containers/storage/overlay/l/4YGUYC6RA5SBAAM66WHTZVM6J5:/var/lib/containers/storage/overlay/l/A4CBBSUHHXRRTIFX7RBPPO7TSV,upperdir=/var/lib/containers/storage/overlay/86fbf2ec30f27782f5e6d8833de95133f92a18f8e2b52dd8550a8cbbc5454971/diff,workdir=/var/lib/containers/storage/overlay/86fbf2ec30f27782f5e6d8833de95133f92a18f8e2b52dd8550a8cbbc5454971/work DEBU[0000] mounted container "d46051f8844b7d7e9f3f699a1dea69e8d56be342ce9e4fae42908791252ebf63" at "/var/lib/containers/storage/overlay/86fbf2ec30f27782f5e6d8833de95133f92a18f8e2b52dd8550a8cbbc5454971/merged" DEBU[0000] Created root filesystem for container d46051f8844b7d7e9f3f699a1dea69e8d56be342ce9e4fae42908791252ebf63 at /var/lib/containers/storage/overlay/86fbf2ec30f27782f5e6d8833de95133f92a18f8e2b52dd8550a8cbbc5454971/merged DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode secret DEBU[0000] Setting CGroups for container d46051f8844b7d7e9f3f699a1dea69e8d56be342ce9e4fae42908791252ebf63 to machine-libpod_pod_a99dc2fdf3a7870ee219b80f7cb9d39c21d6499363a860a8669725ea8057da63.slice:libpod:d46051f8844b7d7e9f3f699a1dea69e8d56be342ce9e4fae42908791252ebf63 DEBU[0000] reading hooks from /usr/share/containers/oci/hooks.d DEBU[0000] reading hooks from /etc/containers/oci/hooks.d DEBU[0000] Created OCI spec for container d46051f8844b7d7e9f3f699a1dea69e8d56be342ce9e4fae42908791252ebf63 at /var/lib/containers/storage/overlay-containers/d46051f8844b7d7e9f3f699a1dea69e8d56be342ce9e4fae42908791252ebf63/userdata/config.json DEBU[0000] /usr/bin/conmon messages will be logged to syslog DEBU[0000] running conmon: /usr/bin/conmon args="[--api-version 1 -s -c d46051f8844b7d7e9f3f699a1dea69e8d56be342ce9e4fae42908791252ebf63 -u d46051f8844b7d7e9f3f699a1dea69e8d56be342ce9e4fae42908791252ebf63 -r /usr/local/bin/crun -b /var/lib/containers/storage/overlay-containers/d46051f8844b7d7e9f3f699a1dea69e8d56be342ce9e4fae42908791252ebf63/userdata -p /var/run/containers/storage/overlay-containers/d46051f8844b7d7e9f3f699a1dea69e8d56be342ce9e4fae42908791252ebf63/userdata/pidfile -l k8s-file:/var/lib/containers/storage/overlay-containers/d46051f8844b7d7e9f3f699a1dea69e8d56be342ce9e4fae42908791252ebf63/userdata/ctr.log --exit-dir /var/run/libpod/exits --socket-dir-path /var/run/libpod/socket --log-level debug --syslog --conmon-pidfile /var/run/containers/storage/overlay-containers/d46051f8844b7d7e9f3f699a1dea69e8d56be342ce9e4fae42908791252ebf63/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /var/run/containers/storage --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /var/run/libpod --exit-command-arg --runtime --exit-command-arg /usr/local/bin/crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg --rm --exit-command-arg d46051f8844b7d7e9f3f699a1dea69e8d56be342ce9e4fae42908791252ebf63]" INFO[0000] Running conmon under slice machine-libpod_pod_a99dc2fdf3a7870ee219b80f7cb9d39c21d6499363a860a8669725ea8057da63.slice and unitName libpod-conmon-d46051f8844b7d7e9f3f699a1dea69e8d56be342ce9e4fae42908791252ebf63.scope DEBU[0000] Received: -1 DEBU[0000] Cleaning up container d46051f8844b7d7e9f3f699a1dea69e8d56be342ce9e4fae42908791252ebf63 DEBU[0000] unmounted container "d46051f8844b7d7e9f3f699a1dea69e8d56be342ce9e4fae42908791252ebf63" DEBU[0000] Cleaning up container d46051f8844b7d7e9f3f699a1dea69e8d56be342ce9e4fae42908791252ebf63 DEBU[0000] Container d46051f8844b7d7e9f3f699a1dea69e8d56be342ce9e4fae42908791252ebf63 storage is already unmounted, skipping... DEBU[0000] Container d46051f8844b7d7e9f3f699a1dea69e8d56be342ce9e4fae42908791252ebf63 storage is already unmounted, skipping... DEBU[0000] ExitCode msg: "mount `mqueue` to `/dev/mqueue`: operation not permitted: oci runtime permission denied error" ERRO[0000] mount `mqueue` to `/dev/mqueue`: Operation not permitted: OCI runtime permission denied error ```

because it lacks definitions for symbols that are present in the current kernel, as you showed with the grep command above

Thank you, I understand this. My question was more "how can be that file be outdated, if the kernel is fine (seems recent enough to me)." and "Is there a (relatively) simple fix for that (eg. update the kernel headers)?"

giuseppe commented 3 years ago

so I had a look under Ubuntu 18.04 and there are some other missing definitions:

#define __NR_move_mount 429
#define __NR_fsopen 430
#define __NR_fsconfig 431
#define __NR_fsmount 432

It works when I add also these ones on top of the previous patches.

For a test, if you add these definitions to the same linux.c file, does it work for you?

These definitions are arch specific so unfortunately I don't think we can add the workaround in crun.

matpen commented 3 years ago

Still not working, but at least now we get a different error:

matteo@test1:/tmp/crun$ sudo podman --runtime /usr/local/bin/crun run  --rm  --pod test_pod --privileged --uidmap 0:0:10000 "docker.io/library/ubuntu:18.04" whoami
Error: create device `/dev/tty`: No such device or address: OCI runtime error

matteo@test1:/tmp/crun$ sudo podman --runtime /usr/local/bin/crun run  --rm  --pod test_pod --uidmap 0:0:10000 "docker.io/library/ubuntu:18.04" whoami
Error: sethostname: Operation not permitted: OCI runtime permission denied error

Note that the first error is the same you mentioned here, so I removed --privileged as you suggested. The second error is instead new to me.

giuseppe commented 3 years ago

indeeed, this is a new one now. Can you try adding --uts=private to the podman run command?

matpen commented 3 years ago

uhmm... I get the following. Too old podman perhaps?

matteo@test1:/tmp/crun$ sudo podman --runtime /usr/local/bin/crun run  --rm --pod test_pod --uidmap 0:0:10000 --uts=private "docker.io/library/ubuntu:18.04" whoami
Error: --uts "private" is not valid

matteo@test1:/tmp/crun$ podman --version
podman version 1.6.2
giuseppe commented 3 years ago

That is ancient, I thought you were on 2.2.1

matpen commented 3 years ago

No problem: I switched to a VM where I had tested 2.2.1, upgraded to 3.0.1, and followed the steps discussed today. And with success!

matteo@ubuntu-vbox:/tmp/crun$ uname -a
Linux ubuntu-vbox 5.4.0-70-generic #78~18.04.1-Ubuntu SMP Sat Mar 20 14:10:07 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

matteo@ubuntu-vbox:/tmp/crun$ podman --version
podman version 3.0.1

matteo@ubuntu-vbox:/tmp/crun$ crun --version
crun version 0.18.90-a72dd-dirty
commit: a72dd4d303f48e23f3a74a103216363167e4407f
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL

matteo@ubuntu-vbox:/tmp/crun$ sudo podman pod create --name test_pod
f54d71be5c18ad74a313922e6ae0614c1738d06267990bc929e21491796cfc76

matteo@ubuntu-vbox:/tmp/crun$ sudo podman --runtime /usr/local/bin/crun run  --rm  --pod test_pod --privileged --uidmap 0:0:10000 "docker.io/library/ubuntu:18.04" whoami
Error: OCI runtime error: create device `/dev/tty`: No such device or address

matteo@ubuntu-vbox:/tmp/crun$ sudo podman --runtime /usr/local/bin/crun run  --rm  --pod test_pod --uidmap 0:0:10000 "docker.io/library/ubuntu:18.04" whoami
root

So finally here is the working patch. Did you mention that it cannot be released with crun? Do you suggest applying it to my systems, or can you envision some more maintainable solution?

giuseppe commented 3 years ago

And with success!

Thanks for confirming!

or can you envision some more maintainable solution?

I think the correct fix is to get the updated header files on Ubuntu 18.04, so that crun can be compiled with the support for the new mount API.

matpen commented 3 years ago

How does one "get the updated header files"? It looks like they already are updated on my system. Is this something that the OS vendor should do?


matteo@ubuntu-vbox:~$ dpkg -s linux-headers-generic-hwe-18.04
Package: linux-headers-generic-hwe-18.04
Status: install ok installed
Priority: optional
Section: kernel
Installed-Size: 13
Maintainer: Ubuntu Kernel Team <kernel-team@lists.ubuntu.com>
Architecture: amd64
Source: linux-meta-hwe-5.4
Version: 5.4.0.70.78~18.04.63
Depends: linux-headers-5.4.0-70-generic
Description: Generic Linux kernel headers
 This package will always depend on the latest generic kernel headers
 available.
giuseppe commented 3 years ago

Is this something that the OS vendor should do?

yes that is not in our control.

You will need an updated /usr/include/linux/mount.h as well as having the definitions for the different syscalls I've listed above.

Could you try on a newer Ubuntu release?