search

containers / podman

Podman: A tool for managing OCI containers and pods.
https://podman.io
Apache License 2.0
23.74k stars 2.41k forks source link

OCI runtime error: crun: mount_setattr `/sys`: Function not implemented #18941

Closed doppelrittberger closed 1 year ago

doppelrittberger commented 1 year ago

Issue Description

Hi

we use rootless podman to run containers inside a gitlab pipeline. Unfortunately we get the following error when trying to run a container:

OCI runtime error: crun: mount_setattr `/sys`: Function not implemented

The same call works using a different environment/newer kernel version. Anything I missed to configure?

Thanks for your great support

Steps to reproduce the issue

Steps to reproduce the issue

  1. Run podman unprivileged inside kubernetes using https://github.com/doppelrittberger/podman-maven
  2. Run podman run ubi8 Hello World
  3. Fail

Describe the results you received

A failing container

Describe the results you expected

A running container

podman info output

host:
  arch: amd64
  buildahVersion: 1.30.0
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.1.7-2.fc38.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: '
  cpuUtilization:
    idlePercent: 97.47
    systemPercent: 0.19
    userPercent: 2.35
  cpus: 48
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    variant: container
    version: "38"
  eventLogger: file
  hostname: no-priv
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 1
      size: 999
    - container_id: 1000
      host_id: 1001
      size: 64535
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 1
      size: 999
    - container_id: 1000
      host_id: 1001
      size: 64535
  kernel: 4.18.0-425.13.1.el8_7.x86_64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 205506797568
  memTotal: 269619920896
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.8.5-1.fc38.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.5
      commit: b6f80f766c9a89eb7b1440c0a70ab287434b17ed
      rundir: /tmp/podman-run-1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  remoteSocket:
    path: /tmp/podman-run-1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-12.fc38.x86_64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 423h 19m 19.00s (Approximately 17.62 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/podman/.config/containers/storage.conf
  containerStore:
    number: 9
    paused: 0
    running: 0
    stopped: 9
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/podman/.local/share/containers/storage
  graphRootAllocated: 1530142773248
  graphRootUsed: 34491006976
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 2
  runRoot: /tmp/containers-user-1000/containers
  transientStore: false
  volumePath: /home/podman/.local/share/containers/storage/volumes
version:
  APIVersion: 4.5.1
  Built: 1685123928
  BuiltTime: Fri May 26 17:58:48 2023
  GitCommit: ""
  GoVersion: go1.20.4
  Os: linux
  OsArch: linux/amd64
  Version: 4.5.1

Podman in a container

Yes

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

Additional environment details

Additional information

Additional debug log:


[podman@no-priv /]$ podman run --log-level debug nexus.custom.com:8220/debian echo Hello
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called run.PersistentPreRunE(podman run --log-level debug nexus.custom.com:8220/debian echo Hello) 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /home/podman/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /home/podman/.local/share/containers/storage 
DEBU[0000] Using run root /tmp/containers-user-1000/containers 
DEBU[0000] Using static dir /home/podman/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /tmp/podman-run-1000/libpod/tmp 
DEBU[0000] Using volume path /home/podman/.local/share/containers/storage/volumes 
DEBU[0000] Using transient store: false                 
DEBU[0000] Not configuring container store              
DEBU[0000] Initializing event backend file              
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument 
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument 
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument 
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument 
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument 
DEBU[0000] Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument 
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
INFO[0000] Setting parallel job count to 145            
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called run.PersistentPreRunE(podman run --log-level debug nexus.custom.com:8220/debian echo Hello) 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /home/podman/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Overriding run root "/tmp/podman-run-1000/containers" with "/tmp/containers-user-1000/containers" from database 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /home/podman/.local/share/containers/storage 
DEBU[0000] Using run root /tmp/containers-user-1000/containers 
DEBU[0000] Using static dir /home/podman/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /tmp/podman-run-1000/libpod/tmp 
DEBU[0000] Using volume path /home/podman/.local/share/containers/storage/volumes 
DEBU[0000] Using transient store: false                 
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Cached value indicated that metacopy is not being used 
DEBU[0000] Cached value indicated that native-diff is usable 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false 
DEBU[0000] Initializing event backend file              
DEBU[0000] Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument 
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument 
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument 
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument 
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument 
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
INFO[0000] Setting parallel job count to 145            
INFO[0000] Failed to detect the owner for the current cgroup: stat /sys/fs/cgroup/systemd/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88562d6b_46bd_4c34_bc68_71ad84b97f0e.slice/cri-containerd-21a3363764595bff2f2e430f5a4a166b2266e525aadda2a34916f158e44ab87f.scope: no such file or directory 
DEBU[0000] Successfully loaded 1 networks               
DEBU[0000] Pulling image nexus.custom.com:8220/debian (policy: missing) 
DEBU[0000] Looking up image "nexus.custom.com:8220/debian" in local containers storage 
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] } 
DEBU[0000] Trying "nexus.custom.com:8220/debian:latest" ... 
DEBU[0000] parsed reference into "[overlay@/home/podman/.local/share/containers/storage+/tmp/containers-user-1000/containers]@49081a1edb0b55df1967387e4c234add2d3f8ef0dc1f4953e7eaf552dc761c5a" 
DEBU[0000] Found image "nexus.custom.com:8220/debian" as "nexus.custom.com:8220/debian:latest" in local containers storage 
DEBU[0000] Found image "nexus.custom.com:8220/debian" as "nexus.custom.com:8220/debian:latest" in local containers storage ([overlay@/home/podman/.local/share/containers/storage+/tmp/containers-user-1000/containers]@49081a1edb0b55df1967387e4c234add2d3f8ef0dc1f4953e7eaf552dc761c5a) 
DEBU[0000] exporting opaque data as blob "sha256:49081a1edb0b55df1967387e4c234add2d3f8ef0dc1f4953e7eaf552dc761c5a" 
DEBU[0000] Looking up image "nexus.custom.com:8220/debian:latest" in local containers storage 
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] } 
DEBU[0000] Trying "nexus.custom.com:8220/debian:latest" ... 
DEBU[0000] parsed reference into "[overlay@/home/podman/.local/share/containers/storage+/tmp/containers-user-1000/containers]@49081a1edb0b55df1967387e4c234add2d3f8ef0dc1f4953e7eaf552dc761c5a" 
DEBU[0000] Found image "nexus.custom.com:8220/debian:latest" as "nexus.custom.com:8220/debian:latest" in local containers storage 
DEBU[0000] Found image "nexus.custom.com:8220/debian:latest" as "nexus.custom.com:8220/debian:latest" in local containers storage ([overlay@/home/podman/.local/share/containers/storage+/tmp/containers-user-1000/containers]@49081a1edb0b55df1967387e4c234add2d3f8ef0dc1f4953e7eaf552dc761c5a) 
DEBU[0000] exporting opaque data as blob "sha256:49081a1edb0b55df1967387e4c234add2d3f8ef0dc1f4953e7eaf552dc761c5a" 
DEBU[0000] User mount /proc:/proc options []            
DEBU[0000] Looking up image "nexus.custom.com:8220/debian" in local containers storage 
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] } 
DEBU[0000] Trying "nexus.custom.com:8220/debian:latest" ... 
DEBU[0000] parsed reference into "[overlay@/home/podman/.local/share/containers/storage+/tmp/containers-user-1000/containers]@49081a1edb0b55df1967387e4c234add2d3f8ef0dc1f4953e7eaf552dc761c5a" 
DEBU[0000] Found image "nexus.custom.com:8220/debian" as "nexus.custom.com:8220/debian:latest" in local containers storage 
DEBU[0000] Found image "nexus.custom.com:8220/debian" as "nexus.custom.com:8220/debian:latest" in local containers storage ([overlay@/home/podman/.local/share/containers/storage+/tmp/containers-user-1000/containers]@49081a1edb0b55df1967387e4c234add2d3f8ef0dc1f4953e7eaf552dc761c5a) 
DEBU[0000] exporting opaque data as blob "sha256:49081a1edb0b55df1967387e4c234add2d3f8ef0dc1f4953e7eaf552dc761c5a" 
DEBU[0000] Inspecting image 49081a1edb0b55df1967387e4c234add2d3f8ef0dc1f4953e7eaf552dc761c5a 
DEBU[0000] exporting opaque data as blob "sha256:49081a1edb0b55df1967387e4c234add2d3f8ef0dc1f4953e7eaf552dc761c5a" 
DEBU[0000] exporting opaque data as blob "sha256:49081a1edb0b55df1967387e4c234add2d3f8ef0dc1f4953e7eaf552dc761c5a" 
DEBU[0000] Inspecting image 49081a1edb0b55df1967387e4c234add2d3f8ef0dc1f4953e7eaf552dc761c5a 
DEBU[0000] Inspecting image 49081a1edb0b55df1967387e4c234add2d3f8ef0dc1f4953e7eaf552dc761c5a 
DEBU[0000] Inspecting image 49081a1edb0b55df1967387e4c234add2d3f8ef0dc1f4953e7eaf552dc761c5a 
DEBU[0000] Inspecting image 49081a1edb0b55df1967387e4c234add2d3f8ef0dc1f4953e7eaf552dc761c5a 
DEBU[0000] User mount /proc:/proc options []            
DEBU[0000] using systemd mode: false                    
DEBU[0000] Loading seccomp profile from "/usr/share/containers/seccomp.json" 
DEBU[0000] Adding mount /dev                            
DEBU[0000] Adding mount /dev/pts                        
DEBU[0000] Adding mount /sys                            
DEBU[0000] Adding mount /dev/mqueue                     
DEBU[0000] Adding mount /sys/fs/cgroup                  
DEBU[0000] Allocated lock 9 for container f07a3f8b55814ae9ea78707c1e966b759ba141c11188b9f0f999228af3fa2fa0 
DEBU[0000] parsed reference into "[overlay@/home/podman/.local/share/containers/storage+/tmp/containers-user-1000/containers]@49081a1edb0b55df1967387e4c234add2d3f8ef0dc1f4953e7eaf552dc761c5a" 
DEBU[0000] exporting opaque data as blob "sha256:49081a1edb0b55df1967387e4c234add2d3f8ef0dc1f4953e7eaf552dc761c5a" 
DEBU[0000] Cached value indicated that idmapped mounts for overlay are not supported 
DEBU[0000] Check for idmapped mounts support            
DEBU[0000] Created container "f07a3f8b55814ae9ea78707c1e966b759ba141c11188b9f0f999228af3fa2fa0" 
DEBU[0000] Container "f07a3f8b55814ae9ea78707c1e966b759ba141c11188b9f0f999228af3fa2fa0" has work directory "/home/podman/.local/share/containers/storage/overlay-containers/f07a3f8b55814ae9ea78707c1e966b759ba141c11188b9f0f999228af3fa2fa0/userdata" 
DEBU[0000] Container "f07a3f8b55814ae9ea78707c1e966b759ba141c11188b9f0f999228af3fa2fa0" has run directory "/tmp/containers-user-1000/containers/overlay-containers/f07a3f8b55814ae9ea78707c1e966b759ba141c11188b9f0f999228af3fa2fa0/userdata" 
DEBU[0000] Not attaching to stdin                       
INFO[0000] Received shutdown.Stop(), terminating!        PID=1180
DEBU[0000] Enabling signal proxying                     
DEBU[0000] overlay: mount_data=lowerdir=/home/podman/.local/share/containers/storage/overlay/l/4UBW7XPPJ3TMHUDR4DPJVBQNVW,upperdir=/home/podman/.local/share/containers/storage/overlay/8b45f12dc022ff11d420fd6df3fc35a1b7263c076bdecb7e753b01f186f66666/diff,workdir=/home/podman/.local/share/containers/storage/overlay/8b45f12dc022ff11d420fd6df3fc35a1b7263c076bdecb7e753b01f186f66666/work,,userxattr 
DEBU[0000] Made network namespace at /tmp/podman-run-1000/netns/netns-da9bfc39-b09f-045d-aec1-fc97e18f06bf for container f07a3f8b55814ae9ea78707c1e966b759ba141c11188b9f0f999228af3fa2fa0 
DEBU[0000] Mounted container "f07a3f8b55814ae9ea78707c1e966b759ba141c11188b9f0f999228af3fa2fa0" at "/home/podman/.local/share/containers/storage/overlay/8b45f12dc022ff11d420fd6df3fc35a1b7263c076bdecb7e753b01f186f66666/merged" 
DEBU[0000] Created root filesystem for container f07a3f8b55814ae9ea78707c1e966b759ba141c11188b9f0f999228af3fa2fa0 at /home/podman/.local/share/containers/storage/overlay/8b45f12dc022ff11d420fd6df3fc35a1b7263c076bdecb7e753b01f186f66666/merged 
DEBU[0000] slirp4netns command: /usr/bin/slirp4netns --disable-host-loopback --mtu=65520 --enable-sandbox --enable-seccomp --enable-ipv6 -c -e 3 -r 4 --netns-type=path /tmp/podman-run-1000/netns/netns-da9bfc39-b09f-045d-aec1-fc97e18f06bf tap0 
WARN[0000] failed to set net.ipv6.conf.default.accept_dad sysctl: open /proc/sys/net/ipv6/conf/default/accept_dad: read-only file system 
DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode subscription 
DEBU[0000] reading hooks from /usr/share/containers/oci/hooks.d 
DEBU[0000] Workdir "/" resolved to host path "/home/podman/.local/share/containers/storage/overlay/8b45f12dc022ff11d420fd6df3fc35a1b7263c076bdecb7e753b01f186f66666/merged" 
DEBU[0000] Created OCI spec for container f07a3f8b55814ae9ea78707c1e966b759ba141c11188b9f0f999228af3fa2fa0 at /home/podman/.local/share/containers/storage/overlay-containers/f07a3f8b55814ae9ea78707c1e966b759ba141c11188b9f0f999228af3fa2fa0/userdata/config.json 
DEBU[0000] /usr/bin/conmon messages will be logged to syslog 
DEBU[0000] Running with no Cgroups                      
DEBU[0000] running conmon: /usr/bin/conmon               args="[--api-version 1 -c f07a3f8b55814ae9ea78707c1e966b759ba141c11188b9f0f999228af3fa2fa0 -u f07a3f8b55814ae9ea78707c1e966b759ba141c11188b9f0f999228af3fa2fa0 -r /usr/bin/crun -b /home/podman/.local/share/containers/storage/overlay-containers/f07a3f8b55814ae9ea78707c1e966b759ba141c11188b9f0f999228af3fa2fa0/userdata -p /tmp/containers-user-1000/containers/overlay-containers/f07a3f8b55814ae9ea78707c1e966b759ba141c11188b9f0f999228af3fa2fa0/userdata/pidfile -n serene_chaum --exit-dir /tmp/podman-run-1000/libpod/tmp/exits --full-attach -l k8s-file:/home/podman/.local/share/containers/storage/overlay-containers/f07a3f8b55814ae9ea78707c1e966b759ba141c11188b9f0f999228af3fa2fa0/userdata/ctr.log --log-level debug --syslog --runtime-arg --cgroup-manager --runtime-arg disabled --conmon-pidfile /tmp/containers-user-1000/containers/overlay-containers/f07a3f8b55814ae9ea78707c1e966b759ba141c11188b9f0f999228af3fa2fa0/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /home/podman/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /tmp/containers-user-1000/containers --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg cgroupfs --exit-command-arg --tmpdir --exit-command-arg /tmp/podman-run-1000/libpod/tmp --exit-command-arg --network-config-dir --exit-command-arg  --exit-command-arg --network-backend --exit-command-arg netavark --exit-command-arg --volumepath --exit-command-arg /home/podman/.local/share/containers/storage/volumes --exit-command-arg --db-backend --exit-command-arg boltdb --exit-command-arg --transient-store=false --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --events-backend --exit-command-arg file --exit-command-arg --syslog --exit-command-arg container --exit-command-arg cleanup --exit-command-arg f07a3f8b55814ae9ea78707c1e966b759ba141c11188b9f0f999228af3fa2fa0]"
[conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied

DEBU[0000] Received: -1                                 
DEBU[0000] Cleaning up container f07a3f8b55814ae9ea78707c1e966b759ba141c11188b9f0f999228af3fa2fa0 
DEBU[0000] Tearing down network namespace at /tmp/podman-run-1000/netns/netns-da9bfc39-b09f-045d-aec1-fc97e18f06bf for container f07a3f8b55814ae9ea78707c1e966b759ba141c11188b9f0f999228af3fa2fa0 
DEBU[0000] Unmounted container "f07a3f8b55814ae9ea78707c1e966b759ba141c11188b9f0f999228af3fa2fa0" 
DEBU[0000] ExitCode msg: "crun: [conmon:d]: failed to write to /proc/self/oom_score_adj: permission denied\n\nmount_setattr `/sys`: function not implemented: oci permission denied" 
Error: crun: [conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied

mount_setattr `/sys`: Function not implemented: OCI permission denied
DEBU[0000] Shutting down engines
Luap99 commented 1 year ago

@giuseppe PTAL

mount_setattr was added in 5.12 according to the man page.

giuseppe commented 1 year ago

I think it is caused by https://github.com/containers/crun/commit/908bfc43087a845ba3f779cfed933e2a539b714a, that is an intentional change. To mount a fresh sysfs, /sys must be fully visible in the current context, which is not the case when running an unprivileged pod. If we bind mount it, we risk to expose the cgroup file system as writeable (in your case it would not matter since anyway you are in a container).

I suggest to workaround it either with --net host or -v /sys:/sys since we know you are already in a container, so there is not much you need to protect the nested container from