search

containers / podman

Podman: A tool for managing OCI containers and pods.
https://podman.io
Apache License 2.0
23.73k stars 2.41k forks source link

[Regression] relative idmapped mounts prevent containers from starting in 4.4.0+ #17517

Closed Syquel closed 1 year ago

Syquel commented 1 year ago

Issue Description

If an idmapped mount is specified which is relative to the container user namespace the container cannot start in Podman 4.4.0+.
Example: test-volume:/mnt/test:idmap=uids=@0-1001-1;gids=@0-1001-1

The container startup is aborted with the following error message: Error: expected integer

This feature is documented in crun.1.md and worked in Podman 4.3.1.

Steps to reproduce the issue

Steps to reproduce the issue

  1. Install Podman 4.4.0+
  2. Run podman --debug run -it --rm --volume 'test-volume:/mnt/test:idmap=uids=@0-1001-1;gids=@0-1001-1' fedora:37

Describe the results you received

The container startup is aborted with the error message Error: expected integer.

Describe the results you expected

The container starts successfully and the volume is idmapped relatively to the container user namespace.

podman info output

host:
  arch: amd64
  buildahVersion: 1.29.0
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - rdma
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: /usr/bin/conmon is owned by conmon 1:2.1.6-1
    path: /usr/bin/conmon
    version: 'conmon version 2.1.6, commit: 158b5421dbac6bda96b1457955cf2e3c34af29bc'
  cpuUtilization:
    idlePercent: 97.42
    systemPercent: 0.75
    userPercent: 1.83
  cpus: 12
  distribution:
    distribution: arch
    version: unknown
  eventLogger: journald
  hostname: syquel.de
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.1.11-arch1-1
  linkmode: dynamic
  logDriver: journald
  memFree: 59464073216
  memTotal: 67230724096
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: /usr/bin/crun is owned by crun 1.8-1
    path: /usr/bin/crun
    version: |-
      crun version 1.8
      commit: 0356bf4aff9a133d655dc13b1d9ac9424706cac4
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: true
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: /usr/bin/slirp4netns is owned by slirp4netns 1.2.0-1
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.4
  swapFree: 0
  swapTotal: 0
  uptime: 9h 35m 35.00s (Approximately 0.38 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries: {}
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 467645579264
  graphRootUsed: 23679283200
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 58
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.4.1
  Built: 1676117906
  BuiltTime: Sat Feb 11 13:18:26 2023
  GitCommit: 34e8f3933242f2e566bbbbf343cf69b7d506c1cf-dirty
  GoVersion: go1.20
  Os: linux
  OsArch: linux/amd64
  Version: 4.4.1

Podman in a container

No

Privileged Or Rootless

Privileged

Upstream Latest Release

Yes

Additional environment details

No response

Additional information

Downgrading podman to version 4.3.1 resolves the issue.
Normal idmapped mounts which are not relative to the container user namespace are working in version 4.4.0.

Full debug log:

# podman --debug run -it --rm --volume 'test-volume:/mnt/test:idmap=uids=@0-1001-1;gids=@0-1001-1' fedora:37
INFO[0000] podman filtering at log level debug
DEBU[0000] Called run.PersistentPreRunE(podman --debug run -it --rm --volume test-volume:/mnt/test:idmap=uids=@0-1001-1;gids=@0-1001-1 fedora:37)
DEBU[0000] Using conmon: "/usr/bin/conmon"
DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /var/lib/containers/storage
DEBU[0000] Using run root /run/containers/storage
DEBU[0000] Using static dir /var/lib/containers/storage/libpod
DEBU[0000] Using tmp dir /run/libpod
DEBU[0000] Using volume path /var/lib/containers/storage/volumes
DEBU[0000] Using transient store: false
DEBU[0000] Set libpod namespace to ""
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that metacopy is being used
DEBU[0000] Cached value indicated that native-diff is not being used
INFO[0000] Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=true
DEBU[0000] Initializing event backend journald
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument
DEBU[0000] Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument
DEBU[0000] Using OCI runtime "/usr/bin/crun"
INFO[0000] Setting parallel job count to 37
DEBU[0000] Successfully loaded network podman1: &{podman1 7ad3417bb4f87d1cf26b28bdecf6ab907ff3fd6cce65f2eca81e310c2752978e bridge podman1 2022-10-26 10:32:45.354703422 +0200 CEST [{{{10.89.0.0 ffffff00}} 10.89.0.1 <nil>} {{{fd05:a84b:c2aa:96:: ffffffffffffffff0000000000000000}} fd05:a84b:c2aa:96::1 <nil>}] true false true [] map[] map[] map[driver:host-local]}
DEBU[0000] Successfully loaded 1 networks
DEBU[0000] Pulling image fedora:37 (policy: missing)
DEBU[0000] Looking up image "fedora:37" in local containers storage
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] }
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf"
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/00-shortnames.conf"
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/01-mirror.conf"
DEBU[0000] Trying "registry.fedoraproject.org/fedora:37" ...
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev]@19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a"
DEBU[0000] Found image "fedora:37" as "registry.fedoraproject.org/fedora:37" in local containers storage
DEBU[0000] Found image "fedora:37" as "registry.fedoraproject.org/fedora:37" in local containers storage ([overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev]@19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a)
DEBU[0000] exporting opaque data as blob "sha256:19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a"
DEBU[0000] Looking up image "registry.fedoraproject.org/fedora:37" in local containers storage
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] }
DEBU[0000] Trying "registry.fedoraproject.org/fedora:37" ...
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev]@19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a"
DEBU[0000] Found image "registry.fedoraproject.org/fedora:37" as "registry.fedoraproject.org/fedora:37" in local containers storage
DEBU[0000] Found image "registry.fedoraproject.org/fedora:37" as "registry.fedoraproject.org/fedora:37" in local containers storage ([overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev]@19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a)
DEBU[0000] exporting opaque data as blob "sha256:19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a"
DEBU[0000] User mount test-volume:/mnt/test options [idmap=uids=@0-1001-1;gids=@0-1001-1]
DEBU[0000] Looking up image "fedora:37" in local containers storage
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] }
DEBU[0000] Trying "registry.fedoraproject.org/fedora:37" ...
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev]@19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a"
DEBU[0000] Found image "fedora:37" as "registry.fedoraproject.org/fedora:37" in local containers storage
DEBU[0000] Found image "fedora:37" as "registry.fedoraproject.org/fedora:37" in local containers storage ([overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev]@19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a)
DEBU[0000] exporting opaque data as blob "sha256:19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a"
DEBU[0000] Inspecting image 19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a
DEBU[0000] exporting opaque data as blob "sha256:19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a"
DEBU[0000] exporting opaque data as blob "sha256:19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a"
DEBU[0000] Inspecting image 19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a
DEBU[0000] Inspecting image 19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a
DEBU[0000] Inspecting image 19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a
DEBU[0000] Inspecting image 19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a
DEBU[0000] using systemd mode: false
DEBU[0000] No hostname set; container's hostname will default to runtime default
DEBU[0000] Found apparmor_parser binary in /sbin/apparmor_parser
DEBU[0000] Loading seccomp profile from "/etc/containers/seccomp.json"
DEBU[0000] Allocated lock 0 for container 1486532d5b931a7da2ffc3d996f4d0e6f295ea140c89219dc54d6655dbde1aba
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev]@19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a"
DEBU[0000] exporting opaque data as blob "sha256:19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a"
DEBU[0000] Cached value indicated that idmapped mounts for overlay are supported
DEBU[0000] overlay: mount_data=lowerdir=/var/lib/containers/storage/overlay/l/UHZ6JQHRA2VH24WJWU2U56WK4L,upperdir=/var/lib/containers/storage/overlay/52385078191a0824270ab2ad04e299e834e0090d8d5a1f17d59c3215ed58a1f6/diff,workdir=/var/lib/containers/storage/overlay/52385078191a0824270ab2ad04e299e834e0090d8d5a1f17d59c3215ed58a1f6/work,nodev
DEBU[0000] Created container "1486532d5b931a7da2ffc3d996f4d0e6f295ea140c89219dc54d6655dbde1aba"
DEBU[0000] Container "1486532d5b931a7da2ffc3d996f4d0e6f295ea140c89219dc54d6655dbde1aba" has work directory "/var/lib/containers/storage/overlay-containers/1486532d5b931a7da2ffc3d996f4d0e6f295ea140c89219dc54d6655dbde1aba/userdata"
DEBU[0000] Container "1486532d5b931a7da2ffc3d996f4d0e6f295ea140c89219dc54d6655dbde1aba" has run directory "/run/containers/storage/overlay-containers/1486532d5b931a7da2ffc3d996f4d0e6f295ea140c89219dc54d6655dbde1aba/userdata"
DEBU[0000] Creating new volume test-volume for container
DEBU[0000] Validating options for local driver
DEBU[0000] Handling terminal attach
DEBU[0000] Cached value indicated that volatile is being used
DEBU[0000] overlay: mount_data=lowerdir=/var/lib/containers/storage/overlay/ef39eb2b8fa009a5049949b623a220b3a5529a951e97fd8fd76d4b9acea8f478/mapped/0/l/UHZ6JQHRA2VH24WJWU2U56WK4L,upperdir=/var/lib/containers/storage/overlay/ef39eb2b8fa009a5049949b623a220b3a5529a951e97fd8fd76d4b9acea8f478/diff,workdir=/var/lib/containers/storage/overlay/ef39eb2b8fa009a5049949b623a220b3a5529a951e97fd8fd76d4b9acea8f478/work,nodev,volatile
DEBU[0000] Mounted container "1486532d5b931a7da2ffc3d996f4d0e6f295ea140c89219dc54d6655dbde1aba" at "/var/lib/containers/storage/overlay/ef39eb2b8fa009a5049949b623a220b3a5529a951e97fd8fd76d4b9acea8f478/merged"
DEBU[0000] Going to mount named volume test-volume
DEBU[0000] Copying up contents from container 1486532d5b931a7da2ffc3d996f4d0e6f295ea140c89219dc54d6655dbde1aba to volume test-volume
DEBU[0000] Created root filesystem for container 1486532d5b931a7da2ffc3d996f4d0e6f295ea140c89219dc54d6655dbde1aba at /var/lib/containers/storage/overlay/ef39eb2b8fa009a5049949b623a220b3a5529a951e97fd8fd76d4b9acea8f478/merged
DEBU[0000] Using /sbin/apparmor_parser binary
INFO[0000] Successfully loaded AppAmor profile "containers-default-0.51.0"
DEBU[0000] Skipping unrecognized mount in /etc/containers/mounts.conf: "# Configuration file for default mounts in containers (see man 5"
DEBU[0000] Skipping unrecognized mount in /etc/containers/mounts.conf: "# containers-mounts.conf for further information)"
DEBU[0000] Skipping unrecognized mount in /etc/containers/mounts.conf: ""
DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode subscription
DEBU[0000] Cleaning up container 1486532d5b931a7da2ffc3d996f4d0e6f295ea140c89219dc54d6655dbde1aba
DEBU[0000] Network is already cleaned up, skipping...
DEBU[0000] Unmounted container "1486532d5b931a7da2ffc3d996f4d0e6f295ea140c89219dc54d6655dbde1aba"
DEBU[0000] Removing container 1486532d5b931a7da2ffc3d996f4d0e6f295ea140c89219dc54d6655dbde1aba
DEBU[0000] Cleaning up container 1486532d5b931a7da2ffc3d996f4d0e6f295ea140c89219dc54d6655dbde1aba
DEBU[0000] Network is already cleaned up, skipping...
DEBU[0000] Container 1486532d5b931a7da2ffc3d996f4d0e6f295ea140c89219dc54d6655dbde1aba storage is already unmounted, skipping...
DEBU[0000] Removing all exec sessions for container 1486532d5b931a7da2ffc3d996f4d0e6f295ea140c89219dc54d6655dbde1aba
DEBU[0000] Container 1486532d5b931a7da2ffc3d996f4d0e6f295ea140c89219dc54d6655dbde1aba storage is already unmounted, skipping...
DEBU[0000] ExitCode msg: "expected integer"
Error: expected integer
DEBU[0000] Shutting down engines
Syquel commented 1 year ago

After a little bit of digging through the source code I think this is related to the change in https://github.com/containers/podman/commit/fdcc2257df0fb0cb72d3fbe1b5aa8625955e1219 which expects the idmap option argument to be %d-%d-%d and does not account for relative idmappings. https://github.com/containers/podman/blob/fdcc2257df0fb0cb72d3fbe1b5aa8625955e1219/libpod/container_internal_common.go#L60-L75

This function is called in parseIDMapMountOption which should probably handle the relative idmapping. https://github.com/containers/podman/blob/fdcc2257df0fb0cb72d3fbe1b5aa8625955e1219/libpod/container_internal_common.go#L77-L118

giuseppe commented 1 year ago

before we were relying on an experimental feature in crun, since there was no support for it in OCI. Now that there is idmapping support in the OCI runtime specs, we are using that to pass down the information.

I'll take a look and see if it is possible to add the same feature to Podman

giuseppe commented 1 year ago

PR here: https://github.com/containers/podman/pull/17522

Syquel commented 1 year ago

Maybe this should not be implemented then and mounts / volumes should always be idmapped relatively to the user namespace.

I would prefer that solution because I can not imagine any use case where I would want to do an absolute idmapping or where that would even work.