search

containers / podman

Podman: A tool for managing OCI containers and pods.
https://podman.io
Apache License 2.0
22.98k stars 2.34k forks source link

Rootless podman in rootless podman container fails with inconsistent messages #20812

Closed adelton closed 7 months ago

adelton commented 9 months ago

Issue Description

I try to debug some rootless setups in OpenShift per https://www.redhat.com/sysadmin/podman-inside-kubernetes. I know it says

Disable SELinux: SELinux does not allow containerized processes to mount all of the file systems required to run inside a container. So we need to disable SELinux on the host that is running the Kubernetes cluster.

and I try to find out what exactly would fail, to possibly amend the SELinux / OpenShift policies. So it is expected that my attempt to run a rootless podman container in a rootless container fails.

However, the error message I get when running a rootless container in a rootless container seem not stable which is worrying.

Steps to reproduce the issue

Steps to reproduce the issue

  1. $ podman run --rm -ti --user podman quay.io/podman/stable
  2. [podman@8c2d1ecef2c7 /]$ podman run --rm -ti --user podman quay.io/podman/stable
  3. [podman@8c2d1ecef2c7 /]$ podman run --rm -ti --user podman quay.io/podman/stable

Describe the results you received

$ podman run --rm -ti --user podman quay.io/podman/stable
Trying to pull quay.io/podman/stable:latest...
Getting image source signatures
Copying blob 4d9671c387bd done   | 
Copying blob 6eb636413202 done   | 
Copying blob 2d9850dbb0db done   | 
Copying blob f2ead6108236 done   | 
Copying blob 85325264fc3e done   | 
Copying blob ec2b80c473bf done   | 
Copying blob 0cdb70f634e5 done   | 
Copying blob 5bf68aba73a4 done   | 
Copying blob 6df6e4a6e148 done   | 
Copying config d716b1dbdf done   | 
Writing manifest to image destination
WARN[0006] Path "/run/secrets/etc-pki-entitlement" from "/etc/containers/mounts.conf" doesn't exist, skipping 
WARN[0006] Path "/run/secrets/rhsm" from "/etc/containers/mounts.conf" doesn't exist, skipping 
Error: container create failed (no logs from conmon): conmon bytes "": readObjectStart: expect { or n, but found , error found in #0 byte of ...||..., bigger context ...||...
[podman@744b807a6ee9 /]$ podman run --rm -ti --user podman quay.io/podman/stable
WARN[0000] Path "/run/secrets/etc-pki-entitlement" from "/etc/containers/mounts.conf" doesn't exist, skipping 
WARN[0000] Path "/run/secrets/rhsm" from "/etc/containers/mounts.conf" doesn't exist, skipping 
Error: container create failed (no logs from conmon): conmon bytes "": readObjectStart: expect { or n, but found , error found in #0 byte of ...||..., bigger context ...||...
[podman@744b807a6ee9 /]$ podman run --rm -ti --user podman quay.io/podman/stable
WARN[0000] Path "/run/secrets/etc-pki-entitlement" from "/etc/containers/mounts.conf" doesn't exist, skipping 
WARN[0000] Path "/run/secrets/rhsm" from "/etc/containers/mounts.conf" doesn't exist, skipping 
Error: crun: set propagation for `proc`: Permission denied: OCI permission denied
[podman@744b807a6ee9 /]$ podman run --rm -ti --user podman quay.io/podman/stable
WARN[0000] Path "/run/secrets/etc-pki-entitlement" from "/etc/containers/mounts.conf" doesn't exist, skipping 
WARN[0000] Path "/run/secrets/rhsm" from "/etc/containers/mounts.conf" doesn't exist, skipping 
Error: container create failed (no logs from conmon): conmon bytes "": readObjectStart: expect { or n, but found , error found in #0 byte of ...||..., bigger context ...||...
[podman@744b807a6ee9 /]$ podman run --rm -ti --user podman quay.io/podman/stable
WARN[0000] Path "/run/secrets/etc-pki-entitlement" from "/etc/containers/mounts.conf" doesn't exist, skipping 
WARN[0000] Path "/run/secrets/rhsm" from "/etc/containers/mounts.conf" doesn't exist, skipping 
Error: container create failed (no logs from conmon): conmon bytes "": readObjectStart: expect { or n, but found , error found in #0 byte of ...||..., bigger context ...||...
[podman@744b807a6ee9 /]$ podman run --rm -ti --user podman quay.io/podman/stable
WARN[0000] Path "/run/secrets/etc-pki-entitlement" from "/etc/containers/mounts.conf" doesn't exist, skipping 
WARN[0000] Path "/run/secrets/rhsm" from "/etc/containers/mounts.conf" doesn't exist, skipping 
Error: container create failed (no logs from conmon): conmon bytes "": readObjectStart: expect { or n, but found , error found in #0 byte of ...||..., bigger context ...||...
[podman@744b807a6ee9 /]$ podman run --rm -ti --user podman quay.io/podman/stable
WARN[0000] Path "/run/secrets/etc-pki-entitlement" from "/etc/containers/mounts.conf" doesn't exist, skipping 
WARN[0000] Path "/run/secrets/rhsm" from "/etc/containers/mounts.conf" doesn't exist, skipping 
Error: crun: set propagation for `proc`: Permission denied: OCI permission denied

Describe the results you expected

I expect the error message to be the same every time.

podman info output

On the host:

$ podman info
host:
  arch: amd64
  buildahVersion: 1.32.0
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.8-2.fc39.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.8, commit: '
  cpuUtilization:
    idlePercent: 98.25
    systemPercent: 0.36
    userPercent: 1.39
  cpus: 2
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    version: "39"
  eventLogger: journald
  freeLocks: 2048
  hostname: cc-vm2p.tpb.lab.eng.brq.redhat.com
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 524288
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 524288
      size: 65536
  kernel: 6.5.12-300.fc39.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 1435463680
  memTotal: 3029131264
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.8.0-1.fc39.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.8.0
    package: netavark-1.8.0-2.fc39.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.8.0
  ociRuntime:
    name: crun
    package: crun-1.11.2-1.fc39.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.11.2
      commit: ab0edeef1c331840b025e8f1d38090cfb8a0509d
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20231107.g56d9f6d-1.fc39.x86_64
    version: |
      pasta 0^20231107.g56d9f6d-1.fc39.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: false
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-1.fc39.x86_64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 3028283392
  swapTotal: 3028283392
  uptime: 0h 43m 28.00s
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/test/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/test/.local/share/containers/storage
  graphRootAllocated: 16039018496
  graphRootUsed: 2292322304
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/test/.local/share/containers/storage/volumes
version:
  APIVersion: 4.7.2
  Built: 1698762721
  BuiltTime: Tue Oct 31 15:32:01 2023
  GitCommit: ""
  GoVersion: go1.21.1
  Os: linux
  OsArch: linux/amd64
  Version: 4.7.2

In the container:

[podman@744b807a6ee9 /]$ podman info
host:
  arch: amd64
  buildahVersion: 1.32.0
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.8-2.fc39.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.8, commit: '
  cpuUtilization:
    idlePercent: 98.01
    systemPercent: 0.42
    userPercent: 1.58
  cpus: 2
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    variant: container
    version: "39"
  eventLogger: file
  freeLocks: 2048
  hostname: 744b807a6ee9
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 1
      size: 999
    - container_id: 1000
      host_id: 1001
      size: 64535
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 1
      size: 999
    - container_id: 1000
      host_id: 1001
      size: 64535
  kernel: 6.5.12-300.fc39.x86_64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 871120896
  memTotal: 3029131264
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.8.0-1.fc39.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.8.0
    package: netavark-1.8.0-2.fc39.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.8.0
  ociRuntime:
    name: crun
    package: crun-1.11.2-1.fc39.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.11.2
      commit: ab0edeef1c331840b025e8f1d38090cfb8a0509d
      rundir: /tmp/podman-run-1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20231107.g56d9f6d-1.fc39.x86_64
    version: |
      pasta 0^20231107.g56d9f6d-1.fc39.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: false
    path: /tmp/podman-run-1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-1.fc39.x86_64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 3028283392
  swapTotal: 3028283392
  uptime: 0h 45m 21.00s
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/podman/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/podman/.local/share/containers/storage
  graphRootAllocated: 16039018496
  graphRootUsed: 2841436160
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /tmp/containers-user-1000/containers
  transientStore: false
  volumePath: /home/podman/.local/share/containers/storage/volumes
version:
  APIVersion: 4.7.2
  Built: 1698762721
  BuiltTime: Tue Oct 31 14:32:01 2023
  GitCommit: ""
  GoVersion: go1.21.1
  Os: linux
  OsArch: linux/amd64
  Version: 4.7.2


### Podman in a container

Yes

### Privileged Or Rootless

Rootless

### Upstream Latest Release

Yes

### Additional environment details

Additional environment details

### Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting
rhatdan commented 9 months ago

Try: podman run --rm -ti --security-opt unmask=/proc --user podman quay.io/podman/stable

We need to fix these Warnings about the /run/secrets directory.

rhatdan commented 9 months ago

Also can you do podman run --rm -ti -v /dev/null:/etc/containers/mounts.conf --security-opt unmask=/proc --user podman quay.io/podman/stable To see if this cleans it up.

adelton commented 9 months ago

Do you mean to try these for the first podman, or that podman-in-podman invocation?

rhatdan commented 9 months ago

First podman, Kernel does not allow a processes to modify a modified /proc.

adelton commented 9 months ago

The behaviour, including the nondeterminism, seems to still be there:

$ podman run --rm -ti -v /dev/null:/etc/containers/mounts.conf --security-opt unmask=/proc --user podman quay.io/podman/stable
[podman@5c678908c060 /]$ mount | grep proc
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /proc/acpi type tmpfs (ro,relatime,context="system_u:object_r:container_file_t:s0:c392,c957",size=0k,uid=1000,gid=1000,inode64)
devtmpfs on /proc/kcore type devtmpfs (ro,nosuid,seclabel,size=4096k,nr_inodes=4062785,mode=755,inode64)
devtmpfs on /proc/keys type devtmpfs (ro,nosuid,seclabel,size=4096k,nr_inodes=4062785,mode=755,inode64)
devtmpfs on /proc/latency_stats type devtmpfs (ro,nosuid,seclabel,size=4096k,nr_inodes=4062785,mode=755,inode64)
devtmpfs on /proc/timer_list type devtmpfs (ro,nosuid,seclabel,size=4096k,nr_inodes=4062785,mode=755,inode64)
tmpfs on /proc/scsi type tmpfs (ro,relatime,context="system_u:object_r:container_file_t:s0:c392,c957",size=0k,uid=1000,gid=1000,inode64)
proc on /proc/asound type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/bus type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/fs type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/irq type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/sys type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/sysrq-trigger type proc (ro,nosuid,nodev,noexec,relatime)
[podman@5c678908c060 /]$ mount | grep ' on /proc '
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
[podman@5c678908c060 /]$ podman run --rm -ti --user podman quay.io/podman/stable
Trying to pull quay.io/podman/stable:latest...
Getting image source signatures
Copying blob 77c9c34091b4 done   | 
Copying blob 1e910112bc7f done   | 
Copying blob 10327c9af971 done   | 
Copying blob 9fa763129095 done   | 
Copying blob 4e8a4684a6a4 done   | 
Copying blob 718a00fe3212 done   | 
Copying blob fd72a3378718 done   | 
Copying blob 3f30707a1d42 done   | 
Copying blob a681dc7022f8 done   | 
Copying config 314b296d26 done   | 
Writing manifest to image destination
Error: container create failed (no logs from conmon): conmon bytes "": readObjectStart: expect { or n, but found , error found in #0 byte of ...||..., bigger context ...||...
[podman@5c678908c060 /]$ podman run --rm -ti --user podman quay.io/podman/stable
Error: crun: set propagation for `proc`: Permission denied: OCI permission denied
dominic-p commented 7 months ago

In a possibly related issue, I'm seeing similar warning messages running quay.io/buildah/stable on my CRI-O kubernetes cluster:

WARN[0000] Path "/run/secrets/etc-pki-entitlement" from "/etc/containers/mounts.conf" doesn't exist, skipping 
WARN[0000] Path "/run/secrets/rhsm" from "/etc/containers/mounts.conf" doesn't exist, skipping
rhatdan commented 7 months ago

Yes we need to figure out how to fix this on systems that do not have subscption-manager installed on them, Probably drop the Warning to info.

adelton commented 7 months ago

This issue however was about that

Error: container create failed (no logs from conmon): conmon bytes "": readObjectStart: expect { or n, but found , error found in #0 byte of ...||..., bigger context ...||...

vs.

Error: crun: set propagation for `proc`: Permission denied: OCI permission denied

not about those mountpoint WARNs.

edsantiago commented 7 months ago

"no logs from conmon" is #10927, one of our longest-standing and most annoying flakes. I've never seen it on run, only exec. And I tried your reproducer on my f39 laptop, no luck.