Rootless Overlay Fails With Userns

[cjreyn]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Running a rootless Centos:7 container with the "Development Tools" group package installed, produces an odd error for certain uid/gid maps. This issue only occurs with overlayfs and user --userns=keep-id. The error is:

Error: error creating container storage: creating an ID-mapped copy of layer "92be6f86914eb84e250d3ee9524503fdc5f2e7d6d92fc272776de15fe55f0844": error during chown: link var/lib/yum/yumdb/a/11022c68e971dbfeaa906d125086652491b1fcbd-apr-util-1.5.2-6.el7-x86_64/command_line var/lib/yum/yumdb/f/cb7e013b0931dc495c9295d40ffbd0f49e31484b-fipscheck-lib-1.4.1-6.el7-x86_64/command_line: no such file or directory: exit status 1

Steps to reproduce the issue:

As root:

1. Add a new user and group with high ID: groupadd -g 1214323 podmantest2 adduser -u 1214323 podmantest2

2. Add user to /etc/subuid and /etc/subgid with offsets: echo "podmantest2:10000000:65536" >> /etc/subuid echo "podmantest2:10000000:65536" >> /etc/subgid

3. Make sure dirs exists mkdir /run/user/1214323 chown podmantest2:podmantest2 /run/user/1214323 mkdir -p /scratch/podmantest2/podman chown podmantest2:podmantest2 /scratch/podmantest2/podman

4. Configure per user podman config: /home/podmantest2/.config/containers/libpod.conf

[podmantest2@vws-999 ~]$ cat .config/containers/libpod.conf 
volume_path = "/scratch/podmantest2/podman"
image_default_transport = "docker://"
runtime = "crun"
conmon_path = ["/usr/libexec/podman/conmon", "/usr/local/lib/podman/conmon", "/usr/bin/conmon", "/usr/sbin/conmon", "/usr/local/bin/conmon", "/usr/local/sbin/conmon"]
conmon_env_vars = ["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"]
cgroup_manager = "cgroupfs"
init_path = "/usr/libexec/podman/catatonit"
static_dir = "/scratch/podmantest2/podman"
tmp_dir = "/run/user/1214323/libpod/tmp"
max_log_size = -1
no_pivot_root = false
cni_config_dir = "/etc/cni/net.d/"
cni_plugin_dir = ["/usr/libexec/cni", "/usr/lib/cni", "/usr/local/lib/cni", "/opt/cni/bin"]
infra_image = "k8s.gcr.io/pause:3.1"
infra_command = "/pause"
enable_port_reservation = true
label = true
network_cmd_path = ""
num_locks = 2048
events_logger = "journald"
EventsLogFilePath = ""
detach_keys = "ctrl-p,ctrl-q"

[runtimes]
#  runc = ["/usr/bin/runc", "/usr/sbin/runc", "/usr/local/bin/runc", "/usr/local/sbin/runc", "/sbin/runc", "/bin/runc", "/usr/lib/cri-o-runc/sbin/runc"]
  crun = ["/usr/bin/crun", "/usr/sbin/crun", "/usr/local/bin/crun", "/usr/local/sbin/crun", "/sbin/crun", "/bin/crun", "/usr/lib/cri-o-runc/sbin/crun"]

/home/podmantest/.config/containers/storage.conf

[podmantest2@vws-999 ~]$ cat .config/containers/storage.conf 
[storage]
  driver = "overlay"
  runroot = "/run/user/1214323"
  graphroot = "/scratch/podmantest2/podman"
  [storage.options]
    size = ""
    remap-uids = ""
    remap-gids = ""
    remap-user = ""
    remap-group = ""
    skip_mount_home = ""
    mount_program = "/bin/fuse-overlayfs"
    mountopt = ""
    [storage.options.overlay]
      mountopt = "nodev, index=off"
      ignore_chown_errors = "true"
    [storage.options.thinpool]
      autoextend_percent = ""
      autoextend_threshold = ""
      basesize = ""
      blocksize = ""
      directlvm_device = ""
      directlvm_device_force = ""
      fs = ""
      log_level = ""
      min_free_space = ""
      mkfsarg = ""
      mountopt = ""
      use_deferred_deletion = ""
      use_deferred_removal = ""
      xfs_nospace_max_retries = ""

/home/podmantest/.config/containers/containers.conf

[containers]
default_capabilities = [
    "NET_RAW",
  "CHOWN",
  "DAC_OVERRIDE",
  "FOWNER",
  "FSETID",
  "KILL",
  "NET_BIND_SERVICE",
  "SETFCAP",
  "SETGID",
  "SETPCAP",
  "SETUID",
  "SYS_CHROOT"
]
default_sysctls = [
  "net.ipv4.ping_group_range=0 0",
]
log_driver = "k8s-file"
[secrets]
[secrets.opts]
[network]
network_backend = "cni"
[engine]
events_logger = "file"
runtime = "crun"
[engine.runtimes]
[engine.volume_plugins]
[machine]

5. Build the example container:

   podman build -t bugtest -f - . <<EOF
   FROM centos:7
   RUN yum groupinstall -y "Development Tools"
   EOF

6. Run the container with --userns=keep-id: podman run -it --userns=keep-id bugtest

Describe the results you received: Error: error creating container storage: creating an ID-mapped copy of layer "92be6f86914eb84e250d3ee9524503fdc5f2e7d6d92fc272776de15fe55f0844": error during chown: link var/lib/yum/yumdb/a/11022c68e971dbfeaa906d125086652491b1fcbd-apr-util-1.5.2-6.el7-x86_64/command_line var/lib/yum/yumdb/f/cb7e013b0931dc495c9295d40ffbd0f49e31484b-fipscheck-lib-1.4.1-6.el7-x86_64/command_line: no such file or directory: exit status 1

Describe the results you expected: When running the same container with a different user called "podmantest" which has a uid/gid of 1000, the container starts fine:

[podmantest@vws-999:~/.config/containers]$ podman run -it --userns=keep-id bugtest
bash-4.2$ id
uid=1000(podmantest) gid=1000(podmantest) groups=1000(podmantest)

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

[podmantest2@vws-999 ~]$ podman version
Client:       Podman Engine
Version:      4.2.0
API Version:  4.2.0
Go Version:   go1.18.7
Built:        Wed Oct 26 20:23:47 2022
OS/Arch:      linux/amd64

Output of podman info:

[podmantest2@vws-999 ~]$ podman info
host:
  arch: amd64
  buildahVersion: 1.27.1
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.1.4-1.module+el8.7.0+17064+3b31f55c.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.4, commit: 64e1fe3ac604668d46b6efda338a9ba5a9b91b98'
  cpuUtilization:
    idlePercent: 97.22
    systemPercent: 0.9
    userPercent: 1.88
  cpus: 4
  distribution:
    distribution: '"rhel"'
    version: "8.7"
  eventLogger: file
  hostname: vws-999.diamond.ac.uk
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1214323
      size: 1
    - container_id: 1
      host_id: 10000000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1214323
      size: 1
    - container_id: 1
      host_id: 10000000
      size: 65536
  kernel: 4.18.0-372.9.1.el8.x86_64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 2507264000
  memTotal: 16595484672
  networkBackend: cni
  ociRuntime:
    name: crun
    package: crun-1.5-1.module+el8.7.0+17064+3b31f55c.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.5
      commit: 54ebb8ca8bf7e6ddae2eb919f5b82d1d96863dea
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1214323/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-2.module+el8.7.0+17064+3b31f55c.x86_64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.2
  swapFree: 8584388608
  swapTotal: 8589930496
  uptime: 3364h 19m 53.00s (Approximately 140.17 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /home/podmantest2/.config/containers/storage.conf
  containerStore:
    number: 3
    paused: 0
    running: 0
    stopped: 3
  graphDriverName: overlay
  graphOptions:
    overlay.ignore_chown_errors: "true"
    overlay.mount_program:
      Executable: /bin/fuse-overlayfs
      Package: fuse-overlayfs-1.9-1.module+el8.7.0+17064+3b31f55c.x86_64
      Version: |-
        fusermount3 version: 3.3.0
        fuse-overlayfs: version 1.9
        FUSE library version 3.3.0
        using FUSE kernel interface version 7.26
    overlay.mountopt: nodev, index=off
  graphRoot: /scratch/podmantest2/podman
  graphRootAllocated: 75984642048
  graphRootUsed: 3171459072
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 2
  runRoot: /run/user/1214323
  volumePath: /scratch/podmantest2/podman/volumes
version:
  APIVersion: 4.2.0
  Built: 1666812227
  BuiltTime: Wed Oct 26 20:23:47 2022
  GitCommit: ""
  GoVersion: go1.18.7
  Os: linux
  OsArch: linux/amd64
  Version: 4.2.0

Package info (e.g. output of rpm -q podman or apt list podman or brew info podman):

[root@vws-999 ~]# rpm -q podman
podman-4.2.0-4.module+el8.7.0+17064+3b31f55c.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

Virtual machine running Rhel 8.7. Issue also exhibits on Kernel 425.3.1.el8 and other workstations.

[mheon]

@giuseppe PTAL

[giuseppe]

mount_program = "/bin/fuse-overlayfs"

any reason for using fuse-overlayfs instead of native overlay?

[cjreyn]

Hi @giuseppe we set it to deal with https://github.com/containers/podman/blob/main/troubleshooting.md#15-rootless-podman-build-fails-when-using-overlayfs

[giuseppe]

opened a PR: https://github.com/containers/fuse-overlayfs/pull/381