podman run unable to start container in rootless mode

[ntsbtz]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description podman run -dt -p 8080:80/tcp docker.io/library/httpd throwing below error while running in rootless mode Error: OCI runtime error: runc: runc create failed: unable to start container process: chdir to cwd ("/usr/local/apache2") set in config.json failed: operation not supported

Steps to reproduce the issue:

1.Login to rootless user

2.Pull the httpd images

3.podman run -dt -p 8080:80/tcp docker.io/library/httpd

Describe the results you received: Error: OCI runtime error: runc: runc create failed: unable to start container process: chdir to cwd ("/usr/local/apache2") set in config.json failed: operation not supported

Describe the results you expected:

It should start the containers successfully Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Client:       Podman Engine
Version:      4.2.0
API Version:  4.2.0
Go Version:   go1.18.1
Built:        Wed Sep  7 10:32:56 2022
OS/Arch:      linux/amd64

Output of podman info:

host:
  arch: amd64
  buildahVersion: 1.27.0
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.1.4-1.ph4.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.4, commit: '
  cpuUtilization:
    idlePercent: 99.95
    systemPercent: 0.04
    userPercent: 0.01
  cpus: 16
  distribution:
    distribution: photon
    version: "4.0"
  eventLogger: journald
  hostname: testvm
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 100
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.10.118-14.ph4
  linkmode: dynamic
  logDriver: journald
  memFree: 1526992896
  memTotal: 4130152448
  networkBackend: cni
  ociRuntime:
    name: runc
    package: runc-1.1.1-3.ph4.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.1.1
      spec: 1.0.2-dev
      go: go1.18.1
      libseccomp: 2.5.0
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-1.ph4.x86_64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.0
  swapFree: 0
  swapTotal: 0
  uptime: 428h 57m 26.00s (Approximately 17.83 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries: {}
store:
  configFile: /home/regusr/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/regusr/.local/share/containers/storage
  graphRootAllocated: 33620226048
  graphRootUsed: 2711851008
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /run/user/1000/containers
  volumePath: /home/regusr/.local/share/containers/storage/volumes
version:
  APIVersion: 4.2.0
  Built: 1662546776
  BuiltTime: Wed Sep  7 10:32:56 2022
  GitCommit: ""
  GoVersion: go1.18.1
  Os: linux
  OsArch: linux/amd64
  Version: 4.2.0

Package info (e.g. output of rpm -q podman or apt list podman):

podman-4.2.0-1.ph4.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes/No

Additional environment details (AWS, VirtualBox, physical, etc.):

[vrothberg]

Thanks for reaching out, @ntsbtz!

I fail to reproduce. @giuseppe, do you have suspicion what may be causing the issue?

[giuseppe]

it could be a weird seccomp profile, or the underlying file system.

@ntsbtz could you please share the output of stat -f /home/regusr/.local/share/containers/storage and cat /proc/self/mountinfo?

Could you also try adding --security-opt seccomp=unconfined after podman run?

[ntsbtz]

it could be a weird seccomp profile, or the underlying file system.

@ntsbtz could you please share the output of stat -f /home/regusr/.local/share/containers/storage and cat /proc/self/mountinfo?

Could you also try adding --security-opt seccomp=unconfined after podman run?

Hi @giuseppe thanks for your quick response. Please check the output for below commands:

1. stat -f /home/regusr/.local/share/containers/storage/

   File: "/home/regusr/.local/share/containers/storage/" ID: d3ffaa436be39555 Namelen: 255 Type: ext2/ext3 Block size: 4096 Fundamental block size: 4096 Blocks: Total: 8208063 Free: 7545988 Available: 7122654 Inodes: Total: 2097152 Free: 2038462

2. cat /proc/self/mountinfo

22 93 0:20 / /proc rw,nosuid,nodev,noexec,relatime shared:23 - proc proc rw 23 93 0:21 / /sys rw,nosuid,nodev,noexec,relatime shared:2 - sysfs sysfs rw 24 93 0:5 / /dev rw,nosuid,noexec shared:19 - devtmpfs devtmpfs rw,size=4096k,nr_inodes=65536,mode=755 25 23 0:6 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime shared:3 - securityfs securityfs rw 26 24 0:22 / /dev/shm rw,nosuid,nodev shared:20 - tmpfs tmpfs rw 27 24 0:23 / /dev/pts rw,nosuid,noexec,relatime shared:21 - devpts devpts rw,gid=5,mode=620,ptmxmode=000 28 93 0:24 / /run rw,nosuid,nodev shared:22 - tmpfs tmpfs rw,size=806672k,nr_inodes=819200,mode=755 29 23 0:25 / /sys/fs/cgroup ro,nosuid,nodev,noexec shared:4 - tmpfs tmpfs ro,size=4096k,nr_inodes=1024,mode=755 30 29 0:26 / /sys/fs/cgroup/systemd rw,nosuid,nodev,noexec,relatime shared:5 - cgroup cgroup rw,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 31 23 0:27 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime shared:17 - pstore pstore rw 32 23 0:28 / /sys/fs/bpf rw,nosuid,nodev,noexec,relatime shared:18 - bpf none rw,mode=700 33 29 0:29 / /sys/fs/cgroup/freezer rw,nosuid,nodev,noexec,relatime shared:6 - cgroup cgroup rw,freezer 34 29 0:30 / /sys/fs/cgroup/cpu,cpuacct rw,nosuid,nodev,noexec,relatime shared:7 - cgroup cgroup rw,cpu,cpuacct 35 29 0:31 / /sys/fs/cgroup/devices rw,nosuid,nodev,noexec,relatime shared:8 - cgroup cgroup rw,devices 36 29 0:32 / /sys/fs/cgroup/net_cls,net_prio rw,nosuid,nodev,noexec,relatime shared:9 - cgroup cgroup rw,net_cls,net_prio 37 29 0:33 / /sys/fs/cgroup/perf_event rw,nosuid,nodev,noexec,relatime shared:10 - cgroup cgroup rw,perf_event 38 29 0:34 / /sys/fs/cgroup/memory rw,nosuid,nodev,noexec,relatime shared:11 - cgroup cgroup rw,memory 39 29 0:35 / /sys/fs/cgroup/cpuset rw,nosuid,nodev,noexec,relatime shared:12 - cgroup cgroup rw,cpuset 40 29 0:36 / /sys/fs/cgroup/blkio rw,nosuid,nodev,noexec,relatime shared:13 - cgroup cgroup rw,blkio 41 29 0:37 / /sys/fs/cgroup/pids rw,nosuid,nodev,noexec,relatime shared:14 - cgroup cgroup rw,pids 42 29 0:38 / /sys/fs/cgroup/hugetlb rw,nosuid,nodev,noexec,relatime shared:15 - cgroup cgroup rw,hugetlb 43 29 0:39 / /sys/fs/cgroup/rdma rw,nosuid,nodev,noexec,relatime shared:16 - cgroup cgroup rw,rdma 93 1 8:3 / / rw,noatime shared:1 - ext4 /dev/sda3 rw,noacl 44 22 0:40 / /proc/sys/fs/binfmt_misc rw,relatime shared:24 - autofs systemd-1 rw,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 45 24 0:41 / /dev/hugepages rw,relatime shared:25 - hugetlbfs hugetlbfs rw,pagesize=2M 46 24 0:19 / /dev/mqueue rw,nosuid,nodev,noexec,relatime shared:26 - mqueue mqueue rw 47 23 0:7 / /sys/kernel/debug rw,nosuid,nodev,noexec,relatime shared:27 - debugfs debugfs rw 48 23 0:11 / /sys/kernel/tracing rw,nosuid,nodev,noexec,relatime shared:28 - tracefs tracefs rw 49 93 0:42 / /tmp rw,nosuid,nodev shared:29 - tmpfs tmpfs rw,nr_inodes=409600 50 23 0:43 / /sys/kernel/config rw,nosuid,nodev,noexec,relatime shared:30 - configfs configfs rw 116 23 0:44 / /sys/fs/fuse/connections rw,nosuid,nodev,noexec,relatime shared:64 - fusectl fusectl rw 119 93 8:2 / /boot/efi rw,relatime shared:66 - vfat /dev/sda2 rw,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro 519 28 0:55 / /run/user/0 rw,nosuid,nodev,relatime shared:298 - tmpfs tmpfs rw,size=403332k,nr_inodes=100833,mode=700 532 28 0:56 / /run/user/1000 rw,nosuid,nodev,relatime shared:305 - tmpfs tmpfs rw,size=403332k,nr_inodes=100833,mode=700,uid=1000,gid=100 595 28 0:24 /netns /run/netns rw,nosuid,nodev shared:22 - tmpfs tmpfs rw,size=806672k,nr_inodes=819200,mode=755 581 44 0:57 / /proc/sys/fs/binfmt_misc rw,nosuid,nodev,noexec,relatime shared:313 - binfmt_misc binfmt_misc rw 583 28 0:58 / /run/user/1001 rw,nosuid,nodev,relatime shared:315 - tmpfs tmpfs rw,size=403332k,nr_inodes=100833,mode=700,uid=1001,gid=100 656 93 8:3 /var/lib/containers/storage/overlay /var/lib/containers/storage/overlay rw,noatime - ext4 /dev/sda3 rw,noacl

3. podman run --security-opt seccomp=unconfined -dt -p 8080:80/tcp docker.io/library/httpd also have the same OCI error as mention above

[giuseppe]

does it work if you run as root?

Could you please strace podman with podman unshare strace -o strace.log -f -v -s 1000 podman run ... and attach the strace.log file here?

Would it be possible to you to try with crun as well?

[ntsbtz]

podman unshare strace -o strace.log -f -v -s 1000

Yes it works fine as root. I cannot run with crun .

Note: In first attempt once i setup the rootless environment and try to run it is throwing the above mention error. But after that if i logout from rootless user and chown the /home/rootlessuser with rootlessuser. Login back to rootlessuser it will run successfully .

Please find the attached strace.log file .

strace.log

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@giuseppe @ntsbtz Any update on this?

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

Since we heard no feedback, I am going to close, Reopen if you have any feedback.