search

containers / podman

Podman: A tool for managing OCI containers and pods.
https://podman.io
Apache License 2.0
22.61k stars 2.31k forks source link

Access Denied when running 'podman images' command #22565

Open ankurmalhotra07 opened 2 months ago

ankurmalhotra07 commented 2 months ago

Issue Description

Describe your issue Want to use additional image stores as explained in this guide However, running into permission access denied issues when trying to run podman images/pull commands.

Steps to reproduce the issue

Steps to reproduce the issue

  1. Provision EFS share in AWS
  2. Mount EFS share mount -t efs -o tls fs-123...:/ /var/lib/mycontainers
  3. Pull image using podman time podman --root /var/lib/mycontainers pull docker.io/amazoncorretto:latest
  4. Make EFS share read only
  5. Run podman images

Describe the results you received

Describe the results you received podman images `+ podman images

Error: open /var/lib/mycontainers/overlay-images/images.lock: permission denied

script returned exit code 125`

Describe the results you expected

Describe the results you expected Images are displayed without any errors

podman info output

+ podman images

Error: open /var/lib/mycontainers/overlay-images/images.lock: permission denied

script returned exit code 125

Podman in a container

Yes

Privileged Or Rootless

Privileged

Upstream Latest Release

Yes

Additional environment details

No response

Additional information

No response

giuseppe commented 2 months ago

please strace the command and report the failing syscall. You can do it by running strace -o /tmp/podman.log -f -v -s 1000 -Z podman images and attach the /tmp/podman.log file you get.

ankurmalhotra07 commented 2 months ago

@giuseppe here you go- `+ cat /tmp/podman.log

97 archprctl(0x3001 /* ARCH??? */, 0x7ffe289839d0) = -1 EINVAL (Invalid argument)

97 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)

97 statfs("/selinux", 0x7ffe28983990) = -1 ENOENT (No such file or directory)

97 access("/etc/selinux/config", F_OK) = -1 ENOENT (No such file or directory)

97 epoll_ctl(4, EPOLL_CTL_ADD, 3, {events=EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, data={u32=1045954561, u64=9213806973933846529}}) = -1 EPERM (Operation not permitted)

97 seccomp(SECCOMP_SET_MODE_STRICT, 0x1, NULL) = -1 EINVAL (Invalid argument)

97 seccomp(SECCOMP_SET_MODE_FILTER, SECCOMP_FILTER_FLAG_TSYNC, NULL) = -1 EFAULT (Bad address)

97 seccomp(SECCOMP_SET_MODE_FILTER, SECCOMP_FILTER_FLAG_LOG, NULL) = -1 EFAULT (Bad address)

97 seccomp(SECCOMP_SET_MODE_FILTER, SECCOMP_FILTER_FLAG_SPEC_ALLOW, NULL) = -1 EFAULT (Bad address)

97 seccomp(SECCOMP_SET_MODE_FILTER, SECCOMP_FILTER_FLAG_NEW_LISTENER, NULL) = -1 EFAULT (Bad address)

97 seccomp(SECCOMP_SET_MODE_FILTER, SECCOMP_FILTER_FLAG_TSYNC_ESRCH, NULL) = -1 EFAULT (Bad address)

97 futex(0x55e0d35e06c0, FUTEX_WAIT_PRIVATE, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)

97 futex(0x55e0d35e06c0, FUTEX_WAIT_PRIVATE, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)

99 futex(0xc00007e948, FUTEX_WAIT_PRIVATE, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)

101 futex(0xc000100148, FUTEX_WAIT_PRIVATE, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)

97 futex(0x55e0d35e06c0, FUTEX_WAIT_PRIVATE, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)

101 futex(0xc000100148, FUTEX_WAIT_PRIVATE, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)

100 futex(0xc00007ed48, FUTEX_WAIT_PRIVATE, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)

100 futex(0x55e0d35e24d8, FUTEX_WAIT_PRIVATE, 2, NULL) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)

104 futex(0x55e0d35e24d8, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)

100 --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=97, si_uid=0} ---

100 futex(0x55e0d35e24d8, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)

100 futex(0xc00007ed48, FUTEX_WAIT_PRIVATE, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)

97 --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=97, si_uid=0} ---

100 futex(0x55e0d35e25d8, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=100000}) = -1 ETIMEDOUT (Connection timed out)

100 futex(0x55e0d35e25d8, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=100000}) = -1 ETIMEDOUT (Connection timed out)

97 futex(0x55e0d35e11a8, FUTEX_WAIT_PRIVATE, 0, NULL) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)

97 --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=97, si_uid=0} ---

97 futex(0x55e0d35e11a8, FUTEX_WAIT_PRIVATE, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)

105 futex(0x55e0d35e25d8, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=100000}) = -1 ETIMEDOUT (Connection timed out)

97 futex(0x55e0d35e11a8, FUTEX_WAIT_PRIVATE, 0, NULL) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)

97 --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=97, si_uid=0} ---

97 futex(0x55e0d35e11a8, FUTEX_WAIT_PRIVATE, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)

97 futex(0x55e0d35e24d8, FUTEX_WAIT_PRIVATE, 2, NULL) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)

97 --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=97, si_uid=0} ---

97 futex(0x55e0d35e24d8, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)

97 futex(0x55e0d35e24d8, FUTEX_WAIT_PRIVATE, 2, NULL) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)

97 --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=97, si_uid=0} ---

97 futex(0x55e0d35e24d8, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)

99 futex(0xc00007e948, FUTEX_WAIT_PRIVATE, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)

105 futex(0x55e0d35e25d8, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=100000}) = -1 EAGAIN (Resource temporarily unavailable)

97 statfs("/sys/fs/cgroup/unified", 0xc0006bfc98) = -1 ENOENT (No such file or directory)

97 epoll_ctl(4, EPOLL_CTL_ADD, 3, {events=EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, data={u32=1045954565, u64=9213806973933846533}}) = -1 EPERM (Operation not permitted)

97 newfstatat(AT_FDCWD, "/var/../run/containers", 0xc00011f148, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory)

97 newfstatat(AT_FDCWD, "/var/lib/containers/storage", 0xc00011f628, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory)

97 epoll_ctl(4, EPOLL_CTL_ADD, 3, {events=EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, data={u32=1045954566, u64=9213806973933846534}}) = -1 EPERM (Operation not permitted)

97 newfstatat(AT_FDCWD, "/etc/containers/containers.conf.d", 0xc000146378, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory)

97 epoll_ctl(4, EPOLL_CTL_ADD, 3, {events=EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, data={u32=1045954570, u64=9213806973933846538}}) = -1 EPERM (Operation not permitted)

97 epoll_ctl(4, EPOLL_CTL_ADD, 3, {events=EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, data={u32=1045954571, u64=9213806973933846539}}) = -1 EPERM (Operation not permitted)

97 --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=97, si_uid=0} ---

105 futex(0xc000518148, FUTEX_WAIT_PRIVATE, 0, NULL) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)

105 --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=97, si_uid=0} ---

106 futex(0xc000600148, FUTEX_WAIT_PRIVATE, 0, NULL) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)

106 --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=97, si_uid=0} ---

99 futex(0xc00007e948, FUTEX_WAIT_PRIVATE, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)

97 futex(0x55e0d35e11a8, FUTEX_WAIT_PRIVATE, 0, NULL) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)

97 --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=97, si_uid=0} ---

97 futex(0x55e0d35e11a8, FUTEX_WAIT_PRIVATE, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)

97 epoll_ctl(4, EPOLL_CTL_ADD, 3, {events=EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, data={u32=1045954572, u64=9213806973933846540}}) = -1 EPERM (Operation not permitted)

100 futex(0x55e0d35e25d8, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=100000}) = -1 ETIMEDOUT (Connection timed out)

97 epoll_ctl(4, EPOLL_CTL_ADD, 3, {events=EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, data={u32=1045954573, u64=9213806973933846541}}) = -1 EPERM (Operation not permitted)

97 newfstatat(AT_FDCWD, "/etc/containers/containers.conf.d", 0xc00011ed38, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory)

97 epoll_ctl(4, EPOLL_CTL_ADD, 3, {events=EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, data={u32=1045954575, u64=9213806973933846543}}) = -1 EPERM (Operation not permitted)

97 epoll_ctl(4, EPOLL_CTL_ADD, 3, {events=EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, data={u32=1045954576, u64=9213806973933846544}}) = -1 EPERM (Operation not permitted)

97 futex(0x55e0d35e11a8, FUTEX_WAIT_PRIVATE, 0, NULL) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)

97 --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=97, si_uid=0} ---

102 --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=97, si_uid=0} ---

102 --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=97, si_uid=0} ---

102 futex(0xc00007f148, FUTEX_WAIT_PRIVATE, 0, NULL) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)

102 --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=97, si_uid=0} ---

102 futex(0xc00007f148, FUTEX_WAIT_PRIVATE, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)

97 futex(0x55e0d35e24d8, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)

97 --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=97, si_uid=0} ---

97 rt_sigreturn({mask=[]}) = -1 EAGAIN (Resource temporarily unavailable)

105 futex(0x55e0d35e25d8, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=100000}) = -1 ETIMEDOUT (Connection timed out)

97 --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=97, si_uid=0} ---

97 futex(0x55e0d35e24d8, FUTEX_WAIT_PRIVATE, 2, NULL) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)

97 --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=97, si_uid=0} ---

97 futex(0x55e0d35e24d8, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)

97 futex(0x55e0d35e24d8, FUTEX_WAIT_PRIVATE, 2, NULL) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)

97 --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=97, si_uid=0} ---

97 futex(0x55e0d35e24d8, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)

105 futex(0x55e0d35e25d8, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=100000}) = -1 EAGAIN (Resource temporarily unavailable)

105 futex(0xc000518148, FUTEX_WAIT_PRIVATE, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)

100 newfstatat(AT_FDCWD, "/usr/libexec/podman/conmon", 0xc0004f6038, 0) = -1 ENOENT (No such file or directory)

100 newfstatat(AT_FDCWD, "/usr/local/libexec/podman/conmon", 0xc0004f6108, 0) = -1 ENOENT (No such file or directory)

100 newfstatat(AT_FDCWD, "/usr/local/lib/podman/conmon", 0xc0004f61d8, 0) = -1 ENOENT (No such file or directory)

100 newfstatat(AT_FDCWD, "/var/lib/containers/storage/libpod", 0xc0004f6378, 0) = -1 ENOENT (No such file or directory)

100 newfstatat(AT_FDCWD, "/var/lib/containers/storage", 0xc0004f6448, 0) = -1 ENOENT (No such file or directory)

100 newfstatat(AT_FDCWD, "/run/libpod", 0xc0004f65e8, 0) = -1 ENOENT (No such file or directory)

100 newfstatat(AT_FDCWD, "/var/lib/containers/storage/libpod/bolt_state.db", 0xc0004f6788, 0) = -1 ENOENT (No such file or directory)

100 newfstatat(AT_FDCWD, "/var/lib/containers/storage/db.sql", 0x7fde054389b0, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory)

100 newfstatat(AT_FDCWD, "/var/lib/containers/storage/db.sql-journal", 0x7fde054399b0, 0) = -1 ENOENT (No such file or directory)

100 newfstatat(AT_FDCWD, "/var/lib/containers/storage/db.sql-wal", 0x7fde054399b0, 0) = -1 ENOENT (No such file or directory)

100 newfstatat(AT_FDCWD, "/var/lib/containers/storage/db.sql-journal", 0x7fde0543a8a0, 0) = -1 ENOENT (No such file or directory)

100 newfstatat(AT_FDCWD, "/var/lib/containers/storage/db.sql-wal", 0x7fde0543a8a0, 0) = -1 ENOENT (No such file or directory)

100 newfstatat(AT_FDCWD, "/var/lib/containers/storage/db.sql-journal", 0x7fde0543a8c0, 0) = -1 ENOENT (No such file or directory)

100 newfstatat(AT_FDCWD, "/var/lib/containers/storage/db.sql-wal", 0x7fde0543a8c0, 0) = -1 ENOENT (No such file or directory)

100 newfstatat(AT_FDCWD, "/var/lib/containers/storage/db.sql-journal", 0x7fde0543a8a0, 0) = -1 ENOENT (No such file or directory)

100 newfstatat(AT_FDCWD, "/var/lib/containers/storage/db.sql-wal", 0x7fde0543a8a0, 0) = -1 ENOENT (No such file or directory)

100 --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=97, si_uid=0} ---

100 newfstatat(AT_FDCWD, "/var/run/containers/storage", 0xc0004f6b98, 0) = -1 ENOENT (No such file or directory)

100 newfstatat(AT_FDCWD, "/var/run/containers", 0xc0004f6c68, 0) = -1 ENOENT (No such file or directory)

100 newfstatat(AT_FDCWD, "/var/lib/containers/storage/overlay", 0xc0004f6ed8, 0) = -1 ENOENT (No such file or directory)

100 newfstatat(AT_FDCWD, "/var/lib/containers/storage/overlay/l", 0xc0004f72e8, 0) = -1 ENOENT (No such file or directory)

100 newfstatat(AT_FDCWD, "/var/lib/containers/storage/overlay/l", 0xc0004f77c8, 0) = -1 ENOENT (No such file or directory)

100 newfstatat(AT_FDCWD, "/var/run/containers/storage/overlay", 0xc0004f7968, 0) = -1 ENOENT (No such file or directory)

100 newfstatat(AT_FDCWD, "/var/run/containers/storage/overlay", 0xc0004f7d78, 0) = -1 ENOENT (No such file or directory)

100 epoll_ctl(4, EPOLL_CTL_ADD, 8, {events=EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, data={u32=1045954577, u64=9213806973933846545}}) = -1 EPERM (Operation not permitted)

100 quotactl(QCMD(Q_XSETQLIM, PRJQUOTA), "/var/lib/containers/storage/overlay/backingFsBlockDev", 459145140, {d_version=1, d_flags=FS_PROJ_QUOTA, d_fieldmask=0, d_id=459145140, d_blk_hardlimit=0, d_blk_softlimit=0, d_ino_hardlimit=0, d_ino_softlimit=0, d_bcount=0, d_icount=0, d_itimer=0, d_btimer=0, d_iwarns=0, d_bwarns=0, d_rtb_hardlimit=0, d_rtb_softlimit=0, d_rtbcount=0, d_rtbtimer=0, d_rtbwarns=0}) = -1 ENOSYS (Function not implemented)

100 newfstatat(AT_FDCWD, "/var/lib/containers/storage/overlay-images", 0xc000570518, 0) = -1 ENOENT (No such file or directory)

100 openat(AT_FDCWD, "/var/lib/containers/storage/overlay-images/images.json", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

100 newfstatat(AT_FDCWD, "/var/lib/containers/storage/overlay-containers", 0xc000570788, 0) = -1 ENOENT (No such file or directory)

100 newfstatat(AT_FDCWD, "/var/run/containers/storage/overlay-containers", 0xc000570928, 0) = -1 ENOENT (No such file or directory)

100 openat(AT_FDCWD, "/var/lib/containers/storage/overlay-containers/containers.json", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

100 openat(AT_FDCWD, "/var/lib/containers/storage/overlay-containers/volatile-containers.json", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

100 openat(AT_FDCWD, "/var/lib/mycontainers/overlay-images/images.lock", O_RDONLY|O_CREAT|O_CLOEXEC, 0644) = -1 EACCES (Permission denied)

106 +++ exited with 125 +++

104 +++ exited with 125 +++

103 +++ exited with 125 +++

102 +++ exited with 125 +++

101 +++ exited with 125 +++

100 +++ exited with 125 +++

99 +++ exited with 125 +++

98 +++ exited with 125 +++

105 +++ exited with 125 +++

97 +++ exited with 125 +++`

giuseppe commented 2 months ago 
100 openat(AT_FDCWD, "/var/lib/mycontainers/overlay-images/images.lock", O_RDONLY|O_CREAT|O_CLOEXEC, 0644) = -1 EACCES (Permission denied)

This error depends on your file system, it looks like it doesn't honor CAP_DAC_OVERRIDE, and the look up inside /var/lib/mycontainers fails for that reason. It is not something we can address in Podman

giuseppe commented 2 months ago

did you run chmod -R 755 /var/lib/mycontainers?

github-actions[bot] commented 1 month ago

A friendly reminder that this issue had no activity for 30 days.