github-actions[bot]
commented
6 months ago A friendly reminder that this issue had no activity for 30 days.
Open virzak opened 7 months ago
github-actions[bot]
commented
6 months ago A friendly reminder that this issue had no activity for 30 days.
jobcespedes
commented
2 months ago Same message when using --userns=keep-id, podman version --userns=keep-id
rhatdan
commented
2 months ago Please submit podman info
jobcespedes
commented
2 months ago Please submit podman info
host:
arch: arm64
buildahVersion: 1.36.0
cgroupControllers:
- cpuset
- cpu
- io
- memory
- pids
- rdma
- misc
cgroupManager: systemd
cgroupVersion: v2
conmon:
package: conmon-2.1.10-1.fc40.aarch64
path: /usr/bin/conmon
version: 'conmon version 2.1.10, commit: '
cpuUtilization:
idlePercent: 99.83
systemPercent: 0.09
userPercent: 0.09
cpus: 2
databaseBackend: sqlite
distribution:
distribution: fedora
variant: coreos
version: "40"
eventLogger: journald
freeLocks: 2048
hostname: dev-kaws-us-east-1-lms-01-lb-instance-01
idMappings:
gidmap: null
uidmap: null
kernel: 6.8.11-300.fc40.aarch64
linkmode: dynamic
logDriver: journald
memFree: 153567232
memTotal: 946298880
networkBackend: netavark
networkBackendInfo:
backend: netavark
dns:
package: aardvark-dns-1.11.0-1.fc40.aarch64
path: /usr/libexec/podman/aardvark-dns
version: aardvark-dns 1.11.0
package: netavark-1.11.0-1.fc40.aarch64
path: /usr/libexec/podman/netavark
version: netavark 1.11.0
ociRuntime:
name: crun
package: crun-1.15-1.fc40.aarch64
path: /usr/bin/crun
version: |-
crun version 1.15
commit: e6eacaf4034e84185fd8780ac9262bbf57082278
rundir: /run/crun
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
os: linux
pasta:
executable: /usr/bin/pasta
package: passt-0^20240510.g7288448-1.fc40.aarch64
version: |
pasta 0^20240510.g7288448-1.fc40.aarch64-pasta
Copyright Red Hat
GNU General Public License, version 2 or later
<https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
remoteSocket:
exists: false
path: /run/podman/podman.sock
rootlessNetworkCmd: pasta
security:
apparmorEnabled: false
capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
rootless: false
seccompEnabled: true
seccompProfilePath: /usr/share/containers/seccomp.json
selinuxEnabled: true
serviceIsRemote: false
slirp4netns:
executable: /usr/bin/slirp4netns
package: slirp4netns-1.2.2-2.fc40.aarch64
version: |-
slirp4netns version 1.2.2
commit: 0ee2d87523e906518d34a6b423271e4826f71faf
libslirp: 4.7.0
SLIRP_CONFIG_VERSION_MAX: 4
libseccomp: 2.5.5
swapFree: 0
swapTotal: 0
uptime: 11h 5m 11.00s (Approximately 0.46 days)
variant: v8
plugins:
authorization: null
log:
- k8s-file
- none
- passthrough
- journald
network:
- bridge
- macvlan
- ipvlan
volume:
- local
registries:
search:
- registry.fedoraproject.org
- registry.access.redhat.com
- docker.io
store:
configFile: /usr/share/containers/storage.conf
containerStore:
number: 0
paused: 0
running: 0
stopped: 0
graphDriverName: overlay
graphOptions:
overlay.imagestore: /usr/lib/containers/storage
overlay.mountopt: nodev,metacopy=on
graphRoot: /var/lib/containers/storage
graphRootAllocated: 10132369408
graphRootUsed: 2901696512
graphStatus:
Backing Filesystem: xfs
Native Overlay Diff: "false"
Supports d_type: "true"
Supports shifting: "true"
Supports volatile: "true"
Using metacopy: "true"
imageCopyTmpDir: /var/tmp
imageStore:
number: 9
runRoot: /run/containers/storage
transientStore: false
volumePath: /var/lib/containers/storage/volumes
version:
APIVersion: 5.1.0
Built: 1716940800
BuiltTime: Tue May 28 18:00:00 2024
GitCommit: ""
GoVersion: go1.22.3
Os: linux
OsArch: linux/arm64
Version: 5.1.0--userns is being used in following podman systemd unit:
[Unit]
Description=Podman container-keepalived.service
Documentation=man:podman-generate-systemd(1)
Wants=network-online.target
After=network-online.target
RequiresMountsFor=%t/containers
# User-defined dependencies
After=keepalived-image-build.service
Requires=keepalived-image-build.service
[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=on-failure
RestartSec=15
TimeoutStartSec=180
TimeoutStopSec=90
ExecStart=/usr/bin/podman run \
--cidfile=%t/%n.ctr-id \
--cgroups=no-conmon \
--rm \
--sdnotify=conmon \
-d \
--replace \
--pull never \
--stop-timeout 30 \
--cap-add=NET_ADMIN \
--cap-add=NET_BROADCAST \
--cap-add=NET_RAW \
--net=host \
--user=1001 \
--userns=keep-id \
--volume /etc/keepalived/keepalived.conf:/etc/keepalived/keepalived.conf:ro \
--volume /etc/keepalived/tls:/etc/keepalived/tls:ro \
--name keepalived keepalived
ExecStop=/usr/bin/podman stop \
--ignore -t 30 \
--cidfile=%t/%n.ctr-id
ExecStopPost=/usr/bin/podman rm \
-f \
--ignore -t 30 \
--cidfile=%t/%n.ctr-id
Type=notify
NotifyAccess=all
[Install]
WantedBy=default.targetNotes:
--tmpfs /sys changes type of error--userns at all got me the same functionality I was looking for, without error: container user id 1001 can access files from host user id 1001rhatdan
commented
2 months ago Is the user running the command UID=1001?
If you run /usr/bin/podman run \ --cidfile=%t/%n.ctr-id \ --cgroups=no-conmon \ --rm \ --sdnotify=conmon \ -d \ --replace \ --pull never \ --stop-timeout 30 \ --cap-add=NET_ADMIN \ --cap-add=NET_BROADCAST \ --cap-add=NET_RAW \ --net=host \ --user=1001 \ --userns=keep-id \ --volume /etc/keepalived/keepalived.conf:/etc/keepalived/keepalived.conf:ro \ --volume /etc/keepalived/tls:/etc/keepalived/tls:ro \ --name keepalived keepalived
By hand when logged in, does it work?
jobcespedes
commented
2 months ago Is the user running the command UID=1001?
No. It is root (0) running the systemd unit
By hand when logged in, does it work?
No. It is the same error
rhatdan
commented
1 month ago @giuseppe ideas?
giuseppe
commented
1 month ago Can you show me the output of cat /proc/self/mountinfo?
Also, could you try without --net host? With --net host, you could simply bind mount /sys from the host
jobcespedes
commented
1 month ago Can you show me the output of
cat /proc/self/mountinfo?
67 70 259:4 / /sysroot ro,relatime - xfs /dev/nvme0n1p4 rw,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota
70 1 259:4 /ostree/deploy/fedora-coreos/deploy/c8f502b11881597a1386a090a80f6eb680871a9afc4c625796f2ae0f6a1bf7c4.0 / rw,relatime shared:1 - xfs /dev/nvme0n1p4 rw,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota
71 70 259:4 /ostree/deploy/fedora-coreos/deploy/c8f502b11881597a1386a090a80f6eb680871a9afc4c625796f2ae0f6a1bf7c4.0/etc /etc rw,relatime shared:2 - xfs /dev/nvme0n1p4 rw,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota
72 70 259:4 /ostree/deploy/fedora-coreos/deploy/c8f502b11881597a1386a090a80f6eb680871a9afc4c625796f2ae0f6a1bf7c4.0/usr /usr ro,relatime shared:3 - xfs /dev/nvme0n1p4 rw,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota
73 67 259:4 /ostree/deploy/fedora-coreos/var /sysroot/ostree/deploy/fedora-coreos/var rw,relatime - xfs /dev/nvme0n1p4 rw,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota
36 70 0:6 / /dev rw,nosuid shared:6 - devtmpfs devtmpfs rw,seclabel,size=4096k,nr_inodes=96132,mode=755,inode64
37 36 0:26 / /dev/shm rw,nosuid,nodev shared:7 - tmpfs tmpfs rw,seclabel,inode64
38 36 0:27 / /dev/pts rw,nosuid,noexec,relatime shared:8 - devpts devpts rw,seclabel,gid=5,mode=620,ptmxmode=000
39 70 0:25 / /sys rw,nosuid,nodev,noexec,relatime shared:9 - sysfs sysfs rw,seclabel
40 39 0:7 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime shared:10 - securityfs securityfs rw
41 39 0:29 / /sys/fs/cgroup rw,nosuid,nodev,noexec,relatime shared:11 - cgroup2 cgroup2 rw,seclabel,nsdelegate,memory_recursiveprot
42 39 0:30 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime shared:12 - pstore pstore rw,seclabel
43 39 0:31 / /sys/firmware/efi/efivars rw,nosuid,nodev,noexec,relatime shared:13 - efivarfs efivarfs rw
44 39 0:32 / /sys/fs/bpf rw,nosuid,nodev,noexec,relatime shared:14 - bpf bpf rw,mode=700
45 39 0:33 / /sys/kernel/config rw,nosuid,nodev,noexec,relatime shared:15 - configfs configfs rw
46 70 0:24 / /proc rw,nosuid,nodev,noexec,relatime shared:17 - proc proc rw
47 70 0:28 / /run rw,nosuid,nodev shared:18 - tmpfs tmpfs rw,seclabel,size=184788k,nr_inodes=819200,mode=755,inode64
26 39 0:23 / /sys/fs/selinux rw,nosuid,noexec,relatime shared:16 - selinuxfs selinuxfs rw
25 46 0:34 / /proc/sys/fs/binfmt_misc rw,relatime shared:19 - autofs systemd-1 rw,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=7467
27 36 0:35 / /dev/hugepages rw,nosuid,nodev,relatime shared:20 - hugetlbfs hugetlbfs rw,seclabel,pagesize=2M
28 39 0:8 / /sys/kernel/debug rw,nosuid,nodev,noexec,relatime shared:21 - debugfs debugfs rw,seclabel
29 36 0:22 / /dev/mqueue rw,nosuid,nodev,noexec,relatime shared:22 - mqueue mqueue rw,seclabel
30 70 0:36 / /tmp rw,nosuid,nodev shared:23 - tmpfs tmpfs rw,seclabel,size=461972k,nr_inodes=1048576,inode64
32 39 0:13 / /sys/kernel/tracing rw,nosuid,nodev,noexec,relatime shared:24 - tracefs tracefs rw,seclabel
33 39 0:37 / /sys/fs/fuse/connections rw,nosuid,nodev,noexec,relatime shared:25 - fusectl fusectl rw
35 70 259:4 /ostree/deploy/fedora-coreos/var /var rw,relatime shared:5 - xfs /dev/nvme0n1p4 rw,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota
50 70 259:3 / /boot ro,nosuid,nodev,relatime shared:85 - ext4 /dev/nvme0n1p3 rw,seclabel
53 25 0:40 / /proc/sys/fs/binfmt_misc rw,nosuid,nodev,noexec,relatime shared:90 - binfmt_misc binfmt_misc rw
159 35 0:43 / /var/lib/nfs/rpc_pipefs rw,relatime shared:93 - rpc_pipefs sunrpc rw
145 47 0:28 /netns /run/netns rw,nosuid,nodev shared:18 - tmpfs tmpfs rw,seclabel,size=184788k,nr_inodes=819200,mode=755,inode64
64 35 259:4 /ostree/deploy/fedora-coreos/var/lib/containers/storage/overlay /var/lib/containers/storage/overlay rw,relatime - xfs /dev/nvme0n1p4 rw,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota
319 35 0:53 / /var/lib/containers/storage/overlay-containers/94c4b4159f7bbd078e7e8805048ebf032dba74d9dca371225a57788dd4e45d50/userdata/shm rw,nosuid,nodev,noexec,relatime shared:452 - tmpfs shm rw,context="system_u:object_r:container_file_t:s0:c1,c515",size=64000k,inode64
556 64 0:54 / /var/lib/containers/storage/overlay/8061ec55bddaf662879fdb4e434ca0e18ae82e5cc64016aad6bfcaffebcaa42e/merged rw,nodev,relatime - overlay overlay rw,context="system_u:object_r:container_file_t:s0:c1,c515",lowerdir=/var/lib/containers/storage/overlay/l/H4D7AC6WR7FENOFTWRDGN3KC3V:/var/lib/containers/storage/overlay/l/6WMTNQXLWVET6TJAP6BGTUNJPF:/var/lib/containers/storage/overlay/l/3XKBYHCOES2DCBZFPOMEKDXDQ3:/var/lib/containers/storage/overlay/l/L26X46UIXOBVDMAZJHKYXVPJA4:/var/lib/containers/storage/overlay/l/DU6Z3FXL2NYHESHGI3BM4TWGST:/var/lib/containers/storage/overlay/l/JPZDMNJQTRPF6DFMIFSGSEI3LG:/var/lib/containers/storage/overlay/l/DZXO36XZS2LELBBI3RWYB5YJJF,upperdir=/var/lib/containers/storage/overlay/8061ec55bddaf662879fdb4e434ca0e18ae82e5cc64016aad6bfcaffebcaa42e/diff,workdir=/var/lib/containers/storage/overlay/8061ec55bddaf662879fdb4e434ca0e18ae82e5cc64016aad6bfcaffebcaa42e/work,redirect_dir=on,uuid=on,metacopy=on,volatile
63 47 0:50 / /run/user/1000 rw,nosuid,nodev,relatime shared:375 - tmpfs tmpfs rw,seclabel,size=92392k,nr_inodes=23098,mode=700,uid=1000,gid=1000,inode64Also, could you try without --net host?
Keepalived fails with error IPVS: Can't initialize ipvs: No such file or directory
giuseppe
commented
1 month ago that error is coming from the container payload I guess. Could you run strace -Z -f -vv -s 1000 podman run .... to see what operation is failing with ENOENT?
jobcespedes
commented
1 month ago It is a lengthy output. I am attaching the first 10 and last 10 lines matching ENOENT.
strace -Z -f -vv -s 1000 /usr/bin/podman run \
--rm \
--replace \
--pull never \
--stop-timeout 30 \
--cap-add=NET_ADMIN \
--cap-add=NET_BROADCAST \
--cap-add=NET_RAW \
--net=host \
--user 1001 \
--userns=keep-id \
--tmpfs /sys \
--volume /etc/keepalived/keepalived.conf:/etc/keepalived/keepalived.conf:ro \
--volume /etc/keepalived/tls:/etc/keepalived/tls:ro \
--name keepalived keepalivedfaccessat(AT_FDCWD, "/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
faccessat(AT_FDCWD, "/etc/containers/podman_preexec_hooks.txt", F_OK) = -1 ENOENT (No such file or directory)
[pid 75243] statfs("/sys/fs/cgroup/unified", 0x400035fc90) = -1 ENOENT (No such file or directory)
[pid 75243] faccessat(AT_FDCWD, "/etc/containers/storage.conf", F_OK) = -1 ENOENT (No such file or directory)
[pid 75243] faccessat(AT_FDCWD, "/etc/containers/storage.conf", F_OK) = -1 ENOENT (No such file or directory)
[pid 75243] newfstatat(AT_FDCWD, "/etc/containers/containers.conf.d", 0x40004d83f8, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory)
[pid 75243] newfstatat(AT_FDCWD, "/root/.config/containers/containers.conf.d", 0x40004d84b8, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory)
[pid 75243] openat(AT_FDCWD, "/etc/containers/containers.conf", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 75243] openat(AT_FDCWD, "/root/.config/containers/containers.conf", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 75243] newfstatat(AT_FDCWD, "/etc/containers/containers.conf.d", 0x40004d87b8, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory)[pid 75284] newfstatat(AT_FDCWD, "/var/lib/containers/storage/db.sql-journal", 0xffff4fe0e298, 0) = -1 ENOENT (No such file or directory)
[pid 75284] newfstatat(AT_FDCWD, "/var/lib/containers/storage/db.sql-wal", 0xffff4fe0e298, 0) = -1 ENOENT (No such file or directory)
[pid 75284] openat(AT_FDCWD, "/var/lib/containers/storage/overlay/staging", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 75247] newfstatat(AT_FDCWD, "/var/lib/containers/storage/db.sql-journal", 0xffff3f40e298, 0) = -1 ENOENT (No such file or directory)
[pid 75247] newfstatat(AT_FDCWD, "/var/lib/containers/storage/db.sql-wal", 0xffff3f40e298, 0) = -1 ENOENT (No such file or directory)
[pid 75247] newfstatat(AT_FDCWD, "/var/lib/containers/storage/db.sql-journal", 0xffff3f40e298, 0) = -1 ENOENT (No such file or directory)
[pid 75247] newfstatat(AT_FDCWD, "/var/lib/containers/storage/db.sql-wal", 0xffff3f40e298, 0) = -1 ENOENT (No such file or directory)
[pid 75247] newfstatat(AT_FDCWD, "/var/lib/containers/storage/db.sql-journal", 0xffff3f40e298, 0) = -1 ENOENT (No such file or directory)
[pid 75247] newfstatat(AT_FDCWD, "/var/lib/containers/storage/db.sql-wal", 0xffff3f40e298, 0) = -1 ENOENT (No such file or directory)
[pid 75247] openat(AT_FDCWD, "/var/lib/containers/storage/overlay/staging", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
Issue Description
I'm getting
Error response from daemon: crun: mount
sysfstosys: Operation not permitted: OCI permission deniedNo idea how to avoid, debug or which error logs I need to look into
Steps to reproduce the issue
Steps to reproduce the issue
podman compose upfrom .devcontainer directoryDescribe the results you received
Describe the results you expected
The container should have started.
podman info output