Closed HareToAme closed 1 month ago
mheon
commented
1 month ago With --log-level=debug conmon should be logging to syslog. Are there any messages from Conmon in there? It seems to be failing quite badly, but it may have logged the error it's seeing to syslog.
giuseppe
commented
1 month ago I am not familiar with extfs. Can you please show the output of cat /proc/self/mountinfo and cat /proc/filesystems?
Is /home/xlhuang/.local/lib/containers/storage also on extfs?
You may be hitting something like: https://github.com/containers/podman/blob/main/troubleshooting.md#14-rootless-podman-build-fails-eperm-on-nfs
HareToAme
commented
1 month ago I am a non-root user, so I am unable to view syslog. Below are the outputs of cat /proc/self/mountinfo and cat /proc/filesystems:
(base) xlhuang@a100:~$ podman ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a93243846486 docker.io/library/ubuntu:22.04 /bin/bash 3 days ago Created ubuntu_me
(base) xlhuang@a100:~$ podman start ubuntu_me
Error: unable to start container "a93243846486576fc96fdd426774840c3c4ab471d3fcdc29573677a223e4261b": crun: mount `proc` to `proc`: Operation not permitted: OCI permission denied
(base) xlhuang@a100:~$ podman start ubuntu_me
Error: unable to start container "a93243846486576fc96fdd426774840c3c4ab471d3fcdc29573677a223e4261b": container create failed (no logs from conmon): conmon bytes "": readObjectStart: expect { or n, but found , error found in #0 byte of ...||..., bigger context ...||...
(base) xlhuang@a100:~$ cat /proc/self/mountinfo
24 30 0:22 / /sys rw,nosuid,nodev,noexec,relatime shared:7 - sysfs sysfs rw
25 30 0:23 / /proc rw,nosuid,nodev,noexec,relatime shared:13 - proc proc rw
26 30 0:5 / /dev rw,nosuid,relatime shared:2 - devtmpfs udev rw,size=263940160k,nr_inodes=65985040,mode=755,inode64
27 26 0:24 / /dev/pts rw,nosuid,noexec,relatime shared:3 - devpts devpts rw,gid=5,mode=620,ptmxmode=000
28 30 0:25 / /run rw,nosuid,nodev,noexec,relatime shared:5 - tmpfs tmpfs rw,size=52799548k,mode=755,inode64
30 1 259:4 / / rw,relatime shared:1 - xfs /dev/nvme1n1p2 rw,attr2,inode64,logbufs=8,logbsize=32k,noquota
31 24 0:6 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime shared:8 - securityfs securityfs rw
32 26 0:27 / /dev/shm rw,nosuid,nodev shared:4 - tmpfs tmpfs rw,inode64
33 28 0:28 / /run/lock rw,nosuid,nodev,noexec,relatime shared:6 - tmpfs tmpfs rw,size=5120k,inode64
34 24 0:29 / /sys/fs/cgroup rw,nosuid,nodev,noexec,relatime shared:9 - cgroup2 cgroup2 rw,nsdelegate,memory_recursiveprot
35 24 0:30 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime shared:10 - pstore pstore rw
36 24 0:31 / /sys/firmware/efi/efivars rw,nosuid,nodev,noexec,relatime shared:11 - efivarfs efivarfs rw
37 24 0:32 / /sys/fs/bpf rw,nosuid,nodev,noexec,relatime shared:12 - bpf bpf rw,mode=700
38 25 0:33 / /proc/sys/fs/binfmt_misc rw,relatime shared:14 - autofs systemd-1 rw,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=31760
39 26 0:34 / /dev/hugepages rw,relatime shared:15 - hugetlbfs hugetlbfs rw,pagesize=2M
40 26 0:20 / /dev/mqueue rw,nosuid,nodev,noexec,relatime shared:16 - mqueue mqueue rw
41 24 0:7 / /sys/kernel/debug rw,nosuid,nodev,noexec,relatime shared:17 - debugfs debugfs rw
42 24 0:12 / /sys/kernel/tracing rw,nosuid,nodev,noexec,relatime shared:18 - tracefs tracefs rw
43 25 0:35 / /proc/fs/nfsd rw,relatime shared:19 - nfsd nfsd rw
44 24 0:36 / /sys/fs/fuse/connections rw,nosuid,nodev,noexec,relatime shared:20 - fusectl fusectl rw
45 24 0:21 / /sys/kernel/config rw,nosuid,nodev,noexec,relatime shared:21 - configfs configfs rw
69 28 0:37 / /run/credentials/systemd-sysusers.service ro,nosuid,nodev,noexec,relatime shared:22 - ramfs none rw,mode=700
95 30 259:3 / /boot/efi rw,relatime shared:32 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
98 30 259:6 / /data rw,relatime shared:49 - ext4 /dev/md126p1 rw,stripe=32
119 38 0:38 / /proc/sys/fs/binfmt_misc rw,nosuid,nodev,noexec,relatime shared:63 - binfmt_misc binfmt_misc rw
123 28 0:39 / /run/rpc_pipefs rw,relatime shared:65 - rpc_pipefs sunrpc rw
811 28 0:4 net:[4026531840] /run/docker/netns/default rw shared:441 - nsfs nsfs rw
922 28 0:25 /snapd/ns /run/snapd/ns rw,nosuid,nodev,noexec,relatime - tmpfs tmpfs rw,size=52799548k,mode=755,inode64
1083 30 0:79 / /zhdd rw,noatime shared:488 - nfs4 192.168.1.99:/zhdd rw,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,nocto,proto=tcp,nconnect=16,timeo=600,retrans=2,sec=sys,clientaddr=192.168.1.11,fsc,local_lock=none,addr=192.168.1.99
614 1083 0:72 / /zhdd/home rw,noatime shared:365 - nfs4 192.168.1.99:/zhdd/home rw,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,nocto,proto=tcp,nconnect=16,timeo=600,retrans=2,sec=sys,clientaddr=192.168.1.11,fsc,local_lock=none,addr=192.168.1.99
780 614 0:68 / /zhdd/home/mjyin rw,noatime shared:427 - nfs4 192.168.1.99:/zhdd/home/mjyin rw,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,nocto,proto=tcp,nconnect=16,timeo=600,retrans=2,sec=sys,clientaddr=192.168.1.11,fsc,local_lock=none,addr=192.168.1.99
895 28 0:69 / /run/user/1003 rw,nosuid,nodev,relatime shared:509 - tmpfs tmpfs rw,size=52799544k,nr_inodes=13199886,mode=700,uid=1003,gid=1003,inode64
1654 30 7:0 / /snap/snapd/21465 ro,nodev,relatime shared:647 - squashfs /dev/loop0 ro,errors=continue
1760 30 7:4 / /snap/lxd/28373 ro,nodev,relatime shared:669 - squashfs /dev/loop4 ro,errors=continue
1798 614 0:106 / /zhdd/home/ryzhang rw,noatime shared:735 - nfs4 192.168.1.99:/zhdd/home/ryzhang rw,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,nocto,proto=tcp,nconnect=16,timeo=600,retrans=2,sec=sys,clientaddr=192.168.1.11,fsc,local_lock=none,addr=192.168.1.99
2087 28 0:110 / /run/user/1009 rw,nosuid,nodev,relatime shared:789 - tmpfs tmpfs rw,size=52799544k,nr_inodes=13199886,mode=700,uid=1009,gid=1009,inode64
110 614 0:113 / /zhdd/home/lkzhang rw,noatime shared:57 - nfs4 192.168.1.99:/zhdd/home/lkzhang rw,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,nocto,proto=tcp,nconnect=16,timeo=600,retrans=2,sec=sys,clientaddr=192.168.1.11,fsc,local_lock=none,addr=192.168.1.99
501 28 0:115 / /run/user/1011 rw,nosuid,nodev,relatime shared:221 - tmpfs tmpfs rw,size=52799544k,nr_inodes=13199886,mode=700,uid=1011,gid=1011,inode64
1753 1083 0:123 / /zhdd/models ro,noatime shared:702 - nfs4 192.168.1.99:/zhdd/models rw,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,nocto,proto=tcp,nconnect=16,timeo=600,retrans=2,sec=sys,clientaddr=192.168.1.11,fsc,local_lock=none,addr=192.168.1.99
776 614 0:83 / /zhdd/home/kaicheng rw,noatime shared:457 - nfs4 192.168.1.99:/zhdd/home/kaicheng rw,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,nocto,proto=tcp,nconnect=16,timeo=600,retrans=2,sec=sys,clientaddr=192.168.1.11,fsc,local_lock=none,addr=192.168.1.99
1272 28 0:81 / /run/user/1014 rw,nosuid,nodev,relatime shared:529 - tmpfs tmpfs rw,size=52799544k,nr_inodes=13199886,mode=700,uid=1014,gid=1014,inode64
1350 30 7:3 / /snap/core20/2318 ro,nodev,relatime shared:588 - squashfs /dev/loop3 ro,errors=continue
1338 30 7:1 / /snap/snapd/21759 ro,nodev,relatime shared:668 - squashfs /dev/loop1 ro,errors=continue
1043 614 0:108 / /zhdd/home/hlv rw,noatime shared:613 - nfs4 192.168.1.99:/zhdd/home/hlv rw,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,nocto,proto=tcp,nconnect=16,timeo=600,retrans=2,sec=sys,clientaddr=192.168.1.11,fsc,local_lock=none,addr=192.168.1.99
2327 28 0:84 / /run/user/1018 rw,nosuid,nodev,relatime shared:844 - tmpfs tmpfs rw,size=52799544k,nr_inodes=13199886,mode=700,uid=1018,gid=1018,inode64
191 28 0:82 / /run/user/1013 rw,nosuid,nodev,relatime shared:53 - tmpfs tmpfs rw,size=52799544k,nr_inodes=13199886,mode=700,uid=1013,gid=1013,inode64
689 614 0:104 / /zhdd/home/xiangxv rw,noatime shared:222 - nfs4 192.168.1.99:/zhdd/home/xiangxv rw,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,nocto,proto=tcp,nconnect=16,timeo=600,retrans=2,sec=sys,clientaddr=192.168.1.11,fsc,local_lock=none,addr=192.168.1.99
883 30 0:95 / /var/lib/docker/overlay2/be9a92b27d206dcc2107154711b17d2e834a4afb7222199d6cd768e4b87f3c85/merged rw,relatime shared:589 - overlay overlay rw,lowerdir=/var/lib/docker/overlay2/l/3Y4FNPDYHATYQNVYDEWYZA2NNS:/var/lib/docker/overlay2/l/EKYQV2UGX63I237XX4PWOYCSVR:/var/lib/docker/overlay2/l/75SLVC5TE6FPWPFHCTGSE7QV4Y:/var/lib/docker/overlay2/l/JFJHINYZSBZVCQOBFD6B3LYCIY:/var/lib/docker/overlay2/l/36RLQTOTZJR7K2ZPMDNNJZDZBT:/var/lib/docker/overlay2/l/67MCRGAWALQ53DN7CQQV4B6PKF:/var/lib/docker/overlay2/l/IT2ND7OAB2N6LIT6T3ZSXXNP7O:/var/lib/docker/overlay2/l/ZJFSKIUWMEYHQGCCAXTT4STUYV:/var/lib/docker/overlay2/l/QQ3BNFQIRGXUPDJGR45MKXP7Q3:/var/lib/docker/overlay2/l/WMBWBCXAID4LDYRAJPGDGRCI46:/var/lib/docker/overlay2/l/KEGERX2BZPSCDTNOFLAG66BVVN:/var/lib/docker/overlay2/l/D5Z3ZNZHXRJYI2E37OOIKAETM3:/var/lib/docker/overlay2/l/AMR5YVPAVXSTRFQCCLIMORXRMO,upperdir=/var/lib/docker/overlay2/be9a92b27d206dcc2107154711b17d2e834a4afb7222199d6cd768e4b87f3c85/diff,workdir=/var/lib/docker/overlay2/be9a92b27d206dcc2107154711b17d2e834a4afb7222199d6cd768e4b87f3c85/work
1024 30 0:90 / /var/lib/docker/volumes/mongodb_mongo-data/_data rw,noatime shared:666 - nfs4 192.168.1.99:/zhdd/home/hcgu/docker_service/1-planning/mongodb/mongo-data rw,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,nocto,proto=tcp,nconnect=16,timeo=600,retrans=2,sec=sys,clientaddr=192.168.1.11,fsc,local_lock=none,addr=192.168.1.99
1497 28 0:4 net:[4026536531] /run/docker/netns/549ec4995610 rw shared:665 - nsfs nsfs rw
792 30 0:49 / /var/lib/docker/overlay2/d02b4279cb3687e3823f68b4198f1a1d6a8e48d61fb4a452e3e5814efd2f84d4/merged rw,relatime shared:431 - overlay overlay rw,lowerdir=/var/lib/docker/overlay2/l/T7FUDWP6B7BIDUCWUK5QYFR5KE:/var/lib/docker/overlay2/l/FW3MQYGEJ2KPTTI57ULJNLBJSM:/var/lib/docker/overlay2/l/BI22YSEA2GQD3YC2PYYJOKEYYZ:/var/lib/docker/overlay2/l/OI376SIOTGPGH7UNCQGEHODMQU,upperdir=/var/lib/docker/overlay2/d02b4279cb3687e3823f68b4198f1a1d6a8e48d61fb4a452e3e5814efd2f84d4/diff,workdir=/var/lib/docker/overlay2/d02b4279cb3687e3823f68b4198f1a1d6a8e48d61fb4a452e3e5814efd2f84d4/work
1390 30 7:5 / /snap/lxd/29351 ro,nodev,relatime shared:371 - squashfs /dev/loop5 ro,errors=continue
502 922 0:4 mnt:[4026536271] /run/snapd/ns/lxd.mnt rw - nsfs nsfs rw
154 614 0:90 / /zhdd/home/hcgu rw,noatime shared:60 - nfs4 192.168.1.99:/zhdd/home/hcgu rw,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,nocto,proto=tcp,nconnect=16,timeo=600,retrans=2,sec=sys,clientaddr=192.168.1.11,fsc,local_lock=none,addr=192.168.1.99
613 28 0:51 / /run/user/1001 rw,nosuid,nodev,relatime shared:394 - tmpfs tmpfs rw,size=52799544k,nr_inodes=13199886,mode=700,uid=1001,gid=1001,inode64
735 28 0:91 / /run/user/1020 rw,nosuid,nodev,relatime shared:390 - tmpfs tmpfs rw,size=52799544k,nr_inodes=13199886,mode=700,uid=1020,gid=1020,inode64
784 614 0:100 / /zhdd/home/hychi rw,noatime shared:416 - nfs4 192.168.1.99:/zhdd/home/hychi rw,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,nocto,proto=tcp,nconnect=16,timeo=600,retrans=2,sec=sys,clientaddr=192.168.1.11,fsc,local_lock=none,addr=192.168.1.99
1593 28 0:128 / /run/user/9999 rw,nosuid,nodev,relatime shared:692 - tmpfs tmpfs rw,size=52799544k,nr_inodes=13199886,mode=700,uid=9999,gid=9999,inode64
1615 614 0:75 / /zhdd/home/monsoon rw,noatime shared:720 - nfs4 192.168.1.99:/zhdd/home/monsoon rw,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,nocto,proto=tcp,nconnect=16,timeo=600,retrans=2,sec=sys,clientaddr=192.168.1.11,fsc,local_lock=none,addr=192.168.1.99
1635 25 0:27 / /proc/2806118 rw,nosuid,nodev shared:4 - tmpfs tmpfs rw,inode64
1923 25 0:27 / /proc/2809415 rw,nosuid,nodev shared:4 - tmpfs tmpfs rw,inode64
109 30 0:50 / /var/lib/docker/overlay2/05d0e0edf6aeac19cfffda551b35fc4a5f76f4477dcfbceca9e88ae9aa277f22/merged rw,relatime shared:55 - overlay overlay rw,lowerdir=/var/lib/docker/overlay2/l/3NGKRR7MJ5V7ZP63M5O3CD4V5C:/var/lib/docker/overlay2/l/37E5ZP3VHC7XYRN5P7KRDTCVEJ:/var/lib/docker/overlay2/l/UUXDZS5DJDBWOWJHQPX6GXG6V4:/var/lib/docker/overlay2/l/SUCHM6IWJFEGQKVCQBMT6NLSRP:/var/lib/docker/overlay2/l/2VSS7FFROHYACCBUATFGP4ZZON:/var/lib/docker/overlay2/l/KJ737QUPMJK3PGXEEQFOG4DMHJ,upperdir=/var/lib/docker/overlay2/05d0e0edf6aeac19cfffda551b35fc4a5f76f4477dcfbceca9e88ae9aa277f22/diff,workdir=/var/lib/docker/overlay2/05d0e0edf6aeac19cfffda551b35fc4a5f76f4477dcfbceca9e88ae9aa277f22/work
718 614 0:125 / /zhdd/home/xyzhi rw,noatime shared:382 - nfs4 192.168.1.99:/zhdd/home/xyzhi rw,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,nocto,proto=tcp,nconnect=16,timeo=600,retrans=2,sec=sys,clientaddr=192.168.1.11,fsc,local_lock=none,addr=192.168.1.99
1239 28 0:130 / /run/user/1024 rw,nosuid,nodev,relatime shared:638 - tmpfs tmpfs rw,size=52799544k,nr_inodes=13199886,mode=700,uid=1024,gid=1024,inode64
610 614 0:89 / /zhdd/home/yfye rw,noatime shared:381 - nfs4 192.168.1.99:/zhdd/home/yfye rw,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,nocto,proto=tcp,nconnect=16,timeo=600,retrans=2,sec=sys,clientaddr=192.168.1.11,fsc,local_lock=none,addr=192.168.1.99
959 28 0:88 / /run/user/1023 rw,nosuid,nodev,relatime shared:532 - tmpfs tmpfs rw,size=52799544k,nr_inodes=13199886,mode=700,uid=1023,gid=1023,inode64
624 614 0:48 / /zhdd/home/jqzhang rw,noatime shared:62 - nfs4 192.168.1.99:/zhdd/home/jqzhang rw,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,nocto,proto=tcp,nconnect=16,timeo=600,retrans=2,sec=sys,clientaddr=192.168.1.11,fsc,local_lock=none,addr=192.168.1.99
625 614 0:119 / /zhdd/home/xlhuang rw,noatime shared:425 - nfs4 192.168.1.99:/zhdd/home/xlhuang rw,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,nocto,proto=tcp,nconnect=16,timeo=600,retrans=2,sec=sys,clientaddr=192.168.1.11,fsc,local_lock=none,addr=192.168.1.99
1226 28 0:67 / /run/user/1016 rw,nosuid,nodev,relatime shared:641 - tmpfs tmpfs rw,size=52799544k,nr_inodes=13199886,mode=700,uid=1016,gid=1016,inode64
1665 30 7:6 / /snap/core20/2379 ro,nodev,relatime shared:749 - squashfs /dev/loop6 ro,errors=continue
153 28 0:77 / /run/user/1012 rw,nosuid,nodev,relatime shared:51 - tmpfs tmpfs rw,size=52799544k,nr_inodes=13199886,mode=700,uid=1012,gid=1012,inode64
1194 614 0:85 / /zhdd/home/tjshen rw,noatime shared:458 - nfs4 192.168.1.99:/zhdd/home/tjshen rw,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,nocto,proto=tcp,nconnect=16,timeo=600,retrans=2,sec=sys,clientaddr=192.168.1.11,fsc,local_lock=none,addr=192.168.1.99
1260 28 0:80 / /run/user/1002 rw,nosuid,nodev,relatime shared:590 - tmpfs tmpfs rw,size=52799544k,nr_inodes=13199886,mode=700,uid=1002,gid=1002,inode64
(base) xlhuang@a100:~$ cat /proc/filesystems
nodev sysfs
nodev tmpfs
nodev bdev
nodev proc
nodev cgroup
nodev cgroup2
nodev cpuset
nodev devtmpfs
nodev configfs
nodev debugfs
nodev tracefs
nodev securityfs
nodev sockfs
nodev bpf
nodev pipefs
nodev ramfs
nodev hugetlbfs
nodev devpts
ext3
ext2
ext4
squashfs
vfat
nodev ecryptfs
fuseblk
nodev fuse
nodev fusectl
nodev efivarfs
nodev mqueue
nodev resctrl
nodev pstore
btrfs
xfs
nodev autofs
nodev rpc_pipefs
nodev nfsd
nodev zfs
nodev binfmt_misc
nodev nfs
nodev nfs4
nodev overlay
(base) xlhuang@a100:~$You can see among them
625 614 0:119 / /zhdd/home/xlhuang rw,noatime shared:425 - nfs4 192.168.1.99:/zhdd/home/xlhuang rw,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,nocto,proto=tcp,nconnect=16,timeo=600,retrans=2,sec=sys,clientaddr=192.168.1.11,fsc,local_lock=none,addr=192.168.1.99
98 30 259:6 / /data rw,relatime shared:49 - ext4 /dev/md126p1 rw,stripe=32As mentioned earlier, since /home/xlhuang (linked to /zhdd/home/xlhuang) is a remote directory, I have changed the graphroot to a local directory under /data:
graphroot = "/data/xlhuang/.local/lib/containers/storage"The content of /home/xlhuang/.config/containers/containers.conf is as follows:
# See https://github.com/containers/common/blob/master/pkg/config/containers.conf
[engine]
cgroup_manager = "cgroupfs"
events_logger="file"
conmon_path = ["/home/xlhuang/.local/lib/podman/conmon"]
helper_binaries_dir = ["/home/xlhuang/.local/lib/podman"]
[containers]
init_path = "/home/xlhuang/.local/lib/podman/catatonit"The content of /home/xlhuang/.config/containers/storage.conf is as follows:
# See https://github.com/containers/storage/blob/master/docs/containers-storage.conf.5.md
# and https://github.com/containers/storage/blob/master/storage.conf
# This file is is the configuration file for all tools
# that use the containers/storage library.
# See man 5 containers-storage.conf for more information
# The "container storage" table contains all of the server options.
[storage]
# Default Storage Driver, Must be set for proper operation.
driver = "overlay"
# Temporary storage location
runroot = "/run/user/1016"
# Primary Read/Write location of container storage
# graphroot = "/home/xlhuang/.local/lib/containers/storage"
graphroot = "/data/xlhuang/.local/lib/containers/storage"
# Storage path for rootless users
#
# rootless_storage_path = "$HOME/.local/share/containers/storage"
[storage.options]
# Storage options to be passed to underlying storage drivers
# AdditionalImageStores is used to pass paths to additional Read/Only image stores
# Must be comma separated list.
additionalimagestores = [
]
# Remap-UIDs/GIDs is the mapping from UIDs/GIDs as they should appear inside of
# a container, to the UIDs/GIDs as they should appear outside of the container,
# and the length of the range of UIDs/GIDs. Additional mapped sets can be
# listed and will be heeded by libraries, but there are limits to the number of
# mappings which the kernel will allow when you later attempt to run a
# container.
#
# remap-uids = 0:1668442479:65536
# remap-gids = 0:1668442479:65536
# Remap-User/Group is a user name which can be used to look up one or more UID/GID
# ranges in the /etc/subuid or /etc/subgid file. Mappings are set up starting
# with an in-container ID of 0 and then a host-level ID taken from the lowest
# range that matches the specified name, and using the length of that range.
# Additional ranges are then assigned, using the ranges which specify the
# lowest host-level IDs first, to the lowest not-yet-mapped in-container ID,
# until all of the entries have been used for maps.
#
# remap-user = "containers"
# remap-group = "containers"
# Root-auto-userns-user is a user name which can be used to look up one or more UID/GID
# ranges in the /etc/subuid and /etc/subgid file. These ranges will be partitioned
# to containers configured to create automatically a user namespace. Containers
# configured to automatically create a user namespace can still overlap with containers
# having an explicit mapping set.
# This setting is ignored when running as rootless.
# root-auto-userns-user = "storage"
#
# Auto-userns-min-size is the minimum size for a user namespace created automatically.
# auto-userns-min-size=1024
#
# Auto-userns-max-size is the minimum size for a user namespace created automatically.
# auto-userns-max-size=65536
[storage.options.overlay]
# ignore_chown_errors can be set to allow a non privileged user running with
# a single UID within a user namespace to run containers. The user can pull
# and use any image even those with multiple uids. Note multiple UIDs will be
# squashed down to the default uid in the container. These images will have no
# separation between the users in the container. Only supported for the overlay
# and vfs drivers.
ignore_chown_errors = "true"
# Path to an helper program to use for mounting the file system instead of mounting it
# directly.
mount_program = "/home/xlhuang/.local/bin/fuse-overlayfs"
# mountopt specifies comma separated list of extra mount options
mountopt = "nodev,fsync=0"
# Set to skip a PRIVATE bind mount on the storage home directory.
# skip_mount_home = "false"
# Size is used to set a maximum size of the container image.
# size = ""
# ForceMask specifies the permissions mask that is used for new files and
# directories.
#
# The values "shared" and "private" are accepted.
# Octal permission masks are also accepted.
#
# "": No value specified.
# All files/directories, get set with the permissions identified within the
# image.
# "private": it is equivalent to 0700.
# All files/directories get set with 0700 permissions. The owner has rwx
# access to the files. No other users on the system can access the files.
# This setting could be used with networked based homedirs.
# "shared": it is equivalent to 0755.
# The owner has rwx access to the files and everyone else can read, access
# and execute them. This setting is useful for sharing containers storage
# with other users. For instance have a storage owned by root but shared
# to rootless users as an additional store.
# NOTE: All files within the image are made readable and executable by any
# user on the system. Even /etc/shadow within your image is now readable by
# any user.
#
# OCTAL: Users can experiment with other OCTAL Permissions.
#
# Note: The force_mask Flag is an experimental feature, it could change in the
# future. When "force_mask" is set the original permission mask is stored in
# the "user.containers.override_stat" xattr and the "mount_program" option must
# be specified. Mount programs like "/usr/bin/fuse-overlayfs" present the
# extended attribute permissions to processes within containers rather then the
# "force_mask" permissions.
#
# force_mask = ""
[storage.options.thinpool]
# Storage Options for thinpool
# autoextend_percent determines the amount by which pool needs to be
# grown. This is specified in terms of % of pool size. So a value of 20 means
# that when threshold is hit, pool will be grown by 20% of existing
# pool size.
# autoextend_percent = "20"
# autoextend_threshold determines the pool extension threshold in terms
# of percentage of pool size. For example, if threshold is 60, that means when
# pool is 60% full, threshold has been hit.
# autoextend_threshold = "80"
# basesize specifies the size to use when creating the base device, which
# limits the size of images and containers.
# basesize = "10G"
# blocksize specifies a custom blocksize to use for the thin pool.
# blocksize="64k"
# directlvm_device specifies a custom block storage device to use for the
# thin pool. Required if you setup devicemapper.
# directlvm_device = ""
# directlvm_device_force wipes device even if device already has a filesystem.
# directlvm_device_force = "True"
# fs specifies the filesystem type to use for the base device.
# fs="xfs"
# log_level sets the log level of devicemapper.
# 0: LogLevelSuppress 0 (Default)
# 2: LogLevelFatal
# 3: LogLevelErr
# 4: LogLevelWarn
# 5: LogLevelNotice
# 6: LogLevelInfo
# 7: LogLevelDebug
# log_level = "7"
# min_free_space specifies the min free space percent in a thin pool require for
# new device creation to succeed. Valid values are from 0% - 99%.
# Value 0% disables
# min_free_space = "10%"
# mkfsarg specifies extra mkfs arguments to be used when creating the base
# device.
# mkfsarg = ""
# metadata_size is used to set the `pvcreate --metadatasize` options when
# creating thin devices. Default is 128k
# metadata_size = ""
# Size is used to set a maximum size of the container image.
# size = ""
# use_deferred_removal marks devicemapper block device for deferred removal.
# If the thinpool is in use when the driver attempts to remove it, the driver
# tells the kernel to remove it as soon as possible. Note this does not free
# up the disk space, use deferred deletion to fully remove the thinpool.
# use_deferred_removal = "True"
# use_deferred_deletion marks thinpool device for deferred deletion.
# If the device is busy when the driver attempts to delete it, the driver
# will attempt to delete device every 30 seconds until successful.
# If the program using the driver exits, the driver will continue attempting
# to cleanup the next time the driver is used. Deferred deletion permanently
# deletes the device and all data stored in device will be lost.
# use_deferred_deletion = "True"
# xfs_nospace_max_retries specifies the maximum number of retries XFS should
# attempt to complete IO when ENOSPC (no space) error is returned by
# underlying storage device.
# xfs_nospace_max_retries = "0"But the problem described in this issue still occurs.
giuseppe
commented
1 month ago thanks.
What are the permissions on /data and /data/xlhuang/? Can you show the output of id and ls -ldn / /data /data/xlhuang?
I wonder if you have access to these directories through group ownership. In that case you might lose access once you are in a user namespace.
HareToAme
commented
1 month ago (base) xlhuang@a100:~$ id
uid=1016(xlhuang) gid=1016(xlhuang) groups=1016(xlhuang),999(docker)
(base) xlhuang@a100:~$ ls -ldn / /data /data/xlhuang
drwxr-xr-x 22 0 0 300 Aug 8 02:50 /
drwxrwxrwx 23 0 0 4096 Aug 6 16:05 /data
drwxr-xr-x 7 1016 1016 4096 Sep 10 19:42 /data/xlhuang
(base) xlhuang@a100:~$thanks that looks correct.
I've looked more carefully at your mount points and I see the following mounts:
1635 25 0:27 / /proc/2806118 rw,nosuid,nodev shared:4 - tmpfs tmpfs rw,inode64
1923 25 0:27 / /proc/2809415 rw,nosuid,nodev shared:4 - tmpfs tmpfs rw,inode64I think that is the reason why you cannot mount proc. Inside a user namespace, you can mount a procfs only if a proc mount is already fully visible. In your case it is not, since there are these tmpfs mounts that cover a part of the proc tree.
HareToAme
commented
1 month ago Thank you for your response. Is there a simple method or command that can solve this problem? I can report it to the server administrator to execute and fix the issue.
giuseppe
commented
1 month ago you'll need to run umount /proc/2806118 /proc/2809415. That must be done as root though, as unprivileged user you can only run without a pid namespace: -v /proc:/proc --pid host
HareToAme
commented
1 month ago Thank you very much! I have successfully run the container.
I will close this issue, and I hope this example will be helpful to anyone with a similar situation.
(base) xlhuang@a100:~$ podman --log-level debug run -v /proc:/proc --pid host --name ubuntu_me -it ubuntu:22.04
INFO[0000] podman filtering at log level debug
DEBU[0000] Called run.PersistentPreRunE(podman --log-level debug run -v /proc:/proc --pid host --name ubuntu_me -it ubuntu:22.04)
DEBU[0000] Using conmon: "/home/xlhuang/.local/lib/podman/conmon"
INFO[0000] Using sqlite as database backend
DEBU[0000] systemd-logind: Unknown object '/'.
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /data/xlhuang/.local/lib/containers/storage
DEBU[0000] Using run root /run/user/1016
DEBU[0000] Using static dir /data/xlhuang/.local/lib/containers/storage/libpod
DEBU[0000] Using tmp dir /run/user/1016/libpod/tmp
DEBU[0000] Using volume path /data/xlhuang/.local/lib/containers/storage/volumes
DEBU[0000] Using transient store: false
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] overlay: ignore_chown_errors=true
DEBU[0000] overlay: mount_program=/home/xlhuang/.local/bin/fuse-overlayfs
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false
DEBU[0000] Initializing event backend file
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument
DEBU[0000] using runtime "crun" from $PATH: "/home/xlhuang/.local/bin/crun"
DEBU[0000] Configured OCI runtime crun-vm initialization failed: no valid executable found for OCI runtime crun-vm: invalid argument
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument
DEBU[0000] Using OCI runtime "/home/xlhuang/.local/bin/crun"
INFO[0000] Setting parallel job count to 193
DEBU[0000] Pulling image ubuntu:22.04 (policy: missing)
DEBU[0000] Looking up image "ubuntu:22.04" in local containers storage
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux [] }
DEBU[0000] Loading registries configuration "/home/xlhuang/.config/containers/registries.conf"
DEBU[0000] Trying "docker.io/library/ubuntu:22.04" ...
DEBU[0000] parsed reference into "[overlay@/data/xlhuang/.local/lib/containers/storage+/run/user/1016:overlay.ignore_chown_errors=true,overlay.mount_program=/home/xlhuang/.local/bin/fuse-overlayfs,overlay.mountopt=nodev,fsync=0]@53a843653cbcd9e10be207e951d907dc2481d9c222de57d24cfcac32e5165188"
DEBU[0000] Found image "ubuntu:22.04" as "docker.io/library/ubuntu:22.04" in local containers storage
DEBU[0000] Found image "ubuntu:22.04" as "docker.io/library/ubuntu:22.04" in local containers storage ([overlay@/data/xlhuang/.local/lib/containers/storage+/run/user/1016:overlay.ignore_chown_errors=true,overlay.mount_program=/home/xlhuang/.local/bin/fuse-overlayfs,overlay.mountopt=nodev,fsync=0]@53a843653cbcd9e10be207e951d907dc2481d9c222de57d24cfcac32e5165188)
DEBU[0000] exporting opaque data as blob "sha256:53a843653cbcd9e10be207e951d907dc2481d9c222de57d24cfcac32e5165188"
DEBU[0000] Looking up image "docker.io/library/ubuntu:22.04" in local containers storage
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux [] }
DEBU[0000] Trying "docker.io/library/ubuntu:22.04" ...
DEBU[0000] parsed reference into "[overlay@/data/xlhuang/.local/lib/containers/storage+/run/user/1016:overlay.ignore_chown_errors=true,overlay.mount_program=/home/xlhuang/.local/bin/fuse-overlayfs,overlay.mountopt=nodev,fsync=0]@53a843653cbcd9e10be207e951d907dc2481d9c222de57d24cfcac32e5165188"
DEBU[0000] Found image "docker.io/library/ubuntu:22.04" as "docker.io/library/ubuntu:22.04" in local containers storage
DEBU[0000] Found image "docker.io/library/ubuntu:22.04" as "docker.io/library/ubuntu:22.04" in local containers storage ([overlay@/data/xlhuang/.local/lib/containers/storage+/run/user/1016:overlay.ignore_chown_errors=true,overlay.mount_program=/home/xlhuang/.local/bin/fuse-overlayfs,overlay.mountopt=nodev,fsync=0]@53a843653cbcd9e10be207e951d907dc2481d9c222de57d24cfcac32e5165188)
DEBU[0000] exporting opaque data as blob "sha256:53a843653cbcd9e10be207e951d907dc2481d9c222de57d24cfcac32e5165188"
DEBU[0000] User mount /proc:/proc options []
DEBU[0000] Looking up image "ubuntu:22.04" in local containers storage
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux [] }
DEBU[0000] Trying "docker.io/library/ubuntu:22.04" ...
DEBU[0000] parsed reference into "[overlay@/data/xlhuang/.local/lib/containers/storage+/run/user/1016:overlay.ignore_chown_errors=true,overlay.mount_program=/home/xlhuang/.local/bin/fuse-overlayfs,overlay.mountopt=nodev,fsync=0]@53a843653cbcd9e10be207e951d907dc2481d9c222de57d24cfcac32e5165188"
DEBU[0000] Found image "ubuntu:22.04" as "docker.io/library/ubuntu:22.04" in local containers storage
DEBU[0000] Found image "ubuntu:22.04" as "docker.io/library/ubuntu:22.04" in local containers storage ([overlay@/data/xlhuang/.local/lib/containers/storage+/run/user/1016:overlay.ignore_chown_errors=true,overlay.mount_program=/home/xlhuang/.local/bin/fuse-overlayfs,overlay.mountopt=nodev,fsync=0]@53a843653cbcd9e10be207e951d907dc2481d9c222de57d24cfcac32e5165188)
DEBU[0000] exporting opaque data as blob "sha256:53a843653cbcd9e10be207e951d907dc2481d9c222de57d24cfcac32e5165188"
DEBU[0000] Inspecting image 53a843653cbcd9e10be207e951d907dc2481d9c222de57d24cfcac32e5165188
DEBU[0000] exporting opaque data as blob "sha256:53a843653cbcd9e10be207e951d907dc2481d9c222de57d24cfcac32e5165188"
DEBU[0000] Inspecting image 53a843653cbcd9e10be207e951d907dc2481d9c222de57d24cfcac32e5165188
DEBU[0000] Inspecting image 53a843653cbcd9e10be207e951d907dc2481d9c222de57d24cfcac32e5165188
DEBU[0000] Inspecting image 53a843653cbcd9e10be207e951d907dc2481d9c222de57d24cfcac32e5165188
DEBU[0000] using systemd mode: false
DEBU[0000] setting container name ubuntu_me
DEBU[0000] No hostname set; container's hostname will default to runtime default
DEBU[0000] Loading default seccomp profile
DEBU[0000] Adding mount /dev
DEBU[0000] Adding mount /dev/pts
DEBU[0000] Adding mount /dev/mqueue
DEBU[0000] Adding mount /sys
DEBU[0000] Adding mount /sys/fs/cgroup
DEBU[0000] Allocated lock 0 for container cb3bebd5aec61425ff71bd7c21b1ac8734c7030c749ce27eec5af1bd11749518
DEBU[0000] exporting opaque data as blob "sha256:53a843653cbcd9e10be207e951d907dc2481d9c222de57d24cfcac32e5165188"
DEBU[0000] Created container "cb3bebd5aec61425ff71bd7c21b1ac8734c7030c749ce27eec5af1bd11749518"
DEBU[0000] Container "cb3bebd5aec61425ff71bd7c21b1ac8734c7030c749ce27eec5af1bd11749518" has work directory "/data/xlhuang/.local/lib/containers/storage/overlay-containers/cb3bebd5aec61425ff71bd7c21b1ac8734c7030c749ce27eec5af1bd11749518/userdata"
DEBU[0000] Container "cb3bebd5aec61425ff71bd7c21b1ac8734c7030c749ce27eec5af1bd11749518" has run directory "/run/user/1016/overlay-containers/cb3bebd5aec61425ff71bd7c21b1ac8734c7030c749ce27eec5af1bd11749518/userdata"
DEBU[0000] Handling terminal attach
INFO[0000] Received shutdown.Stop(), terminating! PID=2867581
DEBU[0000] Enabling signal proxying
DEBU[0000] overlay: mount_data=lowerdir=/data/xlhuang/.local/lib/containers/storage/overlay/l/CRDDWRJAW5MIO6OMMFIWGFD7UO,upperdir=/data/xlhuang/.local/lib/containers/storage/overlay/3f389567dbc247310dbef916d63464d703e05964de2e29d5845a218e4aa6943b/diff,workdir=/data/xlhuang/.local/lib/containers/storage/overlay/3f389567dbc247310dbef916d63464d703e05964de2e29d5845a218e4aa6943b/work,nodev,fsync=0
DEBU[0000] Made network namespace at /run/user/1016/netns/netns-149cfe7c-e476-25b9-7093-d041377284a2 for container cb3bebd5aec61425ff71bd7c21b1ac8734c7030c749ce27eec5af1bd11749518
DEBU[0000] pasta arguments: --config-net --dns-forward 169.254.0.1 -t none -u none -T none -U none --no-map-gw --quiet --netns /run/user/1016/netns/netns-149cfe7c-e476-25b9-7093-d041377284a2
DEBU[0000] Mounted container "cb3bebd5aec61425ff71bd7c21b1ac8734c7030c749ce27eec5af1bd11749518" at "/data/xlhuang/.local/lib/containers/storage/overlay/3f389567dbc247310dbef916d63464d703e05964de2e29d5845a218e4aa6943b/merged"
DEBU[0000] Created root filesystem for container cb3bebd5aec61425ff71bd7c21b1ac8734c7030c749ce27eec5af1bd11749518 at /data/xlhuang/.local/lib/containers/storage/overlay/3f389567dbc247310dbef916d63464d703e05964de2e29d5845a218e4aa6943b/merged
DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode subscription
DEBU[0000] reading hooks from /usr/share/containers/oci/hooks.d
DEBU[0000] Workdir "/" resolved to host path "/data/xlhuang/.local/lib/containers/storage/overlay/3f389567dbc247310dbef916d63464d703e05964de2e29d5845a218e4aa6943b/merged"
DEBU[0000] Created OCI spec for container cb3bebd5aec61425ff71bd7c21b1ac8734c7030c749ce27eec5af1bd11749518 at /data/xlhuang/.local/lib/containers/storage/overlay-containers/cb3bebd5aec61425ff71bd7c21b1ac8734c7030c749ce27eec5af1bd11749518/userdata/config.json
DEBU[0000] /home/xlhuang/.local/lib/podman/conmon messages will be logged to syslog
DEBU[0000] running conmon: /home/xlhuang/.local/lib/podman/conmon args="[--api-version 1 -c cb3bebd5aec61425ff71bd7c21b1ac8734c7030c749ce27eec5af1bd11749518 -u cb3bebd5aec61425ff71bd7c21b1ac8734c7030c749ce27eec5af1bd11749518 -r /home/xlhuang/.local/bin/crun -b /data/xlhuang/.local/lib/containers/storage/overlay-containers/cb3bebd5aec61425ff71bd7c21b1ac8734c7030c749ce27eec5af1bd11749518/userdata -p /run/user/1016/overlay-containers/cb3bebd5aec61425ff71bd7c21b1ac8734c7030c749ce27eec5af1bd11749518/userdata/pidfile -n ubuntu_me --exit-dir /run/user/1016/libpod/tmp/exits --persist-dir /run/user/1016/libpod/tmp/persist/cb3bebd5aec61425ff71bd7c21b1ac8734c7030c749ce27eec5af1bd11749518 --full-attach -l k8s-file:/data/xlhuang/.local/lib/containers/storage/overlay-containers/cb3bebd5aec61425ff71bd7c21b1ac8734c7030c749ce27eec5af1bd11749518/userdata/ctr.log --log-level debug --syslog -t --conmon-pidfile /run/user/1016/overlay-containers/cb3bebd5aec61425ff71bd7c21b1ac8734c7030c749ce27eec5af1bd11749518/userdata/conmon.pid --exit-command /zhdd/home/xlhuang/.local/bin/podman --exit-command-arg --root --exit-command-arg /data/xlhuang/.local/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/1016 --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg cgroupfs --exit-command-arg --tmpdir --exit-command-arg /run/user/1016/libpod/tmp --exit-command-arg --network-config-dir --exit-command-arg --exit-command-arg --network-backend --exit-command-arg netavark --exit-command-arg --volumepath --exit-command-arg /data/xlhuang/.local/lib/containers/storage/volumes --exit-command-arg --db-backend --exit-command-arg sqlite --exit-command-arg --transient-store=false --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.ignore_chown_errors=true --exit-command-arg --storage-opt --exit-command-arg overlay.mount_program=/home/xlhuang/.local/bin/fuse-overlayfs --exit-command-arg --storage-opt --exit-command-arg overlay.mountopt=nodev,fsync=0 --exit-command-arg --events-backend --exit-command-arg file --exit-command-arg --syslog --exit-command-arg container --exit-command-arg cleanup --exit-command-arg cb3bebd5aec61425ff71bd7c21b1ac8734c7030c749ce27eec5af1bd11749518]"
INFO[0000] Failed to add conmon to cgroupfs sandbox cgroup: creating cgroup path conmon: open /sys/fs/cgroup/cgroup.subtree_control: permission denied
DEBU[0000] Received: 2867619
INFO[0000] Got Conmon PID as 2867616
DEBU[0000] Created container cb3bebd5aec61425ff71bd7c21b1ac8734c7030c749ce27eec5af1bd11749518 in OCI runtime
DEBU[0000] found local resolver, using "/run/systemd/resolve/resolv.conf" to get the nameservers
DEBU[0000] Attaching to container cb3bebd5aec61425ff71bd7c21b1ac8734c7030c749ce27eec5af1bd11749518
DEBU[0000] Starting container cb3bebd5aec61425ff71bd7c21b1ac8734c7030c749ce27eec5af1bd11749518 with command [/bin/bash]
DEBU[0000] Received a resize event: {Width:184 Height:30}
DEBU[0000] Started container cb3bebd5aec61425ff71bd7c21b1ac8734c7030c749ce27eec5af1bd11749518
DEBU[0000] Notify sent successfully
root@cb3bebd5aec6:/# DEBU[0014] Received a resize event: {Width:187 Height:31}
DEBU[0014] Sending signal 28 to container cb3bebd5aec61425ff71bd7c21b1ac8734c7030c749ce27eec5af1bd11749518
root@cb3bebd5aec6:/# DEBU[0014] Received a resize event: {Width:187 Height:32}
DEBU[0014] Received a resize event: {Width:187 Height:33}
DEBU[0014] Sending signal 28 to container cb3bebd5aec61425ff71bd7c21b1ac8734c7030c749ce27eec5af1bd11749518
root@cb3bebd5aec6:/# DEBU[0014] Sending signal 28 to container cb3bebd5aec61425ff71bd7c21b1ac8734c7030c749ce27eec5af1bd11749518
root@cb3bebd5aec6:/# DEBU[0014] Received a resize event: {Width:187 Height:34}
DEBU[0014] Sending signal 28 to container cb3bebd5aec61425ff71bd7c21b1ac8734c7030c749ce27eec5af1bd11749518
root@cb3bebd5aec6:/# DEBU[0014] Sending signal 28 to container cb3bebd5aec61425ff71bd7c21b1ac8734c7030c749ce27eec5af1bd11749518
DEBU[0014] Received a resize event: {Width:187 Height:35}
root@cb3bebd5aec6:/# DEBU[0014] Received a resize event: {Width:187 Height:36}
DEBU[0014] Received a resize event: {Width:187 Height:37}
root@cb3bebd5aec6:/# DEBU[0014] Sending signal 28 to container cb3bebd5aec61425ff71bd7c21b1ac8734c7030c749ce27eec5af1bd11749518
root@cb3bebd5aec6:/# DEBU[0014] Sending signal 28 to container cb3bebd5aec61425ff71bd7c21b1ac8734c7030c749ce27eec5af1bd11749518
root@cb3bebd5aec6:/# DEBU[0014] Received a resize event: {Width:187 Height:39}
DEBU[0014] Sending signal 28 to container cb3bebd5aec61425ff71bd7c21b1ac8734c7030c749ce27eec5af1bd11749518
root@cb3bebd5aec6:/# DEBU[0014] Received a resize event: {Width:187 Height:41}
DEBU[0014] Sending signal 28 to container cb3bebd5aec61425ff71bd7c21b1ac8734c7030c749ce27eec5af1bd11749518
root@cb3bebd5aec6:/# DEBU[0014] Received a resize event: {Width:187 Height:42}
DEBU[0014] Sending signal 28 to container cb3bebd5aec61425ff71bd7c21b1ac8734c7030c749ce27eec5af1bd11749518
root@cb3bebd5aec6:/# DEBU[0014] Received a resize event: {Width:187 Height:43}
DEBU[0014] Sending signal 28 to container cb3bebd5aec61425ff71bd7c21b1ac8734c7030c749ce27eec5af1bd11749518
root@cb3bebd5aec6:/# DEBU[0014] Received a resize event: {Width:187 Height:45}
DEBU[0014] Sending signal 28 to container cb3bebd5aec61425ff71bd7c21b1ac8734c7030c749ce27eec5af1bd11749518
root@cb3bebd5aec6:/# DEBU[0014] Received a resize event: {Width:187 Height:46}
DEBU[0014] Sending signal 28 to container cb3bebd5aec61425ff71bd7c21b1ac8734c7030c749ce27eec5af1bd11749518
root@cb3bebd5aec6:/# DEBU[0015] Received a resize event: {Width:187 Height:47}
DEBU[0015] Sending signal 28 to container cb3bebd5aec61425ff71bd7c21b1ac8734c7030c749ce27eec5af1bd11749518
root@cb3bebd5aec6:/# DEBU[0015] Received a resize event: {Width:187 Height:48}
DEBU[0015] Sending signal 28 to container cb3bebd5aec61425ff71bd7c21b1ac8734c7030c749ce27eec5af1bd11749518
root@cb3bebd5aec6:/# DEBU[0015] Received a resize event: {Width:187 Height:49}
root@cb3bebd5aec6:/# DEBU[0015] Sending signal 28 to container cb3bebd5aec61425ff71bd7c21b1ac8734c7030c749ce27eec5af1bd11749518
root@cb3bebd5aec6:/# DEBU[0015] Received a resize event: {Width:187 Height:50}
DEBU[0015] Sending signal 28 to container cb3bebd5aec61425ff71bd7c21b1ac8734c7030c749ce27eec5af1bd11749518
root@cb3bebd5aec6:/# DEBU[0015] Received a resize event: {Width:187 Height:51}
DEBU[0015] Sending signal 28 to container cb3bebd5aec61425ff71bd7c21b1ac8734c7030c749ce27eec5af1bd11749518
root@cb3bebd5aec6:/# DEBU[0015] Received a resize event: {Width:187 Height:52}
DEBU[0015] Sending signal 28 to container cb3bebd5aec61425ff71bd7c21b1ac8734c7030c749ce27eec5af1bd11749518
root@cb3bebd5aec6:/# DEBU[0015] Received a resize event: {Width:187 Height:53}
DEBU[0015] Sending signal 28 to container cb3bebd5aec61425ff71bd7c21b1ac8734c7030c749ce27eec5af1bd11749518
DEBU[0015] Received a resize event: {Width:187 Height:54}
root@cb3bebd5aec6:/# DEBU[0015] Received a resize event: {Width:187 Height:55}
root@cb3bebd5aec6:/# DEBU[0015] Sending signal 28 to container cb3bebd5aec61425ff71bd7c21b1ac8734c7030c749ce27eec5af1bd11749518
root@cb3bebd5aec6:/# DEBU[0015] Sending signal 28 to container cb3bebd5aec61425ff71bd7c21b1ac8734c7030c749ce27eec5af1bd11749518
root@cb3bebd5aec6:/# DEBU[0015] Received a resize event: {Width:187 Height:56}
DEBU[0015] Sending signal 28 to container cb3bebd5aec61425ff71bd7c21b1ac8734c7030c749ce27eec5af1bd11749518
root@cb3bebd5aec6:/# DEBU[0015] Received a resize event: {Width:187 Height:57}
DEBU[0015] Sending signal 28 to container cb3bebd5aec61425ff71bd7c21b1ac8734c7030c749ce27eec5af1bd11749518
root@cb3bebd5aec6:/# DEBU[0015] Received a resize event: {Width:187 Height:58}
DEBU[0015] Sending signal 28 to container cb3bebd5aec61425ff71bd7c21b1ac8734c7030c749ce27eec5af1bd11749518
root@cb3bebd5aec6:/# DEBU[0015] Received a resize event: {Width:187 Height:59}
DEBU[0015] Sending signal 28 to container cb3bebd5aec61425ff71bd7c21b1ac8734c7030c749ce27eec5af1bd11749518
root@cb3bebd5aec6:/# DEBU[0015] Received a resize event: {Width:187 Height:60}
DEBU[0015] Sending signal 28 to container cb3bebd5aec61425ff71bd7c21b1ac8734c7030c749ce27eec5af1bd11749518
root@cb3bebd5aec6:/# DEBU[0015] Received a resize event: {Width:187 Height:61}
root@cb3bebd5aec6:/# DEBU[0015] Sending signal 28 to container cb3bebd5aec61425ff71bd7c21b1ac8734c7030c749ce27eec5af1bd11749518
root@cb3bebd5aec6:/# DEBU[0015] Received a resize event: {Width:187 Height:62}
root@cb3bebd5aec6:/# DEBU[0015] Sending signal 28 to container cb3bebd5aec61425ff71bd7c21b1ac8734c7030c749ce27eec5af1bd11749518
root@cb3bebd5aec6:/# DEBU[0015] Received a resize event: {Width:187 Height:63}
DEBU[0015] Sending signal 28 to container cb3bebd5aec61425ff71bd7c21b1ac8734c7030c749ce27eec5af1bd11749518
root@cb3bebd5aec6:/#
Issue Description
This issue seems to be specific to the machine; the same operation yields different results (success or failure) on different machines. I found some similar issues as follows, but they didn't resolve my error.
For detailed information, please see the text below.
Steps to reproduce the issue
As a non-root user, install Podman using the following commands:
The user then changes
graphroot = "/home/xlhuang/.local/lib/containers/storage"tographroot = "/data/xlhuang/.local/lib/containers/storage". (This change has no effect on the issue)Attempt to start a container:
Describe the results you received
When executing the above command, the following error occurs:
and finally
Here is the complete log information.
Added
--cgroups=disabledparameter to the command:This prevented the
Failed to add conmon to cgroupfs sandbox cgroup...error, but the final errorError: container create failed (no logs from conmon): conmon bytes ""...remained the same.Describe the results you expected
The same steps executed on a VMware Linux virtual machine (Ubuntu 20.04) as a non-root user worked successfully.
Here is the complete log information.
podman info output
Podman in a container
No
Privileged Or Rootless
Rootless
Upstream Latest Release
Yes
Additional environment details
No response
Additional information
The initial error that appeared was:
However, after trying again, the error consistently became:
I have reason to believe that this issue is related to some changes on the server. Initially, I was using
podman-4.2.1-r1-x86_64.AppImageon the server without any problems, but one day it suddenly stopped working and the aforementioned error occurred. Therefore, I tried using the latest version5.2.2, but the same error still appeared on that server. I am unable to determine the cause of the problem.Since I belong to the docker group on the server, my temporary solution is to replace podman with docker to execute the startup command. This will not cause any errors.