search

containers / podman

Podman: A tool for managing OCI containers and pods.
https://podman.io
Apache License 2.0
23.73k stars 2.41k forks source link

sometimes, /usr/bin/fuse-overlayfs remains after container exist. #7061

Closed coldbloodx closed 4 years ago

coldbloodx commented 4 years ago

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Steps to reproduce the issue:

1.start a container: [xianwu@rcn07 ~]$ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 34335b25d1e4 localhost/centos:latest /home/xianwu/.lsb... About a minute ago Exited (0) 26 seconds ago ulaworks008.job.4039

  1. after the container exit, check container related processes, the fuse-overlayfs remains [xianwu@rcn07 ~]$ ps -ewf |grep 205298 xianwu 205298 1 0 08:21 ? 00:00:00 /usr/bin/fuse-overlayfs -o lowerdir=/podmanfs/overlay/l/D6ZILYLI2BLBDF65YSKZRRBKOB:/podmanfs/overlay/l/UNZMQEUSSGU7ZCGQZ6ICUEFNHX,upperdir=/podmanfs/overlay/6ea6e0c4da2c6f350a07279c6d71525be9f8c9cb66cc53272b6d42a8a60b26f7/diff,workdir=/podmanfs/overlay/6ea6e0c4da2c6f350a07279c6d71525be9f8c9cb66cc53272b6d42a8a60b26f7/work,uidmapping=0:1:33857:33857:0:1:33858:33858:31679,gidmapping=0:1:10007:10007:0:1:10008:10008:55529 /podmanfs/overlay/6ea6e0c4da2c6f350a07279c6d71525be9f8c9cb66cc53272b6d42a8a60b26f7/merged

Describe the results you received: the /usr/bin/fuse-overlayfs remains.

Describe the results you expected: the /usr/bin/fuse-overlayfs exists automatically.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

[xianwu@rcn07 ~]$ podman version
Version:            1.6.4
RemoteAPI Version:  1
Go Version:         go1.13.4
OS/Arch:            linux/amd64```

**Output of `podman info --debug`:**

[xianwu@rcn07 ~]$ podman info --debug debug: compiler: gc git commit: "" go version: go1.13.4 podman version: 1.6.4 host: BuildahVersion: 1.12.0-dev CgroupVersion: v1 Conmon: package: conmon-2.0.6-1.module_el8.2.0+305+5e198a41.x86_64 path: /usr/bin/conmon version: 'conmon version 2.0.6, commit: a2b11288060ebd7abd20e0b4eb1a834bbf0aec3e' Distribution: distribution: '"centos"' version: "8" IDMappings: gidmap:

Package info (e.g. output of rpm -q podman or apt list podman):

(paste your output here)

Additional environment details (AWS, VirtualBox, physical, etc.): centos 8.2 virtual machine on vmware.

rhatdan commented 4 years ago

Could you check if this still happens on podman 1.9.3 version which should be on RHEL8.2.1 now.

coldbloodx commented 4 years ago

I'll try it out later.

giuseppe commented 4 years ago

that is the equivalent of leaking a mount point. When that happens, can you do: podman unshare cat /proc/self/mountinfo ?

coldbloodx commented 4 years ago

just hit this this issue, see my below test.

no podman process here: [xianwu@rcn07 ~]$ ps -ewf |grep podman xianwu 689115 687539 0 08:52 pts/0 00:00:00 grep --color=auto podman

[xianwu@rcn07 ~]$ podman run -it localhost/centos hostname d8be9f57555e [xianwu@rcn07 ~]$ ps -ewf |grep podman xianwu 689293 1 0 08:54 ? 00:00:00 podman xianwu 689312 1 0 08:54 ? 00:00:00 /usr/bin/fuse-overlayfs -o lowerdir=/podmanfs/overlay/l/D6ZILYLI2BLBDF65YSKZRRBKOB:/podmanfs/overlay/l/UNZMQEUSSGU7ZCGQZ6ICUEFNHX,upperdir=/podmanfs/overlay/ef6d5c3b189742dc20ffe523f4152866d661273e5e8dedf70f4bdef5208ec8df/diff,workdir=/podmanfs/overlay/ef6d5c3b189742dc20ffe523f4152866d661273e5e8dedf70f4bdef5208ec8df/work /podmanfs/overlay/ef6d5c3b189742dc20ffe523f4152866d661273e5e8dedf70f4bdef5208ec8df/merged. --> this one remains!!! xianwu 689356 687539 0 08:54 pts/0 00:00:00 grep --color=auto podman

[xianwu@rcn07 ~]$ ps -ewf |grep podman xianwu 689293 1 0 08:54 ? 00:00:00 podman xianwu 689312 1 0 08:54 ? 00:00:00 /usr/bin/fuse-overlayfs -o lowerdir=/podmanfs/overlay/l/D6ZILYLI2BLBDF65YSKZRRBKOB:/podmanfs/overlay/l/UNZMQEUSSGU7ZCGQZ6ICUEFNHX,upperdir=/podmanfs/overlay/ef6d5c3b189742dc20ffe523f4152866d661273e5e8dedf70f4bdef5208ec8df/diff,workdir=/podmanfs/overlay/ef6d5c3b189742dc20ffe523f4152866d661273e5e8dedf70f4bdef5208ec8df/work /podmanfs/overlay/ef6d5c3b189742dc20ffe523f4152866d661273e5e8dedf70f4bdef5208ec8df/merged xianwu 689423 687539 0 08:55 pts/0 00:00:00 grep --color=auto podman

Here is output of: podman unshare cat /proc/self/mountinfo [xianwu@rcn07 ~]$ podman unshare cat /proc/self/mountinfo 178 177 8:2 / / rw,relatime master:1 - ext4 /dev/sda2 rw 179 178 0:20 / /sys rw,nosuid,nodev,noexec,relatime master:2 - sysfs sysfs rw 180 179 0:7 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime master:3 - securityfs securityfs rw 181 179 0:24 / /sys/fs/cgroup ro,nosuid,nodev,noexec master:4 - tmpfs tmpfs ro,mode=755 182 181 0:25 / /sys/fs/cgroup/systemd rw,nosuid,nodev,noexec,relatime master:5 - cgroup cgroup rw,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 183 181 0:28 / /sys/fs/cgroup/net_cls,net_prio rw,nosuid,nodev,noexec,relatime master:6 - cgroup cgroup rw,net_cls,net_prio 184 181 0:29 / /sys/fs/cgroup/cpu,cpuacct rw,nosuid,nodev,noexec,relatime master:7 - cgroup cgroup rw,cpu,cpuacct 185 181 0:30 / /sys/fs/cgroup/freezer rw,nosuid,nodev,noexec,relatime master:8 - cgroup cgroup rw,freezer 186 181 0:31 / /sys/fs/cgroup/pids rw,nosuid,nodev,noexec,relatime master:9 - cgroup cgroup rw,pids 187 181 0:32 / /sys/fs/cgroup/hugetlb rw,nosuid,nodev,noexec,relatime master:10 - cgroup cgroup rw,hugetlb 188 181 0:33 / /sys/fs/cgroup/devices rw,nosuid,nodev,noexec,relatime master:11 - cgroup cgroup rw,devices 189 181 0:34 / /sys/fs/cgroup/perf_event rw,nosuid,nodev,noexec,relatime master:12 - cgroup cgroup rw,perf_event 190 181 0:35 / /sys/fs/cgroup/memory rw,nosuid,nodev,noexec,relatime master:13 - cgroup cgroup rw,memory 191 181 0:36 / /sys/fs/cgroup/cpuset rw,nosuid,nodev,noexec,relatime master:14 - cgroup cgroup rw,cpuset 192 181 0:37 / /sys/fs/cgroup/blkio rw,nosuid,nodev,noexec,relatime master:15 - cgroup cgroup rw,blkio 193 181 0:38 / /sys/fs/cgroup/rdma rw,nosuid,nodev,noexec,relatime master:16 - cgroup cgroup rw,rdma 194 179 0:26 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime master:17 - pstore pstore rw 195 179 0:27 / /sys/fs/bpf rw,nosuid,nodev,noexec,relatime master:18 - bpf bpf rw,mode=700 196 179 0:39 / /sys/kernel/config rw,relatime master:19 - configfs configfs rw 197 179 0:8 / /sys/kernel/debug rw,relatime master:27 - debugfs debugfs rw 198 179 0:54 / /sys/fs/fuse/connections rw,relatime master:96 - fusectl fusectl rw 199 178 0:6 / /dev rw,nosuid master:20 - devtmpfs devtmpfs rw,size=3038356k,nr_inodes=759589,mode=755 200 199 0:21 / /dev/shm rw,nosuid,nodev master:21 - tmpfs tmpfs rw 201 199 0:22 / /dev/pts rw,nosuid,noexec,relatime master:22 - devpts devpts rw,gid=5,mode=620,ptmxmode=000 202 199 0:40 / /dev/hugepages rw,relatime master:26 - hugetlbfs hugetlbfs rw,pagesize=2M 203 199 0:18 / /dev/mqueue rw,relatime master:28 - mqueue mqueue rw 204 178 0:23 / /run rw,nosuid,nodev master:23 - tmpfs tmpfs rw,mode=755 205 204 0:47 / /run/user/33857 rw,nosuid,nodev,relatime master:155 - tmpfs tmpfs rw,size=609768k,mode=700,uid=33857,gid=10007 206 204 0:49 / /run/user/0 rw,nosuid,nodev,relatime master:165 - tmpfs tmpfs rw,size=609768k,mode=700 207 178 0:4 / /proc rw,nosuid,nodev,noexec,relatime master:24 - proc proc rw 208 207 0:19 / /proc/sys/fs/binfmt_misc rw,relatime master:25 - autofs systemd-1 rw,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13750 209 178 0:41 / /var/lib/nfs/rpc_pipefs rw,relatime master:57 - rpc_pipefs sunrpc rw 210 178 0:44 / /home rw,relatime master:142 - autofs /etc/auto.homexian rw,fd=5,pgrp=651,timeout=300,minproto=5,maxproto=5,indirect,pipe_ino=17258 211 210 0:48 / /home/xianwu rw,relatime master:160 - nfs 9.111.248.224:/data/export/home/xianwu rw,vers=3,rsize=8192,wsize=8192,namlen=255,hard,noacl,proto=tcp,timeo=6000,retrans=2,sec=sys,mountaddr=9.111.248.224,mountvers=3,mountport=63759,mountproto=tcp,local_lock=none,addr=9.111.248.224 212 178 0:45 / /pcc rw,relatime master:147 - autofs /etc/auto.pccxian rw,fd=11,pgrp=651,timeout=300,minproto=5,maxproto=5,indirect,pipe_ino=17267 213 212 0:43 / /pcc/dev rw,relatime master:93 - nfs 9.111.248.224:/data/export/pcc_dev rw,vers=3,rsize=8192,wsize=8192,namlen=255,hard,noacl,proto=tcp,timeo=6000,retrans=2,sec=sys,mountaddr=9.111.248.224,mountvers=3,mountport=63759,mountproto=tcp,local_lock=none,addr=9.111.248.224 214 178 0:46 / /scratch rw,relatime master:151 - autofs /etc/auto.scratchxian rw,fd=17,pgrp=651,timeout=300,minproto=5,maxproto=5,indirect,pipe_ino=17271 215 178 0:52 / /lsf rw,relatime master:99 - nfs 9.111.251.179:/lsf rw,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=9.111.251.179,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=9.111.251.179 216 178 8:2 /podmanfs/overlay /podmanfs/overlay rw,relatime - ext4 /dev/sda2 rw 217 178 0:50 / /podmanfs/overlay-containers/d8be9f57555e519adaf46de02472bd45e1545c6647f91f5fb4471ccf3d908894/userdata/shm rw,nosuid,nodev,noexec,relatime - tmpfs shm rw,size=64000k,uid=33857,gid=10007 220 205 0:47 /netns /run/user/33857/netns rw,nosuid,nodev,relatime shared:102 master:155 - tmpfs tmpfs rw,size=609768k,mode=700,uid=33857,gid=10007 229 220 0:3 net:[4026532606] /run/user/33857/netns/cni-08c1003b-1edb-e284-8551-a39abf0172c4 rw shared:103 - nsfs nsfs rw 230 216 0:51 / /podmanfs/overlay/ef6d5c3b189742dc20ffe523f4152866d661273e5e8dedf70f4bdef5208ec8df/merged rw,nodev,noatime - fuse.fuse-overlayfs fuse-overlayfs rw,user_id=0,group_id=0,default_permissions,allow_other

wwilson commented 4 years ago

I have run into this issue pretty often, and it seems to exist on a wide selection of podman versions.

wwilson commented 4 years ago

You can see if you’re affected by running these commands:

ps aux | grep fuse-overlayfs | wc -l podman ps -q | grep wc -l

The defunct fuse-overlayfs processes are owned by init and are sleeping (not zombies).

rhatdan commented 4 years ago

@giuseppe Thoughts on what is causing this? Do we need something in containers storage to remove these on shutdown?

giuseppe commented 4 years ago

@giuseppe Thoughts on what is causing this? Do we need something in containers storage to remove these on shutdown?

I think the cleanup process fails. I see sometimes leaked mounts from root as well. The difference with fuse-overlayfs is that when we leak the mount we also leak the process that is managing that mount

rhatdan commented 4 years ago

I see 4 fuse-overlayfs running in my homedir right now.

ps -eZ | grep fuse-
unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 280864 ? 00:00:00 fuse-overlayfs
unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 280932 ? 00:00:00 fuse-overlayfs
unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 282384 ? 00:00:00 fuse-overlayfs
ps -eZ | grep podman
unconfined_u:system_r:container_runtime_t:s0-s0:c0.c1023 110035 ? 00:00:00 podman pause
rhatdan commented 4 years ago 
$ mount | grep fuse
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
gvfsd-fuse on /run/user/3267/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,relatime,user_id=3267,group_id=3267)
$ ps aux | grep fuse-overlayfs 
dwalsh    280864  0.0  0.0   7056  4692 ?        Ss   Jul28   0:00 /usr/bin/fuse-overlayfs -o lowerdir=/home/dwalsh/.local/share/containers/storage/overlay/l/D6MYXF5LJWYT63GXZH6ZQTKCIQ,upperdir=/home/dwalsh/.local/share/containers/storage/overlay/9c3847385c56fda91796ec164a50066d744706ae0a8dbd6e22ef7745b2edf67d/diff,workdir=/home/dwalsh/.local/share/containers/storage/overlay/9c3847385c56fda91796ec164a50066d744706ae0a8dbd6e22ef7745b2edf67d/work,context="system_u:object_r:container_file_t:s0:c15,c290" /home/dwalsh/.local/share/containers/storage/overlay/9c3847385c56fda91796ec164a50066d744706ae0a8dbd6e22ef7745b2edf67d/merged
dwalsh    280932  0.0  0.0   8076  5480 ?        Ss   Jul28   0:00 /usr/bin/fuse-overlayfs -o lowerdir=/home/dwalsh/.local/share/containers/storage/overlay/5a0b3dffd3f565da9d8d0a9a16149614220fd142e88804ba019ffb8eb641ad45/diff:/home/dwalsh/.local/share/containers/storage/overlay/5a0b3dffd3f565da9d8d0a9a16149614220fd142e88804ba019ffb8eb641ad45/empty,context="system_u:object_r:container_file_t:s0:c15,c290" /home/dwalsh/.local/share/containers/storage/overlay/5a0b3dffd3f565da9d8d0a9a16149614220fd142e88804ba019ffb8eb641ad45/merged
dwalsh    282384  0.0  0.0   5124  1732 ?        Ss   Jul28   0:00 /usr/bin/fuse-overlayfs -o lowerdir=/home/dwalsh/.local/share/containers/storage/overlay/l/D6MYXF5LJWYT63GXZH6ZQTKCIQ,upperdir=/home/dwalsh/.local/share/containers/storage/overlay/fdb93042a85659ada459b3169f767bc03741d3e63a5a8e5409ff7f8ad8e907bc/diff,workdir=/home/dwalsh/.local/share/containers/storage/overlay/fdb93042a85659ada459b3169f767bc03741d3e63a5a8e5409ff7f8ad8e907bc/work,context="system_u:object_r:container_file_t:s0:c119,c292" /home/dwalsh/.local/share/containers/storage/overlay/fdb93042a85659ada459b3169f767bc03741d3e63a5a8e5409ff7f8ad8e907bc/merged
dwalsh    562517  0.0  0.0 216220  2272 pts/2    S+   17:51   0:00 grep --color=auto fuse-overlayfs

So these are fuse-overlay that are not mounting anything?

wwilson commented 4 years ago

Could this possible dirty shutdown be the same root cause that sometimes corrupts ~/.local/share/containers/storage/overlay-containers/containers.json ?

giuseppe commented 4 years ago

So these are fuse-overlay that are not mounting anything?

so that seems like a fuse-overlayfs issue. Can you please strace these processes? In what syscall are they blocked?

giuseppe commented 4 years ago

$ mount | grep fuse fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime) gvfsd-fuse on /run/user/3267/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,relatime,user_id=3267,group_id=3267) $ ps aux | grep fuse-overlayfs dwalsh 280864 0.0 0.0 7056 4692 ? Ss Jul28 0:00 /usr/bin/fuse-overlayfs -o lowerdir=/home/dwalsh/.local/share/containers/storage/overlay/l/D6MYXF5LJWYT63GXZH6ZQTKCIQ,upperdir=/home/dwalsh/.local/share/containers/storage/overlay/9c3847385c56fda91796ec164a50066d744706ae0a8dbd6e22ef7745b2edf67d/diff,workdir=/home/dwalsh/.local/share/containers/storage/overlay/9c3847385c56fda91796ec164a50066d744706ae0a8dbd6e22ef7745b2edf67d/work,context="system_u:object_r:container_file_t:s0:c15,c290" /home/dwalsh/.local/share/containers/storage/overlay/9c3847385c56fda91796ec164a50066d744706ae0a8dbd6e22ef7745b2edf67d/merged dwalsh 280932 0.0 0.0 8076 5480 ? Ss Jul28 0:00 /usr/bin/fuse-overlayfs -o lowerdir=/home/dwalsh/.local/share/containers/storage/overlay/5a0b3dffd3f565da9d8d0a9a16149614220fd142e88804ba019ffb8eb641ad45/diff:/home/dwalsh/.local/share/containers/storage/overlay/5a0b3dffd3f565da9d8d0a9a16149614220fd142e88804ba019ffb8eb641ad45/empty,context="system_u:object_r:container_file_t:s0:c15,c290" /home/dwalsh/.local/share/containers/storage/overlay/5a0b3dffd3f565da9d8d0a9a16149614220fd142e88804ba019ffb8eb641ad45/merged dwalsh 282384 0.0 0.0 5124 1732 ? Ss Jul28 0:00 /usr/bin/fuse-overlayfs -o lowerdir=/home/dwalsh/.local/share/containers/storage/overlay/l/D6MYXF5LJWYT63GXZH6ZQTKCIQ,upperdir=/home/dwalsh/.local/share/containers/storage/overlay/fdb93042a85659ada459b3169f767bc03741d3e63a5a8e5409ff7f8ad8e907bc/diff,workdir=/home/dwalsh/.local/share/containers/storage/overlay/fdb93042a85659ada459b3169f767bc03741d3e63a5a8e5409ff7f8ad8e907bc/work,context="system_u:object_r:container_file_t:s0:c119,c292" /home/dwalsh/.local/share/containers/storage/overlay/fdb93042a85659ada459b3169f767bc03741d3e63a5a8e5409ff7f8ad8e907bc/merged dwalsh 562517 0.0 0.0 216220 2272 pts/2 S+ 17:51 0:00 grep --color=auto fuse-overlayfs

can you try podman unshare mount | grep fuse

The mounts created by fuse-overlayfs are in the new user+mount namespace, not visible from the host

rhatdan commented 4 years ago

Yes the mount points were in the unshare mount namespace.

wwilson commented 4 years ago

I tried stracing a few of these like giuseppe suggested (not all of them because I umm... apparently have hundreds of these). They all appear to be hung on either read or splice on a file descriptor that does not appear to exist in my system.

giuseppe commented 4 years ago

They all appear to be hung on either read or splice on a file descriptor that does not appear to exist in my system.

so it is waiting for events from FUSE. I think that is expected. I still think the issue is that we fail the cleanup and don't even get to do the Unmount.

You can check if it is fuse-overlays to hang by doing:

podman unshare umount /path/to/the/merged/path/specified/in/the/fuse-overlayfs/cmdline`

Does fuse-overlayfs exit after that?

wwilson commented 4 years ago

It does indeed look like that causes the defunct fuse-overlays to exit.

So this is probably a podman shutdown bug?

mheon commented 4 years ago

If you are on Podman v2.0.3 or later, we should be printing full logs from the cleanup process (podman container cleanup) to syslog if the container is started with --log-level=debug. The failure should be in there.

wwilson commented 4 years ago

I am currently unable to change my podman version. @rhatdan it looked like you had also run into this? Can you try to get the logs that @mheon needs?

rhatdan commented 4 years ago

I have two running on my system right now.

dwalsh    869895  0.0  0.0   4520  1776 ?        Ss   Aug05   0:00 /usr/bin/fuse-overlayfs -o lowerdir=/home/dwalsh/.local/share/containers/storage/overlay/l/UONY5FHLNK2NTAIM3A5XG4R43O,upperdir=/home/dwalsh/.local/share/containers/storage/overlay/243777689c53a0240a0579ee04d026bb9d7e0b9e3e6a31aa93478fc728b46689/diff,workdir=/home/dwalsh/.local/share/containers/storage/overlay/243777689c53a0240a0579ee04d026bb9d7e0b9e3e6a31aa93478fc728b46689/work,context="system_u:object_r:container_file_t:s0:c615,c823" /home/dwalsh/.local/share/containers/storage/overlay/243777689c53a0240a0579ee04d026bb9d7e0b9e3e6a31aa93478fc728b46689/merged
dwalsh   1022603  0.0  0.0   4240  2592 ?        Ss   13:44   0:00 /usr/bin/fuse-overlayfs -o lowerdir=/home/dwalsh/.local/share/containers/storage/overlay/l/4455UXPC5G6NMXLCSLAS45VFSF:/home/dwalsh/.local/share/containers/storage/overlay/l/OS5U2G5S46Y4BSPLFCX2JNIZ5T:/home/dwalsh/.local/share/containers/storage/overlay/l/V4NL7AHHESQZHGDHTW6CC7KRZZ:/home/dwalsh/.local/share/containers/storage/overlay/l/4AUPK5LUJQDWR7OQP3K4KYHKO2:/home/dwalsh/.local/share/containers/storage/overlay/l/DPGJTLOFCHWWZL6RIRQBSPZC5N:/home/dwalsh/.local/share/containers/storage/overlay/l/3HO3BBVOF7QZSWEZ7ZAPR2Q2DM:/home/dwalsh/.local/share/containers/storage/overlay/l/XU5LJTGPV7H7TALNYVWOCLK5LS:/home/dwalsh/.local/share/containers/storage/overlay/l/ELEYLJDEOFQPELSQNVDTSLMM2E:/home/dwalsh/.local/share/containers/storage/overlay/l/SLVLTGJIP2T5TT3GVGS2WQRXTS:/home/dwalsh/.local/share/containers/storage/overlay/l/MTF5NX5ITW7EVOU2VNTJ4MZW55:/home/dwalsh/.local/share/containers/storage/overlay/l/TIVZ2PBB2GCNTXEREVDBVY34O3:/home/dwalsh/.local/share/containers/storage/overlay/l/EP2XCXA4MHLGEXTM56SES6LSC3:/home/dwalsh/.local/share/containers/storage/overlay/l/VHT7FEVQODCAVUEYAIFJIIYFJ4:/home/dwalsh/.local/share/containers/storage/overlay/l/TYDNRNZUFA6PM37LEMI5IPDLSF:/home/dwalsh/.local/share/containers/storage/overlay/l/ZSKIIYOH63XKFUXDJ7KZ7YT77D,upperdir=/home/dwalsh/.local/share/containers/storage/overlay/054ae403adbe18c034e809a0a6fdcff121a0e289f4d9064c80dd72be7fbfbb9a/diff,workdir=/home/dwalsh/.local/share/containers/storage/overlay/054ae403adbe18c034e809a0a6fdcff121a0e289f4d9064c80dd72be7fbfbb9a/work,context="system_u:object_r:container_file_t:s0:c388,c958" /home/dwalsh/.local/share/containers/storage/overlay/054ae403adbe18c034e809a0a6fdcff121a0e289f4d9064c80dd72be7fbfbb9a/merged
$ podman ps
CONTAINER ID  IMAGE   COMMAND  CREATED  STATUS  PORTS   NAMES

But I don't see any mounts in /proc/mounts even if I do a podman unshare. I don't know how to get these to happen.

giuseppe commented 4 years ago

have you used pods? I think https://github.com/containers/podman/pull/7283 could help to clean up leaked mounts

mheon commented 4 years ago

Has anyone seem this since the release of Podman 2.0.5? I suspect it should be fixed with the pod cleanup code having merged

rhatdan commented 4 years ago

Closing, reopen if you have seen it.

wwilson commented 4 years ago

I've upgraded to Podman 2.0.6, and this problem looks like it is still very much a problem.

wwilson commented 4 years ago

What's the right way to reopen this? I'll try to collect the debugging info @mheon was looking for.

wwilson commented 4 years ago

Pretty weird -- I ran with --log-level=debug as suggested, and there is nothing from the shutdown process getting printed at all. Last things I see are:

DEBU[0000] Started container 6823e20fe3f4f7bd212d37f7bb8b5d7a1c9ade4b50761ac506060e7e8c9f179e 
DEBU[0000] Enabling signal proxying                     
DEBU[0000] Called run.PersistentPostRunE([a bunch of stuff])
TomSweeneyRedHat commented 4 years ago

@wwilson thanks for the follow-up. To reopen, you need to click on the "Reopen and comment" button. However, I think that only shows up for members of the "Containers" organization and the reporter of the issue. I'll go ahead and reopen the issue for you assuming that the button is not being displayed.

wwilson commented 4 years ago

Update: the fuse-overlayfs process is killed if the halted container is destroyed. Dunno if that helps narrow down on where the issue could be.

github-actions[bot] commented 4 years ago

A friendly reminder that this issue had no activity for 30 days.

rhatdan commented 4 years ago

I don't believe we have this issue in latest podman in github or fuse-overlayfs in github.