runsc fails mount w/ "permission denied" using rootful Podman >= 5.2.0 (git bisected)

[BinaryKhaos]

Issue Description

I already reported this issue over @ runsc's issue tracker. but since it is somewhat unclear to me who is actually in the best position to at least fix/workaround this in the short term, I am also reporting it here. The following is copy & pasted from my runsc issue, so hopefully that's is okay.

With Podman commit c81f075f436466092372dec7a19c35fe387fe8d3 ("libpod: do not chmod bind mounts"), which is included in release 5.2.0-rc1 and above, runsc fails to bind mount volumes in certain cases with permission denied errors.

In my case, I have a custom container with an unpriviledged user that has several (partly nested) VOLUMEs defined in its BUILDFILE. I have the (local) volumes created with the appropriate sub(u|g)ids and run the container w/ userns=auto and mount the volumes accordingly. Everything in the container is run as the unpriviledged user.

This worked fine w/ runsc and Podman up to release 5.1.2. It fails w/ the 5.2 branch. It does work absolutely fine, though, with either runc or crunc, no matter what Podman version.

Steps to reproduce the issue

This is the most compact reproducer I could come up with.

Everything as root:

1. Add "containers:100000:131072" to /etc/subuid and /etc/subgid
2. podman volume create --opt o=uid=100001,gid=100001 bugtest-volume
3. podman run --userns=auto:size=65536 -v bugtest-volume:/home/bugtest --runtime=runsc --rm -it alpine sh -c "ls -ln /home"

Describe the results you received

This will cause a permission denied error on runsc's side with Podman >= 5.2.0-rc1.

Describe the results you expected

With crun/runc, you will see the correct directory listing:

total 4
drwxr-xr-x    2 1        1             4096 Oct 15 05:53 bugtest

podman info output

host: arch: amd64 buildahVersion: 1.37.4 cgroupControllers:

• cpuset
• cpu
• io
• memory
• hugetlb
• pids
• rdma
• misc cgroupManager: systemd cgroupVersion: v2 conmon: package: app-containers/conmon-2.1.11 path: /usr/libexec/podman/conmon version: 'conmon version 2.1.11, commit: unknown' cpuUtilization: idlePercent: 92.64 systemPercent: 2.81 userPercent: 4.55 cpus: 8 databaseBackend: sqlite distribution: distribution: gentoo version: "2.15" eventLogger: journald freeLocks: 2046 hostname: TARDIS idMappings: gidmap: null uidmap: null kernel: 6.11.3-gentoo-241010-r1 linkmode: dynamic logDriver: journald memFree: 26554200064 memTotal: 33574236160 networkBackend: netavark networkBackendInfo: backend: netavark dns: package: app-containers/aardvark-dns-1.12.2 path: /usr/libexec/podman/aardvark-dns version: aardvark-dns 1.12.2 package: app-containers/netavark-1.12.2 path: /usr/libexec/podman/netavark version: netavark 1.12.2 ociRuntime: name: runsc package: Unknown path: /usr/local/bin/runsc version: |- runsc version release-20241007.0-32-ga81ec225dce9 spec: 1.1.0-rc.1 os: linux pasta: executable: /usr/bin/pasta package: net-misc/passt-2024.09.06 version: | pasta 2024.09.06 Copyright Red Hat GNU General Public License, version 2 or later https://www.gnu.org/licenses/old-licenses/gpl-2.0.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. remoteSocket: exists: false path: /run/podman/podman.sock rootlessNetworkCmd: pasta security: apparmorEnabled: false capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT rootless: false seccompEnabled: true seccompProfilePath: /usr/share/containers/seccomp.json selinuxEnabled: false serviceIsRemote: false slirp4netns: executable: /usr/bin/slirp4netns package: app-containers/slirp4netns-1.2.0 version: |- slirp4netns version 1.2.0 commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383 libslirp: 4.7.0 SLIRP_CONFIG_VERSION_MAX: 4 libseccomp: 2.5.5 swapFree: 0 swapTotal: 0 uptime: 0h 19m 44.00s variant: "" plugins: authorization: null log:
• k8s-file
• none
• passthrough
• journald network:
• bridge
• macvlan
• ipvlan volume:
• local registries: {} store: configFile: /etc/containers/storage.conf containerStore: number: 0 paused: 0 running: 0 stopped: 0 graphDriverName: overlay graphOptions: overlay.mountopt: nodev graphRoot: /var/lib/containers/storage graphRootAllocated: 1978033311744 graphRootUsed: 1057981378560 graphStatus: Backing Filesystem: extfs Native Overlay Diff: "true" Supports d_type: "true" Supports shifting: "true" Supports volatile: "true" Using metacopy: "false" imageCopyTmpDir: /var/tmp imageStore: number: 14 runRoot: /run/containers/storage transientStore: false volumePath: /var/lib/containers/storage/volumes version: APIVersion: 5.2.4 Built: 1728971328 BuiltTime: Tue Oct 15 07:48:48 2024 GitCommit: "" GoVersion: go1.23.1 Os: linux OsArch: linux/amd64 Version: 5.2.4

Podman in a container

No

Privileged Or Rootless

Privileged

Upstream Latest Release

Yes

[eriksjolund]

I tried your reproducer on Fedora CoreOS 41.20241006.1.1 (but not for scrun I just tried crun)

When running step 3

# podman --runtime crun run --userns=auto:size=65536  -v bugtest-volume:/home/bugtest --rm -it alpine sh -c "ls -ln /home"
total 0
drwxr-xr-x    2 65534    65534            6 Oct 17 07:31 bugtest

I also tried the idmap option

# podman --runtime crun run --userns=auto:size=65536 --mount 'type=volume,src=bugtest-volume,dst=/home/bugtest,idmap=uids=@100001-1-1;gids=@100001-1-1' --rm -it alpine sh -c "ls -ln /home"
total 0
drwxr-xr-x    2 1        1                6 Oct 17 07:31 bugtest

[BinaryKhaos]

If you followed the steps to the letter, you should see the described outcome. I just verified it again, to be sure. I was not able to find a packages list for Fedora CoreOS, so I have no clue what crun and podman version you are using on what kernel and so forth. But with podman >= 5.2.0-rc1 (and git main), you will see the failure with runsc (not scrun). Both runc (without the s) and crun will work fine, showing the correct uid/gid. Also, isn't Fedora using SELinux by default? I don't know how much this could influence the result as well.

Since I totally forgot I could use idmap with rootful podman (previously only used rootless), I will see if I can workaround this problem that way and report back.

[BinaryKhaos]

Ok, tested with idmapped mounts and even though they work perfectly fine with crun and runc to achieve the same results I want, it still fails, as expected, for runsc with a permission denied error due to the changes in https://github.com/containers/podman/commit/c81f075f436466092372dec7a19c35fe387fe8d3. Since runsc does not use the new mount api, that change effectively limits what can be done with podman and runsc. I have no chance to test this with Kata Containers at the moment, but I bet those are effected as well.

[baude]

@giuseppe can you read this and offer an opinion ?

[giuseppe]

that is runsc not using the new mount API.

https://github.com/containers/podman/commit/c81f075f436466092372dec7a19c35fe387fe8d3 changes the requirement, so we don't loosen up the directory permissions, but expect the OCI runtime to handle that using the new mount API. I'd prefer that we don't revert the change because it improves security for crun/runc users.

One way to circumvent the issue is to change the directory permissions so they are usable from the user namespace, alternatively, you could create a bind mount and use that for the volume source.

[BinaryKhaos]

No matter what I tried, I couldn't make this work-- neither with changing the permissions on the volume directory nor with some bind mount trickery. Even when I put a directory in the fs root and try to use that as a bind mount with podman, I still get a permission denied error with runsc. Which is actually what I expected since, as far as I understand it, the change made in https://github.com/containers/podman/commit/c81f075f436466092372dec7a19c35fe387fe8d3 affects the mount point under /run and not the original directory. And since that is created dynamically, I cannot really influence it at all.

Is there something I am totally missing here and just being stupid? :-(

[Luap99]

Given this is a regression for other less common runtimes should we add the old code back behind a containers.conf option? Then any user of a runtime that has no support for this would need to set this in containers.conf and we could still default to the more secure version.

[giuseppe]

No matter what I tried, I couldn't make this work-- neither with changing the permissions on the volume directory nor with some bind mount trickery. Even when I put a directory in the fs root and try to use that as a bind mount with podman, I still get a permission denied error with runsc.

why does that happen? The directory is not owned by the user in the container? In this case, could you try adding the :U option to the volume?

Given this is a regression for other less common runtimes should we add the old code back behind a containers.conf option? Then any user of a runtime that has no support for this would need to set this in containers.conf and we could still default to the more secure version.

I guess we can do that if we don't manage to find a reasonable workaround

[BinaryKhaos]

Ok, the problem is even more serious than I initially noticed since I always tried just my test case but runsc no longer works with podman >= 5.2.0-rc1-- period.

# podman run --runtime=runsc --rm -it alpine sh
Error: OCI runtime error: runsc: creating container: cannot create sandbox: cannot read client sync file: waiting for sandbox to start: EOF

This is from the log of the gofer process which fails:

W1023 08:17:45.985373       1 specutils.go:127] noNewPrivileges ignored. PR_SET_NO_NEW_PRIVS is assumed to always be set.
I1023 08:17:45.985882       1 gofer.go:513] Mounting src: "/var/lib/containers/storage/overlay-containers/3bd085dc41322703a52212a3090ee44114a836544c76a85c4138084bd834d6c9/userdata/shm", dst: "/proc/fs/root/dev/shm", flags: 0x100e
W1023 08:17:45.985982       1 util.go:64] FATAL ERROR: error setting up FS: mounting {Destination:/dev/shm Type:bind Source:/var/lib/containers/storage/overlay-containers/3bd085dc41322703a52212a3090ee44114a836544c76a85c4138084bd834d6c9/userdata/shm Options:[bind rprivate nosuid noexec nodev] UIDMappings:[] GIDMappings:[]}: stat("/var/lib/containers/storage/overlay-containers/3bd085dc41322703a52212a3090ee44114a836544c76a85c4138084bd834d6c9/userdata/shm") failed: stat /var/lib/containers/storage/overlay-containers/3bd085dc41322703a52212a3090ee44114a836544c76a85c4138084bd834d6c9/userdata/shm: permission denied
error setting up FS: mounting {Destination:/dev/shm Type:bind Source:/var/lib/containers/storage/overlay-containers/3bd085dc41322703a52212a3090ee44114a836544c76a85c4138084bd834d6c9/userdata/shm Options:[bind rprivate nosuid noexec nodev] UIDMappings:[] GIDMappings:[]}: stat("/var/lib/containers/storage/overlay-containers/3bd085dc41322703a52212a3090ee44114a836544c76a85c4138084bd834d6c9/userdata/shm") failed: stat /var/lib/containers/storage/overlay-containers/3bd085dc41322703a52212a3090ee44114a836544c76a85c4138084bd834d6c9/userdata/shm: permission denied

The error varies from try to try. Here another one:

W1023 08:21:34.475990       1 specutils.go:127] noNewPrivileges ignored. PR_SET_NO_NEW_PRIVS is assumed to always be set.
I1023 08:21:34.476296       1 gofer.go:513] Mounting src: "/run/containers/storage/overlay-containers/69a22ee4afda75ba1121466e934fa30f6b2b00fb2b75ed83e32a105704c3b225/userdata/hosts", dst: "/proc/fs/root/etc/hosts", flags: 0x1000
W1023 08:21:34.476335       1 util.go:64] FATAL ERROR: error setting up FS: mounting {Destination:/etc/hosts Type:bind Source:/run/containers/storage/overlay-containers/69a22ee4afda75ba1121466e934fa30f6b2b00fb2b75ed83e32a105704c3b225/userdata/hosts Options:[bind rprivate] UIDMappings:[] GIDMappings:[]}: stat("/run/containers/storage/overlay-containers/69a22ee4afda75ba1121466e934fa30f6b2b00fb2b75ed83e32a105704c3b225/userdata/hosts") failed: stat /run/containers/storage/overlay-containers/69a22ee4afda75ba1121466e934fa30f6b2b00fb2b75ed83e32a105704c3b225/userdata/hosts: permission denied
error setting up FS: mounting {Destination:/etc/hosts Type:bind Source:/run/containers/storage/overlay-containers/69a22ee4afda75ba1121466e934fa30f6b2b00fb2b75ed83e32a105704c3b225/userdata/hosts Options:[bind rprivate] UIDMappings:[] GIDMappings:[]}: stat("/run/containers/storage/overlay-containers/69a22ee4afda75ba1121466e934fa30f6b2b00fb2b75ed83e32a105704c3b225/userdata/hosts") failed: stat /run/containers/storage/overlay-containers/69a22ee4afda75ba1121466e934fa30f6b2b00fb2b75ed83e32a105704c3b225/userdata/hosts: permission denied

Suffice to say, crun/runc naturally work perfectly fine.

Just in case it gets asked, here the permissions for the directories:

drwxr-xr-x 5 root root 4096 Oct 23 08:15 /var/lib/containers/
drwxr-xr-x 2 root root 4096 Oct 12  2023 /var/lib/containers/sigstore
drwx------ 2 root root 4096 Oct 23 08:15 /var/lib/containers/cache
drwx------ 8 root root 4096 Oct 23 08:21 /var/lib/containers/storage
drwx------ 2 root root 4096 Oct 23 08:21 /var/lib/containers/storage/overlay-containers
drwx------ 3 root root 4096 Oct 23 08:15 /var/lib/containers/storage/overlay-images
drwx------ 2 root root 4096 Oct 23 08:15 /var/lib/containers/storage/overlay-images/91ef0af61f39ece4d6710e465df5ed6ca12112358344fd51ae6a3b886634148b
drwxr-xr-x 4 root root 4096 Oct 23 08:21 /var/lib/containers/storage/overlay
drwx------ 6 root root 4096 Oct 23 08:15 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85
dr-xr-xr-x 19 root root 4096 Oct 23 08:15 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff
drwxr-xr-x 7 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/usr
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/usr/bin
drwxr-xr-x 6 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/usr/share
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/usr/share/misc
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/usr/share/udhcpc
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/usr/share/man
drwxr-xr-x 3 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/usr/share/apk
drwxr-xr-x 11 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/usr/share/apk/keys
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/usr/share/apk/keys/riscv64
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/usr/share/apk/keys/aarch64
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/usr/share/apk/keys/armv7
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/usr/share/apk/keys/armhf
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/usr/share/apk/keys/x86
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/usr/share/apk/keys/ppc64le
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/usr/share/apk/keys/mips64
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/usr/share/apk/keys/s390x
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/usr/share/apk/keys/x86_64
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/usr/sbin
drwxr-xr-x 5 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/usr/lib
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/usr/lib/modules-load.d
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/usr/lib/engines-3
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/usr/lib/ossl-modules
drwxr-xr-x 5 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/usr/local
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/usr/local/bin
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/usr/local/share
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/usr/local/lib
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/dev
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/srv
drwxr-xr-x 12 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/var
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/var/mail
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/var/opt
drwxr-xr-x 4 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/var/cache
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/var/cache/misc
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/var/cache/apk
drwxr-xr-x 3 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/var/spool
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/var/spool/cron
drwxrwxrwt 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/var/tmp
drwxr-xr-x 3 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/var/lock
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/var/lock/subsys
drwxr-xr-x 3 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/var/lib
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/var/lib/misc
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/var/local
dr-xr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/var/empty
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/var/log
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/opt
dr-xr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/proc
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/bin
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/mnt
drwxrwxrwt 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/tmp
drwxr-xr-x 17 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/etc
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/etc/profile.d
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/etc/opt
drwxr-xr-x 8 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/etc/network
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/etc/network/if-pre-down.d
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/etc/network/if-up.d
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/etc/network/if-post-up.d
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/etc/network/if-post-down.d
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/etc/network/if-down.d
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/etc/network/if-pre-up.d
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/etc/modules-load.d
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/etc/ssl1.1
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/etc/udhcpc
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/etc/crontabs
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/etc/logrotate.d
drwxr-xr-x 2 root root 4096 Sep  6 13:33 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/etc/secfixes.d
drwxr-xr-x 4 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/etc/apk
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/etc/apk/protected_paths.d
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/etc/apk/keys
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/etc/modprobe.d
drwxr-xr-x 4 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/etc/ssl
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/etc/ssl/private
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/etc/ssl/certs
drwxr-xr-x 7 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/etc/periodic
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/etc/periodic/daily
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/etc/periodic/weekly
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/etc/periodic/15min
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/etc/periodic/monthly
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/etc/periodic/hourly
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/etc/sysctl.d
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/etc/busybox-paths.d
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/sbin
drwxr-xr-x 6 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/lib
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/lib/modules-load.d
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/lib/firmware
drwxr-xr-x 4 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/lib/apk
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/lib/apk/exec
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/lib/apk/db
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/lib/sysctl.d
drwx------ 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/root
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/sys
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/run
drwxr-xr-x 5 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/media
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/media/cdrom
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/media/usb
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/media/floppy
drwxr-xr-x 2 root root 4096 Sep  6 13:34 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/diff/home
drwx------ 2 root root 4096 Oct 23 08:15 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/work
drwx------ 2 root root 4096 Oct 23 08:15 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/empty
drwx------ 2 root root 4096 Oct 23 08:15 /var/lib/containers/storage/overlay/63ca1fbb43ae5034640e5e6cb3e083e05c290072c5366fcaa9d62435a4cced85/merged
drwxr-xr-x 2 root root 4096 Oct 23 08:21 /var/lib/containers/storage/overlay/l
drwx------ 2 root root 4096 Oct 23 08:15 /var/lib/containers/storage/volumes
drwx------ 2 root root 4096 Oct 23 08:21 /var/lib/containers/storage/overlay-layers
drwx------ 2 root root 4096 Oct 23 08:15 /var/lib/containers/storage/libpod

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.