Fails to initialize container if the image has non-empty /mnt and the host has a symlink as /mnt

[akihikodaki]

Describe the bug It fails to initialize container if the image has non-empty /mnt and the host has a symlink as /mnt.

Steps how to reproduce the behaviour

1. Run toolbox create -i image-with-non-empty-mnt
2. Run toolbox enter image-with-non-empty-mnt-latest
3. See error

Expected behaviour No error will happen.

Actual behaviour It fails to initialize the container.

Screenshots N/A

Output of toolbox --version (v0.0.90+) toolbox version 0.0.99.4

Toolbox package info (rpm -q toolbox) toolbox-0.0.99.4-1.fc38.x86_64

Output of podman version

Client:       Podman Engine
Version:      4.7.0
API Version:  4.7.0
Go Version:   go1.20.8
Built:        Thu Sep 28 03:24:38 2023
OS/Arch:      linux/amd64

Podman package info (rpm -q podman) podman-4.7.0-1.fc38.x86_64

Info about your OS Fedora Silverblue 38

Additional context /mnt is a symlink to /var/mnt on the affected system.

[juhp]

So what is the error?

[akihikodaki]

Here is a simple reproduction:

cat > Containerfile <<EOF
FROM registry.fedoraproject.org/fedora-toolbox:38
RUN touch /mnt/a
EOF
podman build -t a .
toolbox create -i a
toolbox enter a

The last command gives:

Error: failed to initialize container a

[juhp]

Please try podman start --attach <containername>

[akihikodaki]

Below is the output:

$ podman start --attach a
level=debug msg="Running as real user ID 0"
level=debug msg="Resolved absolute path to the executable as /usr/bin/toolbox"
level=debug msg="TOOLBOX_PATH is /usr/bin/toolbox"
level=debug msg="Migrating to newer Podman"
level=debug msg="Migration not needed: running inside a container"
level=debug msg="Setting up configuration"
level=debug msg="Setting up configuration: file /etc/containers/toolbox.conf not found"
level=debug msg="Setting up configuration: file /root/.config/containers/toolbox.conf not found"
level=debug msg="Resolving container and image names"
level=debug msg="Container: ''"
level=debug msg="Distribution (CLI): ''"
level=debug msg="Image (CLI): ''"
level=debug msg="Release (CLI): ''"
level=debug msg="Resolved container and image names"
level=debug msg="Container: 'fedora-toolbox-38'"
level=debug msg="Image: 'fedora-toolbox:38'"
level=debug msg="Release: '38'"
level=debug msg="Creating /run/.toolboxenv"
level=debug msg="Monitoring host"
level=debug msg="Path /run/host/etc exists"
level=debug msg="Resolved /etc/localtime to /run/host/usr/share/zoneinfo/Asia/Tokyo"
level=debug msg="Creating regular file /etc/machine-id"
level=debug msg="Binding /etc/machine-id to /run/host/etc/machine-id"
level=debug msg="Creating directory /run/systemd/journal"
level=debug msg="Binding /run/systemd/journal to /run/host/run/systemd/journal"
level=debug msg="Creating directory /run/systemd/resolve"
level=debug msg="Binding /run/systemd/resolve to /run/host/run/systemd/resolve"
level=debug msg="Creating directory /run/systemd/sessions"
level=debug msg="Binding /run/systemd/sessions to /run/host/run/systemd/sessions"
level=debug msg="Creating directory /run/systemd/system"
level=debug msg="Binding /run/systemd/system to /run/host/run/systemd/system"
level=debug msg="Creating directory /run/systemd/users"
level=debug msg="Binding /run/systemd/users to /run/host/run/systemd/users"
level=debug msg="Creating directory /run/udev/data"
level=debug msg="Binding /run/udev/data to /run/host/run/udev/data"
level=debug msg="Creating directory /run/udev/tags"
level=debug msg="Binding /run/udev/tags to /run/host/run/udev/tags"
level=debug msg="Creating directory /tmp"
level=debug msg="Binding /tmp to /run/host/tmp"
level=debug msg="Creating directory /var/lib/flatpak"
level=debug msg="Binding /var/lib/flatpak to /run/host/var/lib/flatpak"
level=debug msg="Creating directory /var/lib/libvirt"
level=debug msg="Binding /var/lib/libvirt to /run/host/var/lib/libvirt"
level=debug msg="Creating directory /var/lib/systemd/coredump"
level=debug msg="Binding /var/lib/systemd/coredump to /run/host/var/lib/systemd/coredump"
level=debug msg="Creating directory /var/log/journal"
level=debug msg="Binding /var/log/journal to /run/host/var/log/journal"
level=debug msg="Creating directory /var/mnt"
level=debug msg="Binding /var/mnt to /run/host/var/mnt"
level=debug msg="Creating directory /sys/fs/selinux"
level=debug msg="Binding /sys/fs/selinux to /usr/share/empty"
level=debug msg="Preparing to redirect /mnt to /var/mnt"
level=debug msg="/var/mnt isn't a symbolic link"
Error: failed to redirect /mnt to /var/mnt: remove /mnt: directory not empty

[debarshiray]

Thanks for the logs!

I haven't looked at this in detail. In the meantime, why is /var/mnt not empty on your host? Just curious.

[akihikodaki]

@debarshiray /var/mnt is empty on host. /mnt on the container was though. I don't know why /mnt was not empty on the container; I remember a distribution package had a file in /mnt, but I don't remember specifically, and do no longer have the image.

[debarshiray]

@debarshiray /var/mnt is empty on host. /mnt on the container was though. I don't know why /mnt was not empty on the container; I remember a distribution package had a file in /mnt, but I don't remember specifically, and do no longer have the image.

I see, okay!

Yes, I realized later on that it's the image's /mnt that wasn't empty, not the host's. Sorry about that.