toolbox + ubi images appear to have a uid issue

[BadgerOps]

Hello all,

I initially ran across this issue on a RHEL 8.8 hardened host, and attributed it to our hardening, but then also ran into the issue on my F39 / Silverblue workstation at home.

When following this red hat devloper blog post creating a go-toolset from a ubi build, I have a uid based error when toolbox enter - see the following:

Steps how to reproduce the behaviour

$ toolbox create --image registry.access.redhat.com/ubi9/go-toolset
Image required to create toolbox container.
Download registry.access.redhat.com/ubi9/go-toolset ( ... MB)? [y/N]: y
Created container: go-toolset
Enter with: toolbox enter go-toolset

$ toolbox enter go-toolset

Welcome to the Toolbox; a container where you can install and run
all your tools.

 - To create a new tools container, run 'toolbox create'.

mkdir: cannot create directory ‘/opt/app-root/src/.config’: Permission denied
touch: cannot touch '/opt/app-root/src/.config/toolbox/toolbox-welcome-shown': No such file or directory

still within the toolbox image, I check the id - this is my Fedora user, UID, GID, and correct groups:

$ id
uid=1000(badger) gid=1000(badger) groups=1000(badger),10(wheel)

Looking at the directory /opt/app-root/src I see it is owned by default

$ grep 'default' /etc/passwd
default:x:1001:0:Default Application User:/opt/app-root/src:/sbin/nologin

which lines up with line 40 in the Dockerfile

Expected behaviour

toolbox (or the ubi image? :thinking: ) should be able to create and use an image without a uid conflict

Output of toolbox --version (v0.0.90+)

$ toolbox --version
toolbox version 0.0.99.5

Toolbx package info (rpm -q toolbox)

$ rpm -q toolbox
toolbox-0.0.99.5-2.fc39.x86_64

Output of podman version

$ podman version
Client:       Podman Engine
Version:      4.8.3
API Version:  4.8.3
Go Version:   go1.21.5
Built:        Wed Jan  3 15:11:40 2024
OS/Arch:      linux/amd64

Podman package info (rpm -q podman) e.g., podman-1.9.2-1.fc32.x86_64

Info about your OS

Fedora Silverblue 39

Additional context I've seen this with my Fedora39 Silverblue install, as well as RHEL 8.9 server

[BadgerOps]

Podman logs

$ podman ps
CONTAINER ID  IMAGE                                              COMMAND               CREATED         STATUS         PORTS       NAMES
0423d58ee1b9  registry.access.redhat.com/ubi9/go-toolset:latest  toolbox --log-lev...  34 minutes ago  Up 33 minutes              go-toolset

$ podman logs 0423d58ee1b9
level=debug msg="Running as real user ID 0"
level=debug msg="Resolved absolute path to the executable as /usr/bin/toolbox"
level=debug msg="TOOLBOX_PATH is /usr/bin/toolbox"
level=debug msg="Migrating to newer Podman"
level=debug msg="Migration not needed: running inside a container"
level=debug msg="Setting up configuration"
level=debug msg="Setting up configuration: file /etc/containers/toolbox.conf not found"
level=debug msg="Setting up configuration: file /opt/app-root/src/.config/containers/toolbox.conf not found"
level=debug msg="Resolving container and image names"
level=debug msg="Container: ''"
level=debug msg="Distribution (CLI): ''"
level=debug msg="Image (CLI): ''"
level=debug msg="Release (CLI): ''"
level=debug msg="Resolved container and image names"
level=debug msg="Container: 'rhel-toolbox-9.3'"
level=debug msg="Image: 'toolbox:9.3'"
level=debug msg="Release: '9.3'"
level=debug msg="Creating /run/.toolboxenv"
level=debug msg="Path /run/host/etc exists"
level=debug msg="Preparing to redirect /etc/host.conf to /run/host/etc/host.conf"
level=debug msg="/run/host/etc/host.conf isn't a symbolic link"
level=debug msg="Redirecting /etc/host.conf to /run/host/etc/host.conf"
level=debug msg="Preparing to redirect /etc/hosts to /run/host/etc/hosts"
level=debug msg="/run/host/etc/hosts isn't a symbolic link"
level=debug msg="Redirecting /etc/hosts to /run/host/etc/hosts"
level=debug msg="Preparing to redirect /etc/localtime to /run/host/etc/localtime"
level=debug msg="/run/host/etc/localtime is a symbolic link"
level=debug msg="Redirecting /etc/localtime to /run/host/etc/localtime"
level=debug msg="Resolved /etc/localtime to /run/host/usr/share/zoneinfo/Europe/Berlin"
level=debug msg="Preparing to redirect /etc/resolv.conf to /run/host/etc/resolv.conf"
level=debug msg="/run/host/etc/resolv.conf is a symbolic link"
level=debug msg="Redirecting /etc/resolv.conf to /run/host/etc/resolv.conf"
level=debug msg="Preparing to redirect /media to /run/media"
level=debug msg="/run/media isn't a symbolic link"
level=debug msg="Redirecting /media to /run/media"
level=debug msg="Preparing to redirect /mnt to /var/mnt"
level=warning msg="/var/mnt not found"
level=debug msg="Redirecting /mnt to /var/mnt"
level=debug msg="Creating regular file /etc/machine-id"
level=debug msg="Binding /etc/machine-id to /run/host/etc/machine-id"
level=debug msg="Creating directory /run/systemd/journal"
level=debug msg="Binding /run/systemd/journal to /run/host/run/systemd/journal"
level=debug msg="Creating directory /run/systemd/resolve"
level=debug msg="Binding /run/systemd/resolve to /run/host/run/systemd/resolve"
level=debug msg="Creating directory /run/systemd/sessions"
level=debug msg="Binding /run/systemd/sessions to /run/host/run/systemd/sessions"
level=debug msg="Creating directory /run/systemd/system"
level=debug msg="Binding /run/systemd/system to /run/host/run/systemd/system"
level=debug msg="Creating directory /run/systemd/users"
level=debug msg="Binding /run/systemd/users to /run/host/run/systemd/users"
level=debug msg="Creating directory /run/udev/data"
level=debug msg="Binding /run/udev/data to /run/host/run/udev/data"
level=debug msg="Creating directory /run/udev/tags"
level=debug msg="Binding /run/udev/tags to /run/host/run/udev/tags"
level=debug msg="Creating directory /tmp"
level=debug msg="Binding /tmp to /run/host/tmp"
level=debug msg="Creating directory /var/lib/flatpak"
level=debug msg="Binding /var/lib/flatpak to /run/host/var/lib/flatpak"
level=debug msg="Creating directory /var/lib/systemd/coredump"
level=debug msg="Binding /var/lib/systemd/coredump to /run/host/var/lib/systemd/coredump"
level=debug msg="Creating directory /var/log/journal"
level=debug msg="Binding /var/log/journal to /run/host/var/log/journal"
level=debug msg="Creating directory /var/mnt"
level=debug msg="Binding /var/mnt to /run/host/var/mnt"
level=debug msg="Creating directory /sys/fs/selinux"
level=debug msg="Binding /sys/fs/selinux to /usr/share/empty"
level=debug msg="Preparing to redirect /home to /var/home"
level=debug msg="/var/home isn't a symbolic link"
level=debug msg="Redirecting /home to /var/home"
level=debug msg="Looking up group for sudo"
level=debug msg="Group for sudo is wheel"
level=debug msg="Modifying user badger with UID 1000:"
level=debug msg=usermod
level=debug msg=--append
level=debug msg=--groups
level=debug msg=wheel
level=debug msg=--home
level=debug msg=/home/badger
level=debug msg=--password
level=debug
level=debug msg=--shell
level=debug msg=/bin/bash
level=debug msg=--uid
level=debug msg=1000
level=debug msg=badger
level=debug msg="Removing password for user root"
passwd: Note: deleting a password also unlocks the password.
level=debug msg="Setting KCM as the default Kerberos credential cache"
level=debug msg="Configuring RPM to ignore bind mounts"
level=debug msg="Setting up daily ticker"
level=debug msg="Setting up watches for file system events"
level=debug msg="Finished initializing container"
level=debug msg="Creating runtime directory /run/user/1000/toolbox"
level=debug msg="Creating initialization stamp /run/user/1000/toolbox/container-initialized-17106"
level=debug msg="Listening to file system and ticker events"
level=warning msg="Failed to run updatedb(8): updatedb(1) not found"

[BadgerOps]

Looks like specifically the go-toolset image has changed since the blog post was written, the following works as expected:

1. Create a Dockerfile

   FROM registry.access.redhat.com/ubi8/ubi:latest
   RUN yum module install -y go-toolset

2. podman build -t go-toolset-custom -f Dockerfile

3. toolbox create --image go-toolset-custom go-toolset

I'll go ahead and close this, but wanted to wrap up the question with the path forward I took.

[debarshiray]

I am glad that you managed to find a path forward!

Toolbx doesn't expect to be able to create containers from any random OCI image out there. Toolbx containers are set up in very specific ways, which might conflict with some images, and, ultimately, it's not trying to compete with podman run ....

That's why there's this idea of Toolbx images. Notice how only a subset of Podman containers and images show up in toolbox list. These are images with the com.github.containers.toolbox="true" label, which indicates that whoever created the image intended it to work with Toolbx, tested it, etc..

We don't enforce the com.github.containers.toolbox="true" label as diligently as we perhaps should, to give users an escape hatch if they want to venture into unknown territory. eg., the create and enter commands don't. I think we should at least add a warning.

Anyway, thanks for stopping by and playing with Toolbx.

[BadgerOps]

Hey, I appreciate the comment (and toolbx!) - good to know. I was following along with that rhel blogpost, and clearly they've changed their upstream container image which obviously broke what the blog post posited. Very good to know about the label, and it makes sense that there are some specific implementation steps for a toolbx container.

-BadgerOps

[debarshiray]

I was following along with that rhel blogpost, and clearly they've changed their upstream container image which obviously broke what the blog post posited.

Oh, now I actually read the blog post carefully, and I see that it mentions using Toolbx with the Go Toolset container images. Wow, I didn't expect that. :)

[BadgerOps]

I was following along with that rhel blogpost, and clearly they've changed their upstream container image which obviously broke what the blog post posited.

Oh, now I actually read the blog post carefully, and I see that it mentions using Toolbx with the Go Toolset container images. Wow, I didn't expect that. :)

Things evolve over time, and blogs tend to (sadly) atrophy. Glad there is a path forward, I was just super confused at the error output. I learned some more about how the UID mapping works at build, and that was a win overall in my book.