Provide shims for well-known binaries, which can only run on the host, inside the toolbox containers

[debarshiray]

Various commands like flatpak, podman, rpm-ostree etc. can't really be used inside an OCI container. Some of them, like podman, might have some limited use, but, by and large, people expect to run them directly on the host.

At the same time, we expect a good percentage of users of the command line interface to spend most of their time inside a toolbox container, and one of the goals of the Toolbox project is to reduce the cognitive burden of using mutable containers on locked-down and immutable host OSes like Silverblue. Commands like flatpak and rpm-ostree are important tools for interfacing with such OSes.

Therefore, it would be nice if we could provide a user experience that's better than having to switch back and forth between a host and toolbox shell, or having to prefix every command with things like flatpak-spawn --host.

The easiest option is to install aliases in the shell running inside the container. However, it doesn't give us things like manuals and shell completion.

Another option is to pre-install the flatpak, podman, rpm-ostree, etc. RPMs in the fedora-toolbox base images but remove all the code, leaving behind only the manuals and shell completion. Then the toolbox command can place the corresponding shims via bind mounts to forward the calls to the host when starting the toolbox containers.

This will ensure that the fedora-toolbox images don't get bigger with useless Go binaries, and the wrappers can be kept updated through the toolbox package on the host. One advantage of having explicit shims over aliases is that we can intercept those corner cases where command invocations can't be forwarded to the host. eg., they might involve a path that's not shared between the host and the container. Failing with a clear error message is better than an obscure failure or strange side-effects.

However, I don't know what will happen if one of these packages are updated inside the containers. Would they interfere with the bind mounted shims? It would be nice if we could trim the useless bits from those packages during or after updating the RPM.

[cgwalters]

I think it'll be dramatically simpler if we add /usr/libexec/toolbox/bin and change the shell startup scripts to inject that into $PATH first.

[mildred]

Running podman inside the toolbox can be immensely useful. I work with many projects where build scripts is just calling docker images, and being able to run those scripts inside the toolbox can be really useful. There might create hard to detect issues with bind mounts outside the home directory that would reflect the silverblue host and not the toolbox though.

[kamalmarhubi]

I'll add docker-compose to the list of things useful for dev flows. It has a really low barrier to entry for anything needing a data store or two, so a fair number of web app type things use it—open source and not.

[dustymabe]

I build a derivative toolbox FROM registry.fedoraproject.org/f31/fedora-toolbox:31. In this derivative toolbox I solved this problem by creating a host-runner script in /usr/local/bin/host-runner

$ cat /usr/local/bin/host-runner 
#!/bin/bash 
executable=$(basename $0)
set -x
exec flatpak-spawn --host $executable "$@"

I then create a symbolic link to host-runner for anything I want to execute in the host context:

$ ls -l /usr/local/bin/
total 4
lrwxrwxrwx. 1 root root 13 Jan 28 14:16 chromium-browser -> ./host-runner
-rwxr-xr-x. 1 root root 89 Jan 31 10:39 host-runner
lrwxrwxrwx. 1 root root 13 Jan 29 10:51 podman -> ./host-runner
lrwxrwxrwx. 1 root root 13 Jan 28 12:39 systemctl -> ./host-runner
lrwxrwxrwx. 1 root root 13 Jan 28 12:39 virsh -> ./host-runner
lrwxrwxrwx. 1 root root 13 Jan 28 12:39 virt-install -> ./host-runner

NOTE: the list of commands that a user wants proxied to the host probably is different for every user so we should probably just create a generic way to configure it and then let the users do that themselves.

You then execute things from within toolbox, but it gets proxied through to the host. The only extra bit is that the actual command that gets run is printed out on stderr before it does get run. For example:

$ virsh list --all
+ exec flatpak-spawn --host virsh list --all
 Id   Name              State
----------------------------------
 43   f31_vanilla-f31   running
 47   tester            running
 -    fcos              shut off

We could provide something like this in the created toolbox container. I'm still polishing this up a bit. One thing that doesn't work yet that we should think about is what to do if the user wanted to execute the command on the host as sudo.

[Ramblurr]

I tried to duplicate this, but I just placed host-runner in ~/.local/bin rather than build a new image.. and as a result of being lazy like that I got

[user@toolbox ~] podman ps
+ exec flatpak-spawn --host podman ps
+ exec flatpak-spawn --host podman ps
/var/home/user/.local/bin/podman: line 4: exec: flatpak-spawn: not found

Of course since ~/.local/bin is in the PATH on the host and the container, host-runner executed the podman symlink in ~/.local/bin.. which executed host-runner on the host, but flatpak-spawn wasn't installed there. I don't dare to think what nested recursion would have happened if it was :wink:

Anyways I solved it by changing

executable=$(basename $0) to executable=/usr/bin/$(basename $0) which works well enough for now.

EDIT: This causes a few other problems. For example toolbox on the host won't work anymore as it tries to call ~/.local/bin/podman. My solution for now is to have a separate bin path that I only add to PATH in my toolbox container.

[kamalmarhubi]

Using flatpak-spawn --host is a cool trick! Here's a pure sh* version of this that works in ~/.local/bin and avoids the infinite loop if you accidentally run the command directly:

#!/bin/sh
set -o errexit
set -o nounset

executable="$(basename "$0")"

if [ "$(basename "$(realpath "$0")")" = "${executable}" ]; then
  echo "can't run ${executable} via ${executable}" >&2
  exit 1
fi

# This seems like the best way to detect if we're inside a toolbox container.
if [ -n "${TOOLBOX_PATH:-}" ]; then
  set -x
  exec flatpak-spawn --host "${executable}" "$@"
fi

# Otherwise do a little dance to find the executable that would have run if not
# for $0 being on the path, and run that instead.
executable="$(
  # Remove this script's directory from PATH; this assumes that you'll never want
  # to run a sibling via this script.
  dir="$(dirname "$0")"
  PATH="$(echo "${PATH}" | sed "s+:${dir}:++")"
  PATH="$(echo "${PATH}" | sed "s+${dir}:++")"
  PATH="$(echo "${PATH}" | sed "s+:${dir}++")"
  command -v "${executable}"
)"

exec "${executable}" "$@"

(This hasn't been tested this in weird nested execution cases, though I think it should work.)

*assumes realpath is available

[wjt]

Another option could be to make podman within the container always use --remote. The socket for the API server appears to be visible within a toolbox container.

[nanonyme]

Can you please elaborate why flatpak cannot be used inside OCI container? It seems sad if you cannot build new version of flatpak and keep its possibly different runtime deps inside your toolbox but need to somehow hack them to your host.

[vorburger]

make podman within the container always use --remote. The socket for the API server appears to be visible within a toolbox container.

1072 created specifically about potentially pursuing that.

[matus-sabo]

I did create following files: ~/.local/shims/flatpak-spawn-host

#!/bin/bash 

executable=$(basename $0)
exec flatpak-spawn --host "${executable}" "$@"

~/.local/shims/toolbox-run

#!/bin/bash 

executable=$(basename $0)
exec toolbox run "${executable}" "$@"

Example for podman docker & docker-compose on host and toolbox Host symlinks:

sudo ln -s /usr/bin/podman /usr/local/bin/docker 
sudo ln -s ~/.local/shims/toolbox-run /usr/local/bin/docker-compose

Toolbox symlinks:

sudo ln -s ~/.local/shims/flatpak-spawn-host /usr/local/bin/podman
sudo ln -s ~/.local/shims/flatpak-spawn-host /usr/local/bin/docker