rpm-ostree doesn't support distros without UsrMerge (e.g. openSUSE)

[damianatorrpm]

Follow up on email thread %ostree-list% I am trying to build silverblue (for now) on openSUSE Tumbleweed.

libostree-2020.3-2.3.x86_64
bubblewrap-0.4.1-1.2.x86_64
rpm-ostree-2020.4-3.1.x86_64

I cloned https://pagure.io/workstation-ostree-config/tree/master and did:

mkdir -p $HOME/cache
ostree --repo=$HOME/cache init
sudo rpm-ostree --repo=$HOME/cache/ compose tree
$HOME/test/workstation-ostree-config/fedora-silverblue.yaml

It fails with:

bwrap: execvp true: No such file or directory
error: bwrap test failed, see
<https://github.com/projectatomic/rpm-ostree/pull/429>: Executing
bwrap(true): Child process killed by signal 1

I also tried with a custom tree, but I think the problem is something else.

I do have /usr/bin/true

tux@localhost:> which /usr/bin/true
/usr/bin/true

[jlebon]

I'm not sure offhand. As a sanity-check, try running the bwrap-script-shell.sh script targeting / itself, e.g.:

$ scripts/bwrap-script-shell.sh / true
exec bwrap --dev /dev --proc /proc --dir /tmp --chdir / --unshare-pid --unshare-uts --unshare-ipc --unshare-cgroup-try --unshare-net --ro-bind /sys/block /sys/block --ro-bind /sys/bus /sys/bus --ro-bind /sys/class /sys/class --ro-bind /sys/dev /sys/dev --symlink usr/lib lib --symlink usr/lib64 lib64 --symlink usr/bin bin --symlink usr/sbin sbin --ro-bind usr /usr --ro-bind ./var /var --bind ./usr/etc /etc --tmpfs /var/tmp --tmpfs /var/lib/rpm-state true

[damianatorrpm]

@jlebon Thank you for looking into this.

Here is the output of 3 varieties:

tux@localhost:~/Downloads/test> ./bwrap-script-shell.sh /home/tux/Downloads/test true
exec bwrap --dev /dev --proc /proc --dir /tmp --chdir / --unshare-pid --unshare-uts --unshare-ipc --unshare-cgroup-try --unshare-net --ro-bind /sys/block /sys/block --ro-bind /sys/bus /sys/bus --ro-bind /sys/class /sys/class --ro-bind /sys/dev /sys/dev --ro-bind usr /usr --ro-bind ./var /var --bind ./usr/etc /etc --tmpfs /var/tmp --tmpfs /var/lib/rpm-state true
bwrap: Can't find source path usr: No such file or directory

tux@localhost:~/Downloads/test> ./bwrap-script-shell.sh / true
exec bwrap --dev /dev --proc /proc --dir /tmp --chdir / --unshare-pid --unshare-uts --unshare-ipc --unshare-cgroup-try --unshare-net --ro-bind /sys/block /sys/block --ro-bind /sys/bus /sys/bus --ro-bind /sys/class /sys/class --ro-bind /sys/dev /sys/dev --ro-bind usr /usr --ro-bind ./var /var --bind ./usr/etc /etc --tmpfs /var/tmp --tmpfs /var/lib/rpm-state true
bwrap: Can't mkdir /var/lib/rpm-state: Read-only file system

tux@localhost:~/Downloads/test> sudo ./bwrap-script-shell.sh / true
exec bwrap --dev /dev --proc /proc --dir /tmp --chdir / --unshare-pid --unshare-uts --unshare-ipc --unshare-cgroup-try --unshare-net --ro-bind /sys/block /sys/block --ro-bind /sys/bus /sys/bus --ro-bind /sys/class /sys/class --ro-bind /sys/dev /sys/dev --ro-bind usr /usr --ro-bind ./var /var --bind ./usr/etc /etc --tmpfs /var/tmp --tmpfs /var/lib/rpm-state true
bwrap: Can't mkdir /var/lib/rpm-state: Read-only file system

[damianatorrpm]

Additionally I tried:

tux@localhost:~/Downloads/test> sudo mkdir -p /var/lib/rpm-state
[sudo] password for root: 
tux@localhost:~/Downloads/test> ./bwrap-script-shell.sh / true
exec bwrap --dev /dev --proc /proc --dir /tmp --chdir / --unshare-pid --unshare-uts --unshare-ipc --unshare-cgroup-try --unshare-net --ro-bind /sys/block /sys/block --ro-bind /sys/bus /sys/bus --ro-bind /sys/class /sys/class --ro-bind /sys/dev /sys/dev --ro-bind usr /usr --ro-bind ./var /var --bind ./usr/etc /etc --tmpfs /var/tmp --tmpfs /var/lib/rpm-state true
bwrap: execvp true: No such file or directory

[jlebon]

One thing I noticed is that you don't have the --symlink arguments (compare with https://github.com/coreos/rpm-ostree/issues/2188#issuecomment-670130513), which means they're not symlinks in your rootfs either...

And now looking at https://en.opensuse.org/openSUSE:Usr_merge and then https://lists.opensuse.org/opensuse-factory/2020-08/msg00035.html, it seems like UsrMove was never completed in openSUSE. So I think what's going on here is that coreutils is still installing in /bin and not /usr/bin. Can you give the output of ls -l /bin/true /usr/bin/true?

It wouldn't too hard to patch rpm-ostree to support those distros I think, though there's a few spots where we currently assume UsrMove.

[damianatorrpm]

Can you give the output of ls -l /bin/true /usr/bin/true

ls -l /bin/true /usr/bin/true
lrwxrwxrwx 1 root root    15 Aug  2 14:21 /bin/true -> ../usr/bin/true
-rwxr-xr-x 1 root root 31144 Aug  2 14:21 /usr/bin/true

/bin contains 2 files that are there the rest in it are symlinks

[jlebon]

What does your PATH look like? Is it ordering /bin before /usr/bin? Can you try the bwrap shell script again but specifying the full path (/usr/bin/true)?

[damianatorrpm]

What does your PATH look like?

PATH=/usr/local/bin:/usr/bin:/bin:/home/tux/bin

Can you try the bwrap shell script again but specifying the full path (/usr/bin/true)?

I don't see where the mentioned script uses true

[jlebon]

I don't see where the mentioned script uses true

I mean ./bwrap-script-shell.sh / /usr/bin/true.

[damianatorrpm]

Unfortunately same result,

tux@localhost:~/Downloads/test> ./bwrap-script-shell.sh / /usr/bin/true
exec bwrap --dev /dev --proc /proc --dir /tmp --chdir / --unshare-pid --unshare-uts --unshare-ipc --unshare-cgroup-try --unshare-net --ro-bind /sys/block /sys/block --ro-bind /sys/bus /sys/bus --ro-bind /sys/class /sys/class --ro-bind /sys/dev /sys/dev --ro-bind usr /usr --ro-bind ./var /var --bind ./usr/etc /etc --tmpfs /var/tmp --tmpfs /var/lib/rpm-state /usr/bin/true
bwrap: execvp /usr/bin/true: No such file or directory

[jlebon]

Try this:

diff --git a/scripts/bwrap-script-shell.sh b/scripts/bwrap-script-shell.sh
index 1f6ac28b..43a908e6 100755
--- a/scripts/bwrap-script-shell.sh
+++ b/scripts/bwrap-script-shell.sh
@@ -19,6 +19,8 @@ done
 for src in lib{,32,64} bin sbin; do
     if test -L $src; then
         BWRAP_ARGV="$BWRAP_ARGV --symlink usr/$src $src"
+    elif test -d $src; then
+        BWRAP_ARGV="$BWRAP_ARGV --ro-bind $src /$src"
     fi
 done
 BWRAP_ARGV="$BWRAP_ARGV --ro-bind usr /usr --ro-bind ./var /var --bind ./usr/etc /etc \

Even though you don't strictly need the /bin and /sbin bind mounts if all the binaries you care about are already in /usr, you still at least need /lib* which could be hardcoded in binaries. That's probably why you're getting "No such file or directory".

[damianatorrpm]

Yes the script would pass than with and without full path for true

tux@localhost:~/Downloads/test> ./bwrap-script-shell.sh / /usr/bin/true
exec bwrap --dev /dev --proc /proc --dir /tmp --chdir / --unshare-pid --unshare-uts --unshare-ipc --unshare-cgroup-try --unshare-net --ro-bind /sys/block /sys/block --ro-bind /sys/bus /sys/bus --ro-bind /sys/class /sys/class --ro-bind /sys/dev /sys/dev --ro-bind lib /lib --ro-bind lib64 /lib64 --ro-bind bin /bin --ro-bind sbin /sbin --ro-bind usr /usr --ro-bind ./var /var --bind ./usr/etc /etc --tmpfs /var/tmp --tmpfs /var/lib/rpm-state /usr/bin/true

Patching this file would mean rpm-ostree would succeed as well?

[jlebon]

Patching this file would mean rpm-ostree would succeed as well?

No, this script is just a helper to emulate what rpm-ostree itself does. Was digging a bit more into this and I'm not sure if it's worth the complexity to support this if openSUSE is in the process of converting to UsrMove eventually anyway. It would be more productive to help out with that effort instead. :)

[damianatorrpm]

openSUSE packages are migrated slowly towards the UsrMerge. This requires a lot of individual packages to be changed.

Is the assumption correct that rpm-ostree should work once /bin is a symlink to /usr/bin and /sbin to /usr/sbin?

[jlebon]

Is the assumption correct that rpm-ostree should work once /bin is a symlink to /usr/bin and /sbin to /usr/sbin?

Correct, as well as /lib* -> /usr/lib*.

[damianatorrpm]

@jlebon Unfortunately after further communication with suse team, it seems that an UsrMerge is not planned for the near future or may not even happen at all.