support merged-usr filesystem layout

[mythi]

Describe the bug To follow the merged-usr approach, I'm installing a statically linked toybox (with sh) to /install_root/usr/bin and then

...
FROM gcr.io/distroless/static
COPY --from=builder /install_root /

Distroless /bin is a directory so I loose the typical shell shebangs (e.g., /bin/sh). I've tried to workaround it by creating the /bin symlink in /install_root but Buildkit is not happy about it since it cannot deal with /bin/ already being a directory:

ERROR: failed to solve: cannot replace to directory /srv/docker/overlay2/nkktg2540y5ygxuk7fmauz7ec/merged/bin with file

FWIW, it works with DOCKER_BUILDKIT=0 but this is deprecated in favor of Buildkit. What also works OK, is:

...
FROM debian/stable-slim
COPY --from=builder /install_root /

To Reproduce see the description.

Expected behavior "merged /usr directories scheme, i.e. the /{bin,sbin,lib}/ directories becoming symbolic links to /usr/{bin,sbin,lib}/. "

Console Output N/A

Additional context "In February 2021, the Technical Committee has resolved that Debian 'bookworm' should support only the merged-usr root filesystem layout, dropping support for the non-merged-usr layout."

There seems to be strong connection between Debian 12 and the latest distroless so I though this could apply.

[loosebazooka]

Hrmm yeah I think we can adjust for this by creating those symlinks. Not sure what's creating that bin directory though

[mythi]

that's probably base-files but usr-merging adds something on top. This might be useful source https://github.com/debuerreotype/docker-debian-artifacts/issues/131

[mythi]

Distroless /bin is a directory so I loose the typical shell shebangs (e.g., /bin/sh). I've tried to workaround it by creating the

offtopic: I noticed that the :debug images fail on this. What triggered this issue originally was that I'm installing toybox to be able to run simple init scripts using initcontainers. Looks like the debug images (with busybox) cannot be used for this purpose either. (fixing :debug shell shebangs is separate from fixing this also).

[mythi]

Hrmm yeah I think we can adjust for this by creating those symlinks. Not sure what's creating that bin directory though

I looked into this today and my current thinking is that this is not going to be fully possible even with debian12.

I built two new layers:

diff --git a/base/base.bzl b/base/base.bzl
index b4b602e..8682a3c 100644
--- a/base/base.bzl
+++ b/base/base.bzl
@@ -96,6 +96,8 @@ def distro_components(distro):
                     # directory with specific permissions.
                     ":tmp.tar",
                     ":nsswitch.tar",
+                    ":usrmerge-wh.tar",
+                    ":usrmerge-link.tar",
                     "//os_release:os_release_" + distro + ".tar",
                     ":cacerts_" + arch + "_" + distro + ".tar",
                 ],

with:

$ tar tf base/usrmerge-wh.tar 
./
./.wh.bin
./.wh.lib
./.wh.sbin
$ tar tf base/usrmerge-link.tar 
./
./sbin
./lib
./bin

libc6 isn't "converted" so I'm getting /lib/x86_64-linux-gnu/ back etc so this would only work for /{s}bin/. Debian base image is debootstraped which knows how to deal with all this.

FWIW, what works is enough to fix my issue reported here but I don't think this is acceptable solution here?

[mythi]

FWIW, what works is enough to fix my issue reported here but I don't think this is acceptable solution here?

@loosebazooka I can get this change isolated to one layer and with that it's possible to get /bin -> usr/bin and /sbin -> usr/sbin symlinks to the base. tmp.tar seems to be about "patching" base-files too so it would also be possible to re-purpose that to generic base-files adjustments layer.

Any feedback whether /bin -> usr/bin and /sbin -> usr/sbin alone would be acceptable?

[loosebazooka]

I would really have to look into whether all our layer additions don't overwrite this (basically debs and other files don't expand out into /bin or /sbin). It seems to me we'd have to restructure our whole build to do this right.

[loosebazooka]

It's easy to test though, we just add a test for every image to ensure those symlinks still exist.

[mythi]

It's easy to test though, we just add a test for every image to ensure those symlinks still exist.

that part looks straightforward indeed. I cannot get a clean ./test.sh even without any changes but I'll keep looking and hopefully get a draft PR shared soon

[sin-ack]

Could this please be supported now that only Debian 12 is available? I'm currently hitting an issue where packages I install via rules_distroless are pulling libc6, which places the dynamic loader (ld-linux-x86-64.so.2) in /usr/lib64, expecting /lib64 to be a symlink; however, on Distroless images it points to a separate copy of Glibc. This causes the Glibc and dynamic loader versions to be different at load time, and the program fails with a cryptic missing symbol error for version GLIBC_PRIVATE.

As a workaround, I currently:

• Build on top of distroless/static
• Add a pkg_tar layer which adds the lib64 symlink
• Add the packages layer