dockerTools buildImage's contents (and config) nukes the prior layer's files for linked directories

[TomMD]

Describe the bug

If you use buildImage with a base image (fromImage) and define a non-empty contents such as contents = [ pkgs.gnumake ], then the base image's /bin or other linked paths will not be available in the final image.

Steps To Reproduce

Thread: https://discourse.nixos.org/t/building-on-dockerfile-based-images/29583/10

Consider this file:

{ pkgs ? import (fetchTarball "https://github.com/NixOS/nixpkgs/archive/a71e45961e88c8a6dde6287fa1e061f30f8c2fb7.tar.gz") { }
}:

pkgs.dockerTools.buildLayeredImage {
    name = "owner";
    tag = "latest";
    fromImage = pkgs.dockerTools.pullImage {
        imageName = "ubuntu";
        imageDigest = "sha256:83f0c2a8d6f266d687d55b5cb1cb2201148eb7ac449e4202d9646b9083f1cee0";
        sha256 = "sha256-5y6ToMw1UGaLafjaN69YabkjyCX61FT3QxU4mtmXMP0=";
        finalImageName = "ubuntu";
        finalImageTag = "latest";
        os = "linux";
        arch = "x86_64";
    };
    contents = with pkgs ; [ gnumake ];
}

Build it an look at the /bin directory, which should have > 100 files if it had all the base image (Ubuntu) files:

docker load < $(nix-build Dockerfile.nix)  0.89s user 0.21s system 17% cpu 6.164 total
tommd@wr /tmp% docker run --rm -it owner:latest bash
root@f66c66371ee9:/# ls /bin
make

Just the one make file provided by the pkgs.gnumake contents. The story is the same if we use copyToRoot (see above-linked thread).

Expected behavior

The semantics are surprising and quite likely a bug. Most people seem to expect the extra packages to be linked or copied into the target directories (i.e. /bin) and the prior files provided by the base image (ubuntu) to remain available.

nix-env --version output

% nix-env --version
nix-env (Nix) 2.8.1

[roberth]

When /bin is a directory, the container runtime would merge the layer correctly, so I would guess that the "customization layer" of buildLayeredImage makes /bin a symlink which then shadows the lower layers.

You might work around this problem by installing packages manually using fakeRootCommands. We should have some standard code that does that for us, and I think that should be based on NixOS modules.

I've previously made an effort in this direction

• https://github.com/NixOS/nixpkgs/pull/148456

Sometimes I clean up and forward-port a commit or idea from there, but this project really needs more than just me to make some real progress.

[TomMD]

@roberth Yes, that turns out to be the exact issue. /bin is a soft link and the copyToRoot effectively creates a real /bin directory with the only contents being my one derivation. I'll have to consider best options. Thanks.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/building-on-dockerfile-based-images/29583/18

[rvolosatovs]

At least in case of Debian starting image I was able to fix this with minimal changes by using the following copyToRoot for my simple use case (a static binary, which requires CA certs):

copyToRoot = pkgs.buildEnv {
  name = "wasmcloud";
  extraPrefix = "/usr"; # /bin is a symlink to /usr/bin on Debian, add a prefix to avoid replacing original `/bin`
  paths = [
    bin # the binary with a `$out/bin/wasmcloud`

    pkgs.dockerTools.caCertificates
  ];
  postBuild = ''
    mv $out/usr/etc $out/etc
  '';

This merged the binary in /usr/bin with all existing Debian binaries as expected and I was also able to access the CA cert bundle in /etc