dockerTools.buildLayeredImage: contents completely replaces /bin

[aspiwack]

Describe the bug

When adding packages in the contents attribute of buildLayeredImage, the entire /bin directory gets overwritten, effectively breaking the starting image if there is one.

To Reproduce

Without contents

default.nix:

{ pkgs ? import <nixpkgs> {}}:

let
  ubuntu-image = pkgs.dockerTools.pullImage {
    imageName = "ubuntu";
    imageDigest = "sha256:aba80b77e27148d99c034a987e7da3a287ed455390352663418c0f2ed40417fe";
    sha256 = "0yb3gp7v7drhi5d62kq5x4lavphlazjyn784ln6qm7l1i0bv9s2b";
    finalImageName = "ubuntu";
    finalImageTag = "latest";
  };
in
pkgs.dockerTools.streamLayeredImage {
  name = "test";
  tag = "latest";

  fromImage = ubuntu-image;
}

Then

$ $(nix-build) | docker load
$ docker run -it test bash
# ls /bin
'['             clear_console             faillog     ln                 paste              setterm     umount
 addpart        cmp                       fallocate   locale             pathchk            sg          uname
 apt            comm                      false       locale-check       perl               sh          uncompress
 apt-cache      cp                        fgrep       localedef          perl5.30.0         sha1sum     unexpand
 apt-cdrom      csplit                    fincore     logger             pgrep              sha224sum   uniq
 apt-config     cut                       find        login              pidof              sha256sum   unlink
…

With contents

{ pkgs ? import <nixpkgs> {}}:

let
  ubuntu-image = pkgs.dockerTools.pullImage {
    imageName = "ubuntu";
    imageDigest = "sha256:aba80b77e27148d99c034a987e7da3a287ed455390352663418c0f2ed40417fe";
    sha256 = "0yb3gp7v7drhi5d62kq5x4lavphlazjyn784ln6qm7l1i0bv9s2b";
    finalImageName = "ubuntu";
    finalImageTag = "latest";
  };
in
pkgs.dockerTools.streamLayeredImage {
  name = "test";
  tag = "latest";

  fromImage = ubuntu-image;

  contents = [pkgs.hello];
}

Then

$ $(nix-build) | docker load
$ docker run -it test bash
# ls /bin
hello

Expected behavior

The hello symlink should be added to /bin without removing the existing content of the directory.

Additional context/metadata

$ docker --version
Docker version 20.10.7, build f0df350

$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 5.11.0-18-generic, Ubuntu, 21.04 (Hirsute Hippo)`
 - multi-user?: `no`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.3.7`
 - channels(aspiwack): `"nixpkgs-21.11pre299378.db6e089456c, home-manager"`
 - nixpkgs: `/home/aspiwack/.nix-defexpr/channels/nixpkgs`

@LnL7 @roberth

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute:
- dockerTools.buildLayeredImage
# a list of nixos modules affected by the problem
module:

[roberth]

With overlays, the last layer wins. In this case, that's the "customization layer" aka contents layer, where /bin is a symlink to a store path.

It seems that contents should be a function of the base image, at least conceptually. We'll probably want to do the merging in fakeroot.

contents should be redesigned regardless: https://github.com/NixOS/nixpkgs/issues/94636#issuecomment-755246646

[stale[bot]]

I marked this as stale due to inactivity. → More info

[W1M0R]

This issue is probably related #102962

[bohendo]

I hit this problem because by fromImage had a symlink bin -> usr/bin. This comment on issue #240919 points out that normal directories are merged as you'd expect but symlinks are overwritten entirely.

I had trouble with the suggestions in the above issue and related discussion. Solution I have now is to just repair the symlinks at the beginning of my entrypoint script eg:

    # fix clobbered symlinks from the base image
    for f in /usr/bin/*; do /bin/ln -s $f /bin/$(/bin/basename $f) 2>/dev/null || true; done
    for f in /usr/lib/*; do /bin/ln -s $f /lib/$(/bin/basename $f) 2>/dev/null || true; done

I needed to use the full path to ln & basename here (provided by nix coreutils) bc, before fixing the symlink to /usr/lib/x86_64-linux-gnu, all base image binaries in /usr/bin/ were broken and earlier in my path than /bin stuff.