Build failure: dockerTools - unable to create image for pkgs.pkgsCross.musl64.hello

[alekseysidorov]

Steps To Reproduce

When I try to build a simple Docker image on Apple Silicon hardware I get errors like this.

nix-build docker.nix
these 6 derivations will be built:
  /nix/store/l2scib8r1152f1in0b779jx6iwf58snw-hello-base.json.drv
  /nix/store/g2a8l10m3q9079clcgk84zi7ixs9knh5-closure.drv
  /nix/store/5vyr2g6cbby5kj89s2ry1m6ccxf27p4c-closure-paths.drv
  /nix/store/wh4jmwc7n149kmy81dgmwpir9fmw4cd1-hello-conf.json.drv
  /nix/store/if0md9xf34jcbnsdmp8i09hk6795f2si-stream-hello.drv
  /nix/store/r8jcpsyfa03yd1i4lh5gq0nfc8cwqhih-hello.tar.gz.drv
building '/nix/store/l2scib8r1152f1in0b779jx6iwf58snw-hello-base.json.drv'...
building '/nix/store/g2a8l10m3q9079clcgk84zi7ixs9knh5-closure.drv'...
building '/nix/store/5vyr2g6cbby5kj89s2ry1m6ccxf27p4c-closure-paths.drv'...
building '/nix/store/wh4jmwc7n149kmy81dgmwpir9fmw4cd1-hello-conf.json.drv'...
{
  "architecture": "amd64",
  "config": {
    "Cmd": [
      "hello"
    ],
    "WorkingDir": "/"
  },
  "os": "linux",
  "store_dir": "/nix/store",
  "from_image": null,
  "store_layers": [
    [
      "/nix/store/vchdxrmywj32aby8pdg1v7sylw4ydx2v-musl-x86_64-unknown-linux-musl-1.2.3"
    ],
    [
      "/nix/store/3n8bh6zixwpac3cilqh957ya03hj25bb-hello-x86_64-unknown-linux-musl-2.12.1"
    ]
  ],
  "customisation_layer": "/nix/store/cmazylrxwrfhrxzzql0jhz4ph9lngk19-hello-customisation-layer",
  "repo_tag": "hello:lflaljjk8lc1j9zk4xkh0sz68wcj36p7",
  "created": "1970-01-01T00:00:01+00:00"
}
building '/nix/store/if0md9xf34jcbnsdmp8i09hk6795f2si-stream-hello.drv'...
building '/nix/store/r8jcpsyfa03yd1i4lh5gq0nfc8cwqhih-hello.tar.gz.drv'...
No 'fromImage' provided
Creating layer 1 from paths: ['/nix/store/vchdxrmywj32aby8pdg1v7sylw4ydx2v-musl-x86_64-unknown-linux-musl-1.2.3']
Traceback (most recent call last):
  File "/nix/store/ajfxznk2mlmmqvljn86ilf0sjk9gr84p-stream", line 392, in <module>
    main()
  File "/nix/store/ajfxznk2mlmmqvljn86ilf0sjk9gr84p-stream", line 340, in main
    info = add_layer_dir(tar, store_layer, store_dir, mtime=mtime)
  File "/nix/store/ajfxznk2mlmmqvljn86ilf0sjk9gr84p-stream", line 232, in add_layer_dir
    archive_paths_to(
  File "/nix/store/ajfxznk2mlmmqvljn86ilf0sjk9gr84p-stream", line 94, in archive_paths_to
    ti = append_root(tar.gettarinfo(filename))
  File "/nix/store/6y1di05r2i6ks6f737r0ckaih0gw3jss-python3-3.10.11/lib/python3.10/tarfile.py", line 1893, in gettarinfo
    linkname = os.readlink(name)
PermissionError: [Errno 13] Permission denied: '/nix/store/vchdxrmywj32aby8pdg1v7sylw4ydx2v-musl-x86_64-unknown-linux-musl-1.2.3/lib/ld-musl-x86_64.so.1'
error: builder for '/nix/store/r8jcpsyfa03yd1i4lh5gq0nfc8cwqhih-hello.tar.gz.drv' failed with exit code 1;
       last 10 log lines:
       >     main()
       >   File "/nix/store/ajfxznk2mlmmqvljn86ilf0sjk9gr84p-stream", line 340, in main
       >     info = add_layer_dir(tar, store_layer, store_dir, mtime=mtime)
       >   File "/nix/store/ajfxznk2mlmmqvljn86ilf0sjk9gr84p-stream", line 232, in add_layer_dir
       >     archive_paths_to(
       >   File "/nix/store/ajfxznk2mlmmqvljn86ilf0sjk9gr84p-stream", line 94, in archive_paths_to
       >     ti = append_root(tar.gettarinfo(filename))
       >   File "/nix/store/6y1di05r2i6ks6f737r0ckaih0gw3jss-python3-3.10.11/lib/python3.10/tarfile.py", line 1893, in gettarinfo
       >     linkname = os.readlink(name)
       > PermissionError: [Errno 13] Permission denied: '/nix/store/vchdxrmywj32aby8pdg1v7sylw4ydx2v-musl-x86_64-unknown-linux-musl-1.2.3/lib/ld-musl-x86_64.so.1'
       For full logs, run 'nix log /nix/store/r8jcpsyfa03yd1i4lh5gq0nfc8cwqhih-hello.tar.gz.drv'.

Here is the docker.nix file content which I used to build image.

let
  pkgs = import <nixpkgs> { };
  pkgsCross = pkgs.pkgsCross.musl64;
in
pkgsCross.pkgsBuildHost.dockerTools.buildLayeredImage {
  name = "hello";

  contents = [
    pkgsCross.hello
  ];

  config = {
    Cmd = [ "hello" ];
    WorkingDir = "/";
  };
}

Additional context

I found a patch that fixes my problem, but I have to use the patched nixpkgs to fix dockerTools.

Metadata

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"aarch64-darwin"`
 - host os: `Darwin 23.1.0, macOS 14.1.1`
 - multi-user?: `yes`
 - sandbox: `no`
 - version: `nix-env (Nix) 2.17.1`
 - channels(root): `"nixpkgs"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixpkgs`

[Artturin]

Cant see a reason why it would not be readable

https://github.com/bminor/musl/blob/f314e133929b6379eccc632bef32eaebb66a7335/Makefile#L161-L163

https://github.com/bminor/musl/blob/f314e133929b6379eccc632bef32eaebb66a7335/Makefile#L194-L195

https://github.com/bminor/musl/blob/f314e133929b6379eccc632bef32eaebb66a7335/Makefile#L212-L213

[torgeir]

I see similar results on an m3 mac with a fresh nix-darwin install using the same docker.nix

> nix-shell -p nix-info --run "nix-info -m"
 - system: `"aarch64-darwin"`
 - host os: `Darwin 23.2.0, macOS 14.2.1`
 - multi-user?: `yes`
 - sandbox: `no`
 - version: `nix-env (Nix) 2.18.1`
 - channels(root): `"nixpkgs"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixpkgs`

ends up with

...
building '/nix/store/gqhknrkiwdvbrka712hbm5d1zl5jpnap-hello-conf.json.drv'...
{
  "architecture": "amd64",
  "config": {
    "Cmd": [
      "hello"
    ],
    "WorkingDir": "/"
  },
  "os": "linux",
  "store_dir": "/nix/store",
  "from_image": null,
  "store_layers": [
    [
      "/nix/store/wg3p6wqlkg4nm3pj9mmmp7c8860awqsr-musl-x86_64-unknown-linux-musl-1.2.3"
    ],
    [
      "/nix/store/zjnypljflfvfdvn5hlka1ip5w80jkd89-hello-x86_64-unknown-linux-musl-2.12.1"
    ]
  ],
  "customisation_layer": "/nix/store/wl1g5byz0ajzlhp3x35yq55l0dzx37c5-hello-customisation-layer",
  "repo_tag": "hello:y582mjjbsza2136d1hm8d4py9djan9cc",
  "created": "1970-01-01T00:00:01+00:00"
}
building '/nix/store/jhxkxrc6jly3z32lqdz7qynqbanij1jz-stream-hello.drv'...
building '/nix/store/5i2bgdkbh1plh88bp2b22kqksl7rwpgw-hello.tar.gz.drv'...
No 'fromImage' provided
Creating layer 1 from paths: ['/nix/store/wg3p6wqlkg4nm3pj9mmmp7c8860awqsr-musl-x86_64-unknown-linux-musl-1.2.3']
Traceback (most recent call last):
  File "/nix/store/45kjnc5bdy11p9y5j00a8vjbbknnglnj-stream", line 392, in <module>
    main()
  File "/nix/store/45kjnc5bdy11p9y5j00a8vjbbknnglnj-stream", line 340, in main
    info = add_layer_dir(tar, store_layer, store_dir, mtime=mtime)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/45kjnc5bdy11p9y5j00a8vjbbknnglnj-stream", line 232, in add_layer_dir
    archive_paths_to(
  File "/nix/store/45kjnc5bdy11p9y5j00a8vjbbknnglnj-stream", line 94, in archive_paths_to
    ti = append_root(tar.gettarinfo(filename))
                     ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/z0nnwdh4p692yvf3qgwgj3bn78w99q6y-python3-3.11.6/lib/python3.11/tarfile.py", line 2065, in gettarinfo
    linkname = os.readlink(name)
               ^^^^^^^^^^^^^^^^^
PermissionError: [Errno 13] Permission denied: '/nix/store/wg3p6wqlkg4nm3pj9mmmp7c8860awqsr-musl-x86_64-unknown-linux-musl-1.2.3/lib/ld-musl-x86_64.so.1'
error: builder for '/nix/store/5i2bgdkbh1plh88bp2b22kqksl7rwpgw-hello.tar.gz.drv' failed with exit code 1;
       last 10 log lines:
       >            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
       >   File "/nix/store/45kjnc5bdy11p9y5j00a8vjbbknnglnj-stream", line 232, in add_layer_dir
       >     archive_paths_to(
       >   File "/nix/store/45kjnc5bdy11p9y5j00a8vjbbknnglnj-stream", line 94, in archive_paths_to
       >     ti = append_root(tar.gettarinfo(filename))
       >                      ^^^^^^^^^^^^^^^^^^^^^^^^
       >   File "/nix/store/z0nnwdh4p692yvf3qgwgj3bn78w99q6y-python3-3.11.6/lib/python3.11/tarfile.py", line 2065, in gettarinfo
       >     linkname = os.readlink(name)
       >                ^^^^^^^^^^^^^^^^^
       > PermissionError: [Errno 13] Permission denied: '/nix/store/wg3p6wqlkg4nm3pj9mmmp7c8860awqsr-musl-x86_64-unknown-linux-musl-1.2.3/lib/ld-musl-x86_64.so.1'
       For full logs, run 'nix log /nix/store/5i2bgdkbh1plh88bp2b22kqksl7rwpgw-hello.tar.gz.drv'.

(Edit: The slightly different example from nix.dev also fails with the same error, after replacing pkgsLinux.hello with pkgs.pkgsCross.musl64.hello)

[NoRePercussions]

Here's a great writeup by Tim Cuthbertson including a patch -- essentially, cross-compiling a linux package on darwin means that it uses a darwin filesystem, meaning that symlinks have their own permission modes and are affected by umask.