`enableFakechroot` doesn't work on darwin

[Mastermindaxe]

Describe the bug

When building a docker image using dockerTools.buildLayeredImage one can't use enableFakechroot because the package proot is not built for aarch64-darwin. This prevents building the docker image on darwin.

Steps To Reproduce

Flake to reproduce the behavior:

{
  inputs = { nixpkgs.url = "github:nixos/nixpkgs?ref=nixos-unstable"; };

  outputs = { self, nixpkgs }:
    let pkgs = nixpkgs.legacyPackages.aarch64-darwin;
    in {
      packages.aarch64-darwin.default = pkgs.dockerTools.buildLayeredImage {
        name = "enableFacechroot doesn't work on darwin'";
        enableFakechroot = true;
      };
    };
}

Expected behavior

Not sure about this. I think a better error message would be nice. As in "Cannot use enableFakechroot with aarch64-darwin" or sth similar.

Additional context

Stumbled upon this when building a java microservice for work and packaging it into a Docker image.

Documentation of the option in the nixpkgs reference documentation: https://nixos.org/manual/nixpkgs/stable/#:~:text=enableFakechroot%20(Boolean;%20optional)

Notify maintainers

@NixOS/darwin-maintainers and bc I saw you on the git blame: @the-sun-will-rise-tomorrow @Mic92 @mplanchard (Did I miss the official maintainers of dockerTools? I couldn't find them)

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

- system: `"aarch64-darwin"`
- host os: `Darwin 23.4.0, macOS 14.4`
- multi-user?: `yes`
- sandbox: `no`
- version: `nix-env (Nix) 2.18.1`
- channels(root): `"nixpkgs"`
- nixpkgs: `/Users/<me>/.nix-defexpr/channels/nixpkgs`

---

Add a :+1: reaction to issues you find important.

[Mic92]

How do you build docker containers on darwin anyway? Cross-compiling?

[Mastermindaxe]

@Mic92 Not really. Haven't yet figured out how to successfully run a docker image that's built on Darwin, but it should be possible, right? I mean there are Docker images that are specifically built for aarch64-darwin on the docker hub as the underlying architecture. That's the next issue I wanted to take a look at 😄

[Mastermindaxe]

That said: If it's entirely unfeasible to build docker images on a mac I'm totally fine with that. We should add that to the documentation though and notify users that way. I vaguely recall reading about it but can't find it in the reference docs 🤔

[Mic92]

Ok. Looks like it's possible now: https://osxdaily.com/2023/10/13/native-macos-docker-containers-are-now-possible/

[the-sun-will-rise-tomorrow]

Sorry, I don't really know much about Darwin!

Looking online, I can see that proot indeed can't be ported to Darwin because that kernel doesn't provide a way to pause program execution on syscalls.

proot is roughly equivalent to fakeroot + fakechroot. However, though fakeroot has been ported to Darwin, fakechroot hasn't, and I don't see an equivalent.

One option that would work in theory is to run the fakeRootCommands and create the tarball in a VM. But, I don't know if it's even possible to create / run Darwin VMs from Nix, seeing how the kernel is proprietary.

If Docker has a method of running Darwin containers on Darwin without a full VM, maybe that could also be leveraged somehow, but I don't know what that would look like at all.

[Mastermindaxe]

@the-sun-will-rise-tomorrow Thanks for the input and for looking it up!

I'd say in the meantime we could just add a check if you're on darwin and then spit out a better error message. Would that work? Or make the option not usable if you're building it for a darwin system. Is that possible? Still getting the hang of things and especially all the built-in nix functions and stuff

[Mic92]

Last time I checked fakeroot does LD_PRELOAD wrong on macOS. I needs to use dyld_interpose just like libredirect: https://github.com/NixOS/nixpkgs/blob/709f544df6f8391df5405ba3b5990bf6a891c270/pkgs/build-support/libredirect/libredirect.c#L25

[the-sun-will-rise-tomorrow]

I'd say in the meantime we could just add a check if you're on darwin and then spit out a better error message.

Yes, I think this should do it:

diff --git a/pkgs/build-support/docker/default.nix b/pkgs/build-support/docker/default.nix
index 3319551480a..aac4b3eeaaa 100644
--- a/pkgs/build-support/docker/default.nix
+++ b/pkgs/build-support/docker/default.nix
@@ -23,6 +23,7 @@
 , runtimeShell
 , shadow
 , skopeo
+, stdenv
 , storeDir ? builtins.storeDir
 , substituteAll
 , symlinkJoin
@@ -917,6 +918,9 @@ rec {
       assert
       (lib.assertMsg (maxLayers > 1)
         "the maxLayers argument of dockerTools.buildLayeredImage function must be greather than 1 (current value: ${toString maxLayers})");
+      assert
+      (lib.assertMsg (enableFakechroot -> !stdenv.isDarwin)
+        "cannot use enableFakechroot on Darwin");
       let
         baseName = baseNameOf name;

Can you test the patch above?

Last time I checked fakeroot does LD_PRELOAD wrong on macOS.

Interesting... I've made the assumption based on the presence of the brew package. The page claims macOS support, on Intel and Apple silicon, on both Darwin and 64-bit Linux. Maybe they're applying Darwin compatibility patches?

[emilazy]

The “macOS containers” thing is third‐party software that involves disabling System Integrity Protection. macOS doesn’t really have the namespacing features you’d need for containers. You could do it with VMs, but Apple’s EULA restricts you to two of those simultaneously. So containers on macOS are not really a thing, for the most part.

fakeroot has been in Homebrew for many years; I wouldn’t be surprised if it used to work better. Or maybe it does work better from Homebrew and there’s something screwy about our build environment that’s breaking it; someone would have to check.

[emilazy]

I mean there are Docker images that are specifically built for aarch64-darwin on the docker hub as the underlying architecture.

I’m curious as to what images you’re looking at, but I suspect they might relate to the Tart VM solution, which happens to use OCI container images.

[Mastermindaxe]

I mean there are Docker images that are specifically built for aarch64-darwin on the docker hub as the underlying architecture.

I’m curious as to what images you’re looking at, but I suspect they might relate to the Tart VM solution, which happens to use OCI container images.

I'm confused now 😅 I just tried looking it up, but I had to be referring to linux/arm https://docs.docker.com/build/building/multi-platform/ as the docker architecture. I can't find any darwin specific stuff now. Maybe I just mangled those in my head or can't find the documentation referring to that 🤦

[Mastermindaxe]

@the-sun-will-rise-tomorrow Works like a charm under darwin:

error:
       … while calling the 'derivationStrict' builtin

         at /builtin/derivation.nix:9:12: (source not available)

       … while evaluating derivation 'enableFacechroot-doesn-t-work-on-darwin-.tar.gz'
         whose name attribute is located at /nix/store/mm46ykxk6nfggnfbhnry3lhi9zd9cfq3-source/pkgs/stdenv/generic/make-derivation.nix:334:7

       … while evaluating attribute 'buildCommand' of derivation 'enableFacechroot-doesn-t-work-on-darwin-.tar.gz'

         at /nix/store/mm46ykxk6nfggnfbhnry3lhi9zd9cfq3-source/pkgs/build-support/trivial-builders/default.nix:68:16:

           67|         enableParallelBuilding = true;
           68|         inherit buildCommand name;
             |                ^
           69|         passAsFile = [ "buildCommand" ]

       error: cannot use enableFakechroot on Darwin

[Mastermindaxe]

@the-sun-will-rise-tomorrow I've made a small PR with your patch. Hope that's okay. If not, please let me know and I'll remove it again.

PR: https://github.com/NixOS/nixpkgs/pull/327336 Also added a small line to the documentation of the enableFakechroot parameter. Don't know if there is a guide for doc contributions, but I thought I'd might as well add it there

[Mastermindaxe]

PR has landed to improve user experience: https://github.com/NixOS/nixpkgs/pull/327336

Do we want to keep this open to look for alternatives and for visibility or do we close this as we don't have any way of implementing this as of right now?