Full GCC is included in the Nix closure of built artifacts

[thomasjm]

When I build an executable, the Nix dependency closure includes the full GCC compiler. When I use the Nix Docker tools to build a Docker image with a Haskell app, this blows up the size by 143 MB.

To demonstrate, I created a blank Haskell project with stack new and added the basic default.nix from the Haskell.nix docs. You can find the repro here: https://github.com/thomasjm/haskell-nix-closure.

When I build this and look at the closure, I get the following:

/nix_frozen/store/9y11wgd44drql32f24ry82pigy7h7rqb-libunistring-0.9.10
/nix_frozen/store/y2yg7399m0mdgf253nbghsln4dq6li2d-libidn2-2.3.0
/nix_frozen/store/kng3jbgmpdl8g26jjw2bwy19djffna5s-glibc-2.30
/nix_frozen/store/dgallv6p7ll994nsk3jrvbjwk2z8jpvm-gcc-9.2.0-lib
/nix_frozen/store/07gb8sca1y6fvkrrsjg8ipcvyhkqp5vh-gmp-6.2.0
/nix_frozen/store/8f54hfixf98j1q20jdm6n9wzrw72fiqd-libffi-3.3
/nix_frozen/store/c6iliilr5cr2ga1fqblwqxj3zxjrgc17-libffi-3.3-dev
/nix_frozen/store/dimkrzmx10k8rr9k32v1446xmcc0ifp1-glibc-2.30-bin
/nix_frozen/store/fd8h7ivjh4dr8vyyr5hz78rk5kdzi8av-linux-headers-4.19.16
/nix_frozen/store/hc5i2qlw0nvr9qflda6yc0xlwnpi8lj8-glibc-2.30-dev
/nix_frozen/store/rfrn71d1dy43cw1gg6pm7xj72p37dd5z-zlib-1.2.11
/nix_frozen/store/ly011abgbh1n2w63rqxa98j1n6xa8zd4-gcc-9.2.0
/nix_frozen/store/xzzp9ryxrnhv208mf8g1v43xr6c3whns-numactl-2.0.13
/nix_frozen/store/vapxxa4xfiphjc42ymyc29psj5adlkgj-haskell-nix-closure-exe-haskell-nix-closure-exe-0.1.0.0

Notice how in addition to normal runtime libs, this contains gcc-9.2.0-lib (which is small and okay) as well as gcc-9.2.0 (which is big and shouldn't be a runtime dependency).

[michaelpj]

What's the output of nix why-depends <your image> <gcc derivation>?

[rvl]

Yes - would be good to fix - we also see it in the closure size test.

[thomasjm]

@michaelpj it seems to be a direct dependency:

╚═══bin/haskell-nix-closure-exe: ….includes/rts./nix_frozen/store/ly011abgbh1n2w63rqxa98j1n6xa8zd4-gcc-9.2.0/lib/gcc/x86_64-unknow…
    => /nix_frozen/store/ly011abgbh1n2w63rqxa98j1n6xa8zd4-gcc-9.2.0

There's some mention of includes/rts so maybe RTS related?

[michaelpj]

FWIW, non-static Haskell executables from nixpkgs also seem to depend on GCC, so it's not just us.

[thomasjm]

This looks sort of relevant as a way this can happen: https://github.com/NixOS/nixpkgs/issues/311 I think some of the lessons from this one might be relevant also: https://github.com/NixOS/nixpkgs/issues/34376

[thomasjm]

@Profpatsch, friendly ping -- any chance you know why this might happen (both here and in Nixpkgs in general)? I saw that you fixed the closure size issues with Pandoc.

[michaelpj]

Pandoc in nixpkgs is a static executable which is why it avoids this problem, I think. The question is whether it's avoidable for dynamically linked things too.

[eamsden]

I was just investigating this in our builds.

$ grep -oa '/nix/store/9ygpwbg32bc5689hwbzsjx6x58d1l6q7-gcc-9.2.0/.*' path/to/exe

I get a large number of .h files that appear to live in a copy of a nix store under gcc. A sampling:

/nix/store/9ygpwbg32bc5689hwbzsjx6x58d1l6q7-gcc-9.2.0/lib/gcc/x86_64-unknown-linux-gnu/9.2.0/include/nix/store/fwpn2f7a4iqszyydw7ag61zlnp6xk5d3-glibc-2.30-dev/include/bits/typesRtsMain.ctypes.hstdint-intn.hstdint-uintn.hTypes.hTSO.hBlock.hmath.hTicky.hHsFFI.hTime.hstddef.EventLogWriter.hRtsAPI.hClosures.hTypes.hInfoTables.hthread-shared-types.hpthreadtypes.htime.herrno.hOSThreads.hSpinLock.h<built-in>Messages.hThreads.hTask.hMiscClosures.hGC.hMBlock.hstruct_FILE.h    FILE.hstdio.hsys_errlist.hFlags.hStable.hRts.hRtsFlags.hPrelude.hStg.h  `
/nix/store/9ygpwbg32bc5689hwbzsjx6x58d1l6q7-gcc-9.2.0/lib/gcc/x86_64-unknown-linux-gnu/9.2.0/include/nix/store/fwpn2f7a4iqszyydw7ag61zlnp6xk5d3-glibc-2.30-dev/include/bits/typesrts/smProftimer.ctypes.hstdint-intn.hstdint-uintn.hTypes.hRegs.hCCS.hTSO.hGC.hBlock.hmath.hTicky.hHsFFI.hTime.stddef.h EventLogWriter.RtsAPI.hClosures.hCapability.hTypes.InfoTables.hthread-shared-types.hpthreadtypes.htime.herrno.hOSThreads.SpinLock.<built-in>Messages.Threads.Task.hMiscClosures.hMBlock.hstruct_FILE.h
/nix/store/9ygpwbg32bc5689hwbzsjx6x58d1l6q7-gcc-9.2.0/lib/gcc/x86_64-unknown-linux-gnu/9.2.0/include/nix/store/fwpn2f7a4iqszyydw7ag61zlnp6xk5d3-glibc-2.30-dev/include/bits/typesrts/smHsFFI.ctypes.hstdint-intn.hstdint-uintn.hTypes.hHsFFI.hTSO.hBlock.hmath.hTicky.hTime.hstddef.EventLogWriter.hRtsAPI.hClosures.hTypes.hInfoTables.hthread-shared-types.hpthreadtypes.htime.herrno.hOSThreads.hSpinLock.h<built-in>Messages.hThreads.hTask.hMiscClosures.hGC.hMBlock.hstruct_FILE.h    FILE.h  stdio.hsys_errlist.hFlags.hStable.hRts.hHeapAlloc.h

[eamsden]

As expected, setting { dontStrip = false; enableShared = false; } for the executable I tested eliminates the GCC dependency.

[thomasjm]

Ah nice @eamsden! That worked for me using only { dontStrip = false; }. The way that worked best was to apply it to my executable only, not the root modules options, i.e.

packages.my-package.components.exes.my-exe.dontStrip = false;

This raises the question of why haskell.nix seems to use dontStrip = true by default?

[michaelpj]

https://github.com/input-output-hk/haskell.nix/pull/336#discussion_r351491851

Quoth @angerman:

The annoying part with stripping is that it sometimes breaks haskell code. This is mostly due to how haskell code is layed out in object files, and the stripper being a tad bit too agressive.

[onepunchtech]

Where do you add those options? This doesn't seem to work

pkgs.haskell-nix.project {
  src = pkgs.haskell-nix.haskellLib.cleanGit {
    name = "foo";
    src = ./.;
  };
  compiler-nix-name = "ghc884";
  dontPatchElf = false;
  dontStrip = false;
}

[hughjfchen]

I met the same problem even I added the dontStrip = false option. The docker image exceeds 300M. What else should I take a look?

[michaelpj]

Try and use nix why-depends to find out why it depends on gcc and post the output here if you find it.

[hughjfchen]

OK, I figure it out why. In my case, I built a fully static linked haskell executable, yet somehow, there is a reference to the haskell library path in the static binary. Nix scans the static binary and found the reference path so it think that's a runtime dependency which causes it packed glibc, gcc, and the whole haskell dependent libs into the final docker image. After removing the lib reference path from the static binary, the final docker image size drops to around 3Mb. Obviously, for fully static linked executables, only patchelf --shrink-rpath and dontStrip=false is not enough, you need to remove all reference paths from the binary manually with nukeReferences or removeReferencesTo.

[goertzenator]

How do we apply the dontPatchElf and dontStrip options? It has not been stated in this thread nor could I find an example in the manual.

[goertzenator]

Aha, finally figured it out..

pkgs.haskell-nix.project {
  src = pkgs.haskell-nix.haskellLib.cleanGit {
    name = "foo";
    src = ./.;
  };
  compiler-nix-name = "ghc884";
  modules = [{
      packages.yourproject.components.exes.yourproject-exe = {
        dontStrip = false;
      };
  }];
}

[hamishmack]

https://github.com/input-output-hk/haskell.nix/pull/336#discussion_r351491851

Quoth @angerman:

The annoying part with stripping is that it sometimes breaks haskell code. This is mostly due to how haskell code is layed out in object files, and the stripper being a tad bit too agressive.

@angerman is that still the case?

[michaelpj]

If that's true, a question is whether it would be less painful to turn it off in the cases where it breaks.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.