pkgsStatic.llvmPackages.libcxxStdenv does not produce static c++ binaries

[tfc]

Describe the bug

Using pkgs.pkgsStatic, it is easily possible to compile static c++ binaries using -static. The same does not seem to work if stdenv = pkgsStatic.llvmPackages.libcxxStdenv;.

To Reproduce

Minimal working example:

# static.nix
let
  mainCpp = builtins.toFile "main.cpp" ''
    #include <iostream>
    int main() { std::cout << "hello\n"; }
  '';

  deriv = { stdenv }:
    stdenv.mkDerivation {
      name = "hello-app";
      src = mainCpp;
      unpackPhase = ":";
      buildPhase = "$CXX $src -o hello -static";
      installPhase = "mkdir -p $out/bin && install -m755 hello $out/bin/hello";
    };
  overlay = self: super: {
    hello-app = self.callPackage deriv {};
  };
  nixpkgs = builtins.fetchTarball {
    url = "https://github.com/nixos/nixpkgs/archive/f217c0ea7c148ddc0103347051555c7c252dcafb.tar.gz";
    sha256 = "0cyksxg2lnzxd0pss09rmmk2c2axz0lf9wvgvfng59nwf8dpq2kf";
  };
  pkgs = (import nixpkgs { overlays = [ overlay ]; }).pkgsStatic;
in
{
  hello-static-gcc = pkgs.hello-app;
  hello-static-clang = pkgs.hello-app.override {
    stdenv = pkgs.llvmPackages.libcxxStdenv;
  };
}

the gcc derivation works well:

$ ldd $(nix-build static.nix -A hello-static-gcc)/bin/hello
    not a dynamic executable

With clang, the following happens:

$ nix-build static.nix -A hello-static-clang
# ... lots of output omitted ...
(.text+0x2b4): undefined reference to `__cxa_guard_acquire'
/nix/store/2v3vkva29310005gipcqhdw9hxr57j6l-x86_64-unknown-linux-musl-binutils-2.35.1/bin/x86_64-unknown-linux-musl-ld: (.text+0x2f2): undefined reference to `__cxa_guard_release'
/nix/store/2v3vkva29310005gipcqhdw9hxr57j6l-x86_64-unknown-linux-musl-binutils-2.35.1/bin/x86_64-unknown-linux-musl-ld: /nix/store/s1rdpf0z4l4d0m5rsysjl0fna2ayf3db-libc++-7.1.0-x86_64-unknown-linux-musl/lib/libc++.a(future.cpp.o): in function `std::__1::future_error::~future_error()':
(.text+0x2f): undefined reference to `std::logic_error::~logic_error()'
/nix/store/2v3vkva29310005gipcqhdw9hxr57j6l-x86_64-unknown-linux-musl-binutils-2.35.1/bin/x86_64-unknown-linux-musl-ld: /nix/store/s1rdpf0z4l4d0m5rsysjl0fna2ayf3db-libc++-7.1.0-x86_64-unknown-linux-musl/lib/libc++.a(future.cpp.o):(.data.rel.ro._ZTINSt3__112future_errorE[_ZTINSt3__112future_errorE]+0x10): undefined reference to `typeinfo for std::logic_error'
/nix/store/2v3vkva29310005gipcqhdw9hxr57j6l-x86_64-unknown-linux-musl-binutils-2.35.1/bin/x86_64-unknown-linux-musl-ld: /nix/store/s1rdpf0z4l4d0m5rsysjl0fna2ayf3db-libc++-7.1.0-x86_64-unknown-linux-musl/lib/libc++.a(future.cpp.o):(.data.rel.ro._ZTVNSt3__112future_errorE[_ZTVNSt3__112future_errorE]+0x20): undefined reference to `std::logic_error::what() const'
clang-7: error: linker command failed with exit code 1 (use -v to see invocation)
builder for '/nix/store/h89wg44nxqpf5fxbzkq4lykv4l19rlc3-hello-app-x86_64-unknown-linux-musl.drv' failed with exit code 1
error: build of '/nix/store/h89wg44nxqpf5fxbzkq4lykv4l19rlc3-hello-app-x86_64-unknown-linux-musl.drv' failed

Looking at pkgs/development/compilers/llvm/11/libc++/default.nix and pkgs/development/compilers/llvm/11/libc++abi.nix i see that there is some enableShared parametrization which hints at static binaries being possible, but from looking at the code i don't know if that is already supposed to work out of the box or still needs some tweaking.

Expected behavior

Using clang with libcxx on c++ programs with -static should result in successful compilation of static binaries.

Notify maintainers @vlstill @lovek323 @dtzWill @primeos

Metadata Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

 - system: `"x86_64-linux"`
 - host os: `Linux 5.9.16, NixOS, 20.09.2750.85abeab48b5 (Nightingale)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.3.10`
 - channels(root): `"nixos-20.09.2750.85abeab48b5, nixos-unstable-21.03pre261422.1a57d96edd1"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs

[ehmry]

@dtzWill Is there an LLVM version where this is known to work?

[sternenseemann]

pkgsStatic.libcxxStdenv (as opposed to pkgsStatic.llvmPackages.libcxxStdenv should work). Unfortunately the static adapters don't get applied to all llvm stdenvs atm.

118313 should fix this for the most obvious cases (can you check your reproducer against that PR?).

[stale[bot]]

I marked this as stale due to inactivity. → More info

[pwaller]

cc @Ericson2314 @sternenseemann @Mic92 I think I've found a reason why static+llvm stdenvs are broken.

Another issue here is that;

1) The makeStaticBinaries stdenv (used to implement pkgsStatic) passes -static to the linker by setting NIX_CFLAGS_LINK:

https://github.com/NixOS/nixpkgs/blob/b8960cd250e06e5f7f3f874815f609a1f7d0a1dd/pkgs/stdenv/adapters.nix#L64

2) GCC's compiler driver or the linker ignores -Wl,-dynamic-linker=... if you pass -static.

$ nix run nixpkgs#file $(nix build --print-out-paths nixpkgs#pkgsStatic.hello^out)/bin/hello
/nix/store/g6ay1bbrqcli3w51c9ghndw0j1f740ds-hello-static-x86_64-unknown-linux-musl-2.12.1/bin/hello:
ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, not stripped

:point_up: This one, built with the gcc stdenv, does not have a dynamic linker set.

2a) LLVM sets a dynamic linker anyway even if you pass -static:

$ nix build --keep-failed nixpkgs/0d8145a5d81ebf6698077b21042380a3a66a11c7#pkgsStatic.pkgsLLVM.hello

:point_up: This fails during the check phase, because the produced executable segfault on startup.

:point_down: The reason they segfault is because the dynamic linker is running on the (static) binary, which does not work.

nix run nixpkgs#file /tmp/nix-build-hello-static-x86_64-unknown-linux-musl-*/hello-*/hello
/tmp/nix-build-hello-static-x86_64-unknown-linux-musl-2.12.1.drv-0/hello-2.12.1/hello: \
ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, \
interpreter /nix/store/8lqwl0x6194hq7q862w8jn2wdxfb97fl-musl-static-x86_64-unknown-linux-musl-1.2.3/lib/ld-musl-x86_64.so.1, \
with debug_info, not stripped

3) -Wl,-dynamic-linker, inserted by cc-wrapper, is intended to be conditionalised on the $linkType.

https://github.com/NixOS/nixpkgs/blob/b8960cd250e06e5f7f3f874815f609a1f7d0a1dd/pkgs/build-support/cc-wrapper/cc-wrapper.sh#L176-L177

4) The $linkType is inferred from the compiler wrappers arguments, which does not take $NIX_CFLAGS_LINK into account, resulting in $linkType == dynamic for pkgsStatic derivations.

https://github.com/NixOS/nixpkgs/blob/a1bcfbc198add548c2ed42e8f7beb2fac08c0c7f/pkgs/build-support/wrapper-common/utils.bash#L133-L144

https://github.com/NixOS/nixpkgs/blob/b8960cd250e06e5f7f3f874815f609a1f7d0a1dd/pkgs/build-support/cc-wrapper/cc-wrapper.sh#L34

I think if this is fixed more things will work. I wonder if GCC and/or LLVM should refuse the combination of -Wl,-dynamic-linker=... -static in principle.

[Artturin]

@pwaller was this fixed by https://github.com/NixOS/nixpkgs/pull/253116 ?

[pwaller]

@pwaller was this fixed by #253116 ?

@Artturin I think it should be. I just tried to build the example but I hit https://github.com/NixOS/nixpkgs/issues/177129 in the build of libcxxabi-static-x86_64-unknown-linux-musl, during the configure step to check the compiler is working.

That is, that clang specifies gcc_eh to link, when it shouldn't, but this doesn't exist in a static-target gcc build. This is a clang issue, but clang doesn't have a way to know it shouldn't try to link gcc_eh. A workaround is to create an empty gcc_eh.

For libcxxabi, it could be worked around via CMAKE_C_COMPILER_WORKS, but I think a more general solution is needed. Perhaps to provide an empty gcc_eh in static builds so that clang's linker driver can function.

[pwaller]

There is an additional problem. -lc++abi is appearing on the link line before -lc++ which is the root cause of the original messages in this thread. Additionally, -lc needs to appear after both of these. With those changes, I get a binary I can run. nix-support/libcxx-ldflags contains: -stdlib=libc++ -lc++abi. These are the flags clang++ is passing to the linker:

"-lc++abi"
"-lc++"
"-lm"
"--start-group"
"-lgcc"
"-lgcc_eh"
"-lc"
"--end-group"

So as a workaround, I can get a functioning binary by compiling the original example with $CXX $src -o hello -static -lc++.