ld.gold crashes or produces invalid executables when Musl is used as a libc

[albertov]

Issue description

ld.gold crashes or produces invalid executables when using the musl libc. This prevents the use of musl and GHC on armv7l (where it also manifests itself) since GHC requires ld.gold on that platform to work around https://sourceware.org/bugzilla/show_bug.cgi?id=16177

This also affects lld from LLVM 4 apparently

Steps to reproduce

nix-build -E 'let pkgs = (import <nixpkgs> {}).pkgsMusl; in pkgs.hello.overrideDerivation(_: {CFLAGS="-fuse-ld=gold";})'

The above command either fails (on a bare-metal x86_64 system) or produces an executable which segfaults at start (on a x86_64 system running on VM in a laptop). Interestingly, if one passes -static_libgcc the produced executable is apparently correct inside the VM but also fails to be created on the bare-metal machine.

Technical details

$ nix-info -m
 - system: `"x86_64-linux"`
 - host os: `Linux 4.14.78, NixOS, 18.09.1007.f8d156f44fb (Jellyfish)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.1pre6377_954d1f4d`
 - channels(root): `"nixos-18.09.1007.f8d156f44fb"`
 - channels(alberto): `"nixos-18.03pre120031.7ebacd1a43, nixpkgs-18.03pre119383.eafd703a63"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

The above system is the desktop bare-metal one

[nh2]

Just for clarity, is this the error you get when compiling on plain x86?

NIX_PATH=nixpkgs=https://github.com/NixOS/nixpkgs/archive/88ae8f7d.tar.gz nix-build -E 'let pkgs = (import <nixpkgs> {}).pkgsMusl; in pkgs.hello.overrideDerivation(_: {CFLAGS="-fuse-ld=gold";})'
these derivations will be built:
  /nix/store/c7d5hz6ynzc6gyg69w2ja8jdx11537zl-hello-2.10.drv
these paths will be fetched (0.69 MiB download, 0.69 MiB unpacked):
  /nix/store/3x7dwzq014bblazs7kq20p9hyzz0qh8g-hello-2.10.tar.gz
copying path '/nix/store/3x7dwzq014bblazs7kq20p9hyzz0qh8g-hello-2.10.tar.gz' from 'https://cache.nixos.org'...
building '/nix/store/c7d5hz6ynzc6gyg69w2ja8jdx11537zl-hello-2.10.drv'...
unpacking sources
unpacking source archive /nix/store/3x7dwzq014bblazs7kq20p9hyzz0qh8g-hello-2.10.tar.gz
source root is hello-2.10
setting SOURCE_DATE_EPOCH to timestamp 1416139241 of file hello-2.10/ChangeLog
patching sources
configuring
configure flags: --disable-dependency-tracking --prefix=/nix/store/wdl72bp6a41iiqv5ph37nwqbhx1sf825-hello-2.10
checking for a BSD-compatible install... /nix/store/bxwz6wdxr0lgj9c629azmxi35i68qrv2-coreutils-8.29/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /nix/store/bxwz6wdxr0lgj9c629azmxi35i68qrv2-coreutils-8.29/bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... configure: error: in `/tmp/nix-build-hello-2.10.drv-0/hello-2.10':
configure: error: cannot run C compiled programs.
If you meant to cross compile, use `--host'.
See `config.log' for more details
builder for '/nix/store/c7d5hz6ynzc6gyg69w2ja8jdx11537zl-hello-2.10.drv' failed with exit code 1

[albertov]

Exactly. If you see the config.log it should show that the linker has refused to link. Should also be able to reproduce in nix-shell.

On Thu, 25 Oct 2018 at 13:44, Niklas Hambüchen notifications@github.com wrote:

Just for clarity, is this the error you get when compiling on plain x86?

NIXPATH=nixpkgs=https://github.com/NixOS/nixpkgs/archive/88ae8f7d.tar.gz nix-build -E 'let pkgs = (import {}).pkgsMusl; in pkgs.hello.overrideDerivation(: {CFLAGS="-fuse-ld=gold";})' these derivations will be built: /nix/store/c7d5hz6ynzc6gyg69w2ja8jdx11537zl-hello-2.10.drv these paths will be fetched (0.69 MiB download, 0.69 MiB unpacked): /nix/store/3x7dwzq014bblazs7kq20p9hyzz0qh8g-hello-2.10.tar.gz copying path '/nix/store/3x7dwzq014bblazs7kq20p9hyzz0qh8g-hello-2.10.tar.gz' from 'https://cache.nixos.org'... building '/nix/store/c7d5hz6ynzc6gyg69w2ja8jdx11537zl-hello-2.10.drv'... unpacking sources unpacking source archive /nix/store/3x7dwzq014bblazs7kq20p9hyzz0qh8g-hello-2.10.tar.gz source root is hello-2.10 setting SOURCE_DATE_EPOCH to timestamp 1416139241 of file hello-2.10/ChangeLog patching sources configuring configure flags: --disable-dependency-tracking --prefix=/nix/store/wdl72bp6a41iiqv5ph37nwqbhx1sf825-hello-2.10 checking for a BSD-compatible install... /nix/store/bxwz6wdxr0lgj9c629azmxi35i68qrv2-coreutils-8.29/bin/install -c checking whether build environment is sane... yes checking for a thread-safe mkdir -p... /nix/store/bxwz6wdxr0lgj9c629azmxi35i68qrv2-coreutils-8.29/bin/mkdir -p checking for gawk... gawk checking whether make sets $(MAKE)... yes checking whether make supports nested variables... yes checking for gcc... gcc checking whether the C compiler works... yes checking for C compiler default output file name... a.out checking for suffix of executables... checking whether we are cross compiling... configure: error: in /tmp/nix-build-hello-2.10.drv-0/hello-2.10': configure: error: cannot run C compiled programs. If you meant to cross compile, use--host'. See `config.log' for more details builder for '/nix/store/c7d5hz6ynzc6gyg69w2ja8jdx11537zl-hello-2.10.drv' failed with exit code 1

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/NixOS/nixpkgs/issues/49071#issuecomment-433019001, or mute the thread https://github.com/notifications/unsubscribe-auth/AA00CyJD-lmz9ZrnbexYPzfSdMIAdTLNks5uoaQJgaJpZM4X4kDl .

[nh2]

I see, in config.log:

configure:4154: checking whether we are cross compiling
configure:4162: gcc -o conftest -fuse-ld=gold   conftest.c  >&5
configure:4166: $? = 0
configure:4173: ./conftest
./configure: line 4175: 14393 Segmentation fault      ./conftest$ac_cv_exeext
configure:4177: $? = 139
configure:4184: error: in `/tmp/nix-build-hello-2.10.drv-0/hello-2.10':
configure:4186: error: cannot run C compiled programs.

[albertov]

This is the same case that happens in my VM I think. If you try to compile a simple C program inside the nix-shell of hello with -static-libgcc then the produced executable should work, for some reason.

Its really curious that on my bare metal machine ld.gold just bails. I’ve seen the gcc, binutils and musl derivations are the same on both machines yet they behave differently.

Are you running it inside a VM?

On Thu, 25 Oct 2018 at 13:51, Niklas Hambüchen notifications@github.com wrote:

I see, in config.log:

configure:4154: checking whether we are cross compiling configure:4162: gcc -o conftest -fuse-ld=gold conftest.c >&5 configure:4166: $? = 0 configure:4173: ./conftest ./configure: line 4175: 14393 Segmentation fault ./conftest$ac_cv_exeext configure:4177: $? = 139 configure:4184: error: in `/tmp/nix-build-hello-2.10.drv-0/hello-2.10': configure:4186: error: cannot run C compiled programs.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/NixOS/nixpkgs/issues/49071#issuecomment-433020901, or mute the thread https://github.com/notifications/unsubscribe-auth/AA00C7t2_w5o4zz8NUSI1WomtU60MZPAks5uoaW6gaJpZM4X4kDl .

[nh2]

Are you running it inside a VM?

No, I'm running on Nix on a standard Ubuntu 16.04 64-bit.

[nh2]

Related resources:

• http://dbp-consulting.com/tutorials/debugging/linuxProgramStartup.html
• https://www.openwall.com/lists/musl/2015/05/01/3

Looking at nm output like done in that thread:

niklas@ares:/tmp/nix-build-hello-2.10.drv-0/hello-2.10/src$ ls
core  hello.c  system.h  test.c
niklas@ares:/tmp/nix-build-hello-2.10.drv-0/hello-2.10/src$ cat test.c 
int main(int argc, const char *argv[]) {
  return 0;
}
niklas@ares:/tmp/nix-build-hello-2.10.drv-0/hello-2.10/src$ gcc -o hello.bfd test.c
niklas@ares:/tmp/nix-build-hello-2.10.drv-0/hello-2.10/src$ gcc -o hello.gold -fuse-ld=gold test.c

niklas@ares ...build-hello-2.10.drv-0/hello-2.10/src % nm -D conftest-gold
0000000000402008 D __bss_start
                 w __deregister_frame_info
0000000000402008 D _edata
0000000000402070 D _end
00000000004007e2 T _fini
00000000004005b8 T _init
                 w _ITM_deregisterTMCloneTable
                 w _ITM_registerTMCloneTable
                 U __libc_start_main
                 w __register_frame_info
niklas@ares ...build-hello-2.10.drv-0/hello-2.10/src % nm -D conftest-bfd
0000000000601008 B __bss_start
0000000000601008 D _edata
0000000000601070 B _end
00000000004006e2 T _fini
00000000004004e8 T _init
                 U __libc_start_main

[dtzWill]

FWIW it seems to work with clang, which is curious:

NIX_PATH=nixpkgs=channel:nixos-unstable nix-build -E 'with (import <nixpkgs> {}).pkgsMusl; (pkgs.hello.override { inherit (llvmPackages_6) stdenv; }).overrideAttrs (o:{ CFLAGS="-fuse-ld=gold"; NIX_DEBUG=1;})'

As for using gold, may want to try something like: https://github.com/NixOS/nixpkgs/pull/33580/commits/37b2f5960c8863adc5806a840e00090e993145e8

Or otherwise ensuring our wrappers always know what linker to invoke. I don't think this is the problem, but something to consider for future gold usage once this is resolved :).

[dtzWill]

• https://www.openwall.com/lists/musl/2015/05/01/3

The thread mentions building with -fPIE, which is interesting because with NIX_DEBUG=1 I'm seeing this in the log:

HARDENING: disabled flags: pie
HARDENING: Is active (not completely disabled with "all" flag)
HARDENING: enabling fortify
HARDENING: enabling stackprotector
HARDENING: enabling strictoverflow
HARDENING: enabling format
HARDENING: enabling pic

Which may be related? It's disabled on non-musl too, FWIW.

[nh2]

Adding -pie to LDFLAGS makes it work:

NIX_PATH=nixpkgs=https://github.com/NixOS/nixpkgs/archive/88ae8f7d.tar.gz nix-build -E 'let pkgs = (import <nixpkgs> {}).pkgsMusl; in pkgs.hello.overrideDerivation(_: {CFLAGS="-fuse-ld=gold"; LDFLAGS="-pie";})'

Adding -fPIE to CFLAGS doens't help, it seems to be removed (perhaps by the hardening)?

[nh2]

Also does anybody have an idea why

NIX_PATH=nixpkgs=https://github.com/NixOS/nixpkgs/archive/88ae8f7d.tar.gz nix-build -E 'let pkgs = (import <nixpkgs> {}).pkgsMusl; in pkgs.hello.overrideDerivation(_: {CFLAGS="-fuse-ld=gold -fPIE"; hardeningDisable = [ "all" ];})'

doesn't work (using CFLAGS="-fuse-ld=gold -fPIE" and hardeningDisable = [ "all" ])?

[nh2]

seems to be removed (perhaps by the hardening)?

Just asked @fpletz at NixCon, he said that hardening should only add flags, not remove them; @cleverca22 said that happens here: https://github.com/NixOS/nixpkgs/blob/62feeed1a5aaf23b8c0883ae0cc17cd913cdce56/pkgs/build-support/cc-wrapper/add-hardening.sh

So not sure why -fPIE doesn't translate to -pie to the linker.

[nh2]

So not sure why -fPIE doesn't translate to -pie to the linker.

With help of @cleverca22 and looking at https://github.com/NixOS/nixpkgs/blob/62feeed1a5aaf23b8c0883ae0cc17cd913cdce56/pkgs/build-support/cc-wrapper/add-hardening.sh#L47-L52

It seems you need to pass both -pie and -fPIE to the compiler; see man gcc:

       -pie
           Produce a position independent executable on targets that support it.
           For predictable results, you must also specify the same set of
           options used for compilation (-fpie, -fPIE, or model suboptions) when
           you specify this linker option.
...
       -fpie
       -fPIE
           These options are similar to -fpic and -fPIC, but generated position
           independent code can be only linked into executables.  Usually these
           options are used when -pie GCC option is used during linking.

           -fpie and -fPIE both define the macros "__pie__" and "__PIE__".  The
           macros have the value 1 for -fpie and 2 for -fPIE.

And indeed

NIX_PATH=nixpkgs=https://github.com/NixOS/nixpkgs/archive/88ae8f7d.tar.gz nix-build -E 'let pkgs = (import <nixpkgs> {}).pkgsMusl; in pkgs.hello.overrideDerivation(_: {CFLAGS="-fuse-ld=gold -fPIE -pie";})'

works.

[nh2]

I'd expect that hardeningEnable = ["pie"]; should also work, but it doesn't:

NIX_PATH=nixpkgs=https://github.com/NixOS/nixpkgs/archive/88ae8f7d.tar.gz nix-build -E 'let pkgs = (import <nixpkgs> {}).pkgsMusl; in pkgs.hello.overrideDerivation(_: {CFLAGS="-fuse-ld=gold"; hardeningEnable = ["pie"]; })'

Contrary to expectations, this doesn't seem to add the pie flags.

[dtzWill]

It works if you use overrideAttrs instead! \o/

On Thu, 25 Oct 2018 07:30:03 -0700, Niklas Hambüchen notifications@github.com wrote:

I'd expect that hardeningEnable = ["pie"]; should also work, but it doesn't:

NIX_PATH=nixpkgs=https://github.com/NixOS/nixpkgs/archive/88ae8f7d.tar.gz nix-build -E 'let pkgs = (import <nixpkgs> {}).pkgsMusl; in pkgs.hello.overrideDerivation(_: {CFLAGS="-fuse-ld=gold"; hardeningEnable = ["pie"]; })'

Contrary to expectations, this doesn't seem to add the pie flags.

-- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/NixOS/nixpkgs/issues/49071#issuecomment-433073860Non-text part: text/html

[nh2]

It works if you use overrideAttrs instead! \o/

Oh right, this works!

NIX_PATH=nixpkgs=https://github.com/NixOS/nixpkgs/archive/88ae8f7d.tar.gz nix-build -E 'let pkgs = (import <nixpkgs> {}).pkgsMusl; in pkgs.hello.overrideAttrs(_: {CFLAGS="-fuse-ld=gold"; hardeningEnable = ["pie"]; })'                  

@dtzWill Should we add something to the musl bits that detects if gold is used and passes pie accordingly?

Also, I assume there should be some upstream gold bug filed somewhere? The musl thread didn't link any being filed.

[albertov]

The command which worked for @nh2 :

NIX_PATH=nixpkgs=https://github.com/NixOS/nixpkgs/archive/88ae8f7d.tar.gz nix-build -E 'let pkgs = (import <nixpkgs> {}).pkgsMusl; in pkgs.hello.overrideDerivation(_: {CFLAGS="-fuse-ld=gold -fPIE -pie";})'

Failed in my machine :( Same error as always:

checking whether the C compiler works... no
configure: error: in `/build/hello-2.10':
configure: error: C compiler cannot create executables
See `config.log' for more details
builder for '/nix/store/g6m7i019af8p838ldzv4jlfxds40fw2g-hello-2.10.drv' failed with exit code 77
error: build of '/nix/store/g6m7i019af8p838ldzv4jlfxds40fw2g-hello-2.10.drv' failed

[dtzWill]

How about using overrideAttrs instead? Otherwise it didn't work for me either.

Thisis probably due to the same sorts of things that make overrideAttrs generally what you should use: https://nixos.org/nixpkgs/manual/#sec-pkg-overrideAttrs (and warning at top of following blurb about overrideDerivation).

Unfortunately I personally remember this is true for one of them, but never which one is which. Luckily muscle memory usually gets it right as long as I don't think about it too much haha.

On Thu, Oct 25, 2018, 12:05 PM Alberto Valverde notifications@github.com wrote:

The command which worked for @nh2 https://github.com/nh2 :

NIXPATH=nixpkgs=https://github.com/NixOS/nixpkgs/archive/88ae8f7d.tar.gz nix-build -E 'let pkgs = (import {}).pkgsMusl; in pkgs.hello.overrideDerivation(: {CFLAGS="-fuse-ld=gold -fPIE -pie";})'

Failed in my machine :( Same error as always:

checking whether the C compiler works... no configure: error: in /build/hello-2.10': configure: error: C compiler cannot create executables Seeconfig.log' for more details builder for '/nix/store/g6m7i019af8p838ldzv4jlfxds40fw2g-hello-2.10.drv' failed with exit code 77 error: build of '/nix/store/g6m7i019af8p838ldzv4jlfxds40fw2g-hello-2.10.drv' failed

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/NixOS/nixpkgs/issues/49071#issuecomment-433130597, or mute the thread https://github.com/notifications/unsubscribe-auth/AAx4si-ezr7zcBqzmca4l5NjTCIlH7jiks5uoe9wgaJpZM4X4kDl .

[albertov]

Didn't know about overrideAttrs (thanks for letting me know!). Anyway, I get the same error with overrideAttrs too, this is what I'm running and the output:

3$ NIX_PATH=nixpkgs=https://github.com/NixOS/nixpkgs/archive/88ae8f7d.tar.gz nix-build -E 'let pkgs = (import <nixpkgs> {}).pkgsMusl; in pkgs.hello.overrideAttrs(_: {CFLAGS="-fuse-ld=gold -fPIE -pie";})'
these derivations will be built:
  /nix/store/g6m7i019af8p838ldzv4jlfxds40fw2g-hello-2.10.drv
building '/nix/store/g6m7i019af8p838ldzv4jlfxds40fw2g-hello-2.10.drv'...
unpacking sources
unpacking source archive /nix/store/3x7dwzq014bblazs7kq20p9hyzz0qh8g-hello-2.10.tar.gz
source root is hello-2.10
setting SOURCE_DATE_EPOCH to timestamp 1416139241 of file hello-2.10/ChangeLog
patching sources
configuring
configure flags: --disable-dependency-tracking --prefix=/nix/store/rmwms3g2v6isnswxlm1w799vb82zfp24-hello-2.10
checking for a BSD-compatible install... /nix/store/bxwz6wdxr0lgj9c629azmxi35i68qrv2-coreutils-8.29/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /nix/store/bxwz6wdxr0lgj9c629azmxi35i68qrv2-coreutils-8.29/bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking for gcc... gcc
checking whether the C compiler works... no
configure: error: in `/build/hello-2.10':
configure: error: C compiler cannot create executables
See `config.log' for more details
builder for '/nix/store/g6m7i019af8p838ldzv4jlfxds40fw2g-hello-2.10.drv' failed with exit code 77
error: build of '/nix/store/g6m7i019af8p838ldzv4jlfxds40fw2g-hello-2.10.drv' failed

The other command which worked for you, with hardeningEnable, also fails:

$ NIX_PATH=nixpkgs=https://github.com/NixOS/nixpkgs/archive/88ae8f7d.tar.gz nix-build -E 'let pkgs = (import <nixpkgs> {}).pkgsMusl; in pkgs.hello.overrideAttrs(_: {CFLAGS="-fuse-ld=gold"; hardeningEnable = ["pie"]; })'                  
these derivations will be built:
  /nix/store/qllg7c478rnba15rzh73va3adlw88zhk-hello-2.10.drv
building '/nix/store/qllg7c478rnba15rzh73va3adlw88zhk-hello-2.10.drv'...
unpacking sources
unpacking source archive /nix/store/3x7dwzq014bblazs7kq20p9hyzz0qh8g-hello-2.10.tar.gz
source root is hello-2.10
setting SOURCE_DATE_EPOCH to timestamp 1416139241 of file hello-2.10/ChangeLog
patching sources
configuring
configure flags: --disable-dependency-tracking --prefix=/nix/store/2k3lazi3vkm73gm033dxnlc8svasnzid-hello-2.10
checking for a BSD-compatible install... /nix/store/bxwz6wdxr0lgj9c629azmxi35i68qrv2-coreutils-8.29/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /nix/store/bxwz6wdxr0lgj9c629azmxi35i68qrv2-coreutils-8.29/bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking for gcc... gcc
checking whether the C compiler works... no
configure: error: in `/build/hello-2.10':
configure: error: C compiler cannot create executables
See `config.log' for more details
builder for '/nix/store/qllg7c478rnba15rzh73va3adlw88zhk-hello-2.10.drv' failed with exit code 77
error: build of '/nix/store/qllg7c478rnba15rzh73va3adlw88zhk-hello-2.10.drv' failed

My computer must be cursed :(

[nh2]

Is this now also a plain x86_64 machine or something else?

[albertov]

Yes, it's a plain x86_64 machine running Nixos on bare hardware.

[albertov]

For the record, inside a nixos VM in a laptop, passing hardeningEnable=["pie"] makes it build correctly and the executable works. Without hardeningEnable (the original command from the description) the executable segfaults.

I can't understand the discrepancies between machines, both are x86_64 systems. The laptop is a macbook pro while the desktop has this processor according to /proc/cpuinfo:

processor       : 15
vendor_id       : GenuineIntel
cpu family      : 6
model           : 79
model name      : Intel(R) Core(TM) i7-6900K CPU @ 3.20GHz
stepping        : 1
microcode       : 0xb00001c
cpu MHz         : 3499.257
cache size      : 20480 KB
physical id     : 0
siblings        : 16
core id         : 7
cpu cores       : 8
apicid          : 15
initial apicid  : 15
fpu             : yes
fpu_exception   : yes
cpuid level     : 20
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a rdseed adx smap intel_pt xsaveopt cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf
bogomips        : 6398.64
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:

I'll try to run it on other machines today to see if I can spot a pattern

[albertov]

This is /proc/cpuinfo from the laptop:

processor   : 3
vendor_id   : GenuineIntel
cpu family  : 6
model       : 70
model name  : Intel(R) Core(TM) i7-4770HQ CPU @ 2.20GHz
stepping    : 1
microcode   : 0x19
cpu MHz     : 2194.268
cache size  : 6144 KB
physical id : 6
siblings    : 1
core id     : 0
cpu cores   : 1
apicid      : 6
initial apicid  : 6
fpu     : yes
fpu_exception   : yes
cpuid level : 13
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc cpuid aperfmperf pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm cpuid_fault epb invpcid_single pti tpr_shadow vnmi ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 invpcid xsaveopt dtherm ida arat pln pts
bugs        : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf
bogomips    : 4388.53
clflush size    : 64
cache_alignment : 64
address sizes   : 42 bits physical, 48 bits virtual
power management:

[nh2]

That is pretty odd. Perhaps comparing the full config.log output gives a hint (ideally also with NIX_DEBUG=1 set)?

[albertov]

@nh2 I will try that and post the diff if different. I meanwhile tried to compile using clang 6 with:

NIX_PATH=nixpkgs=https://github.com/NixOS/nixpkgs/archive/88ae8f7d.tar.gz nix-build -E 'with (import <nixpkgs> {}).pkgsMusl; (pkgs.hello.override { inherit (llvmPackages_6) stdenv; }).overrideAttrs (o:{ CFLAGS="-fuse-ld=gold"; NIX_DEBUG=1;})'

but llvm fails to build since the gold tests fail with a very similar error:

<....snip....>
Command Output (stderr):
--
/nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold: fatal error: /build/llvm/build/test/tools/gold/X86/Output/visibility.ll.tmp.so: No error information

--

********************
Testing: 0 .. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90..
Testing Time: 165.74s
********************
Failing Tests (34):
    LLVM :: tools/gold/PowerPC/mtriple.ll
    LLVM :: tools/gold/X86/asm_undefined.ll
    LLVM :: tools/gold/X86/asm_undefined2.ll
    LLVM :: tools/gold/X86/bcsection.ll
    LLVM :: tools/gold/X86/cache.ll
    LLVM :: tools/gold/X86/comdat.ll
    LLVM :: tools/gold/X86/common_thinlto.ll
    LLVM :: tools/gold/X86/emit-llvm.ll
    LLVM :: tools/gold/X86/global_with_section.ll
    LLVM :: tools/gold/X86/mixed_lto.ll
    LLVM :: tools/gold/X86/module_asm.ll
    LLVM :: tools/gold/X86/multiple-data.s
    LLVM :: tools/gold/X86/multiple-sections.ll
    LLVM :: tools/gold/X86/opt-level.ll
    LLVM :: tools/gold/X86/parallel.ll
    LLVM :: tools/gold/X86/pr19901.ll
    LLVM :: tools/gold/X86/pr19901_thinlto.ll
    LLVM :: tools/gold/X86/pr25907.ll
    LLVM :: tools/gold/X86/relax-relocs.ll
    LLVM :: tools/gold/X86/relocatable.ll
    LLVM :: tools/gold/X86/relocation-model-pic.ll
    LLVM :: tools/gold/X86/slp-vectorize.ll
    LLVM :: tools/gold/X86/strip_names.ll
1 warning(s) in tests.

    LLVM :: tools/gold/X86/thinlto.ll
    LLVM :: tools/gold/X86/thinlto_afdo.ll
    LLVM :: tools/gold/X86/thinlto_archive.ll
    LLVM :: tools/gold/X86/thinlto_funcimport.ll
    LLVM :: tools/gold/X86/thinlto_internalize.ll
    LLVM :: tools/gold/X86/thinlto_linkonceresolution.ll
    LLVM :: tools/gold/X86/thinlto_weak_library.ll
    LLVM :: tools/gold/X86/thinlto_weak_resolution.ll
    LLVM :: tools/gold/X86/type-merge2.ll
    LLVM :: tools/gold/X86/vectorize.ll
    LLVM :: tools/gold/X86/visibility.ll

  Expected Passes    : 22394
  Expected Failures  : 140
  Unsupported Tests  : 721
  Unexpected Failures: 34
make[3]: *** [CMakeFiles/check-all.dir/build.make:58: CMakeFiles/check-all] Error 1
make[3]: Leaving directory '/build/llvm/build'
make[2]: *** [CMakeFiles/Makefile2:356: CMakeFiles/check-all.dir/all] Error 2
make[2]: Leaving directory '/build/llvm/build'
make[1]: *** [CMakeFiles/Makefile2:363: CMakeFiles/check-all.dir/rule] Error 2
make[1]: Leaving directory '/build/llvm/build'
make: *** [Makefile:251: check-all] Error 2
builder for '/nix/store/qsgcngh2m66k188d1nmqsmdy8z3j312i-llvm-6.0.1.drv' failed with exit code 2

[albertov]

I've compared the config.logs of the following command on both machines:

NIX_PATH=nixpkgs=https://github.com/NixOS/nixpkgs/archive/88ae8f7d.tar.gz nix-build -E 'let pkgs = (import <nixpkgs> {}).pkgsMusl; in pkgs.hello.overrideAttrs(_: {NIX_DEBUG="1"; CFLAGS="-fuse-ld=gold -Wl,--debug=all"; hardeningEnable = ["pie"]; postInstall="exit -1";})' -K

(the postIntall is to intentionally kill the build before it finishes correctly to force the temporary build dir to be preserved in the machine where it works properly)

• Bad config.log
• Good config.log

If I understand it correctly, the "broken" ld.gold has some problem processing libgcc.a since when both logs begin to differ significantly is where the good version starts running task Add_archive_symbols while the bad version never reached that phase.

The beginning of the diff looks like this:

--- hello-config-bad.log    2018-10-28 21:27:33.391979234 +0100
+++ hello-config-good.log 2018-10-28 21:22:29.845499594 +0100
@@ -12,9 +12,9 @@

 hostname = localhost
 uname -m = x86_64
-uname -r = 4.14.78
+uname -r = 4.14.76
 uname -s = Linux
-uname -v = #1-NixOS SMP Sat Oct 20 07:48:54 UTC 2018
+uname -v = #1-NixOS SMP Sat Oct 13 07:27:30 UTC 2018

 /usr/bin/uname -p = unknown
 /bin/uname -X     = unknown
@@ -188,7 +188,7 @@
   -plugin
   /nix/store/4k79vw3p32n2w7z11d9rh4788chq73ja-gcc-7.3.0/libexec/gcc/x86_64-unknown-linux-musl/7.3.0/liblto_plugin.so
   -plugin-opt=/nix/store/4k79vw3p32n2w7z11d9rh4788chq73ja-gcc-7.3.0/libexec/gcc/x86_64-unknown-linux-musl/7.3.0/lto-wrapper
-  -plugin-opt=-fresolution=/build/cciodIpK.res
+  -plugin-opt=-fresolution=/build/ccbNLFDA.res
   -plugin-opt=-pass-through=-lgcc
   -plugin-opt=-pass-through=-lgcc_s
   -plugin-opt=-pass-through=-lc
@@ -214,7 +214,7 @@
   -dynamic-linker
   /nix/store/35mb6imifzx9316y7nknhh6apx1k7a24-musl-1.1.19/lib/ld-musl-x86_64.so.1
   --debug=all
-  /build/cceIfKHc.o
+  /build/ccnaDnPl.o
   -rpath
   /nix/store/xsngrd6w0r2aaqc3fdzyi16a4hpndwa2-hello-2.10/lib64
   -rpath
@@ -279,14 +279,14 @@
 /nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold:   0 completed task Read_symbols /nix/store/4k79vw3p32n2w7z11d9rh4788chq73ja-gcc-7.3.0/lib/gcc/x86_64-unknown-linux-musl/7.3.0/crtbeginS.o (user: 0.000000 sys: 0.000000 wall: 0.000000)
 /nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold:   0 running   task Add_symbols /nix/store/4k79vw3p32n2w7z11d9rh4788chq73ja-gcc-7.3.0/lib/gcc/x86_64-unknown-linux-musl/7.3.0/crtbeginS.o
 /nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold:   0 completed task Add_symbols /nix/store/4k79vw3p32n2w7z11d9rh4788chq73ja-gcc-7.3.0/lib/gcc/x86_64-unknown-linux-musl/7.3.0/crtbeginS.o (user: 0.000000 sys: 0.000000 wall: 0.000000)
-/nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold:   0 running   task Read_symbols /build/cceIfKHc.o
-/nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold: Opened new descriptor 9 for "/build/cceIfKHc.o"
-/nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold: Attempt to open /build/cceIfKHc.o succeeded
-/nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold: Unlocking file "/build/cceIfKHc.o"
-/nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold: Released descriptor 9 for "/build/cceIfKHc.o"
-/nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold:   0 completed task Read_symbols /build/cceIfKHc.o (user: 0.000000 sys: 0.000000 wall: 0.000000)
-/nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold:   0 running   task Add_symbols /build/cceIfKHc.o
-/nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold:   0 completed task Add_symbols /build/cceIfKHc.o (user: 0.000000 sys: 0.000000 wall: 0.000000)
+/nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold:   0 running   task Read_symbols /build/ccnaDnPl.o
+/nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold: Opened new descriptor 9 for "/build/ccnaDnPl.o"
+/nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold: Attempt to open /build/ccnaDnPl.o succeeded
+/nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold: Unlocking file "/build/ccnaDnPl.o"
+/nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold: Released descriptor 9 for "/build/ccnaDnPl.o"
+/nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold:   0 completed task Read_symbols /build/ccnaDnPl.o (user: 0.000000 sys: 0.000000 wall: 0.000000)
+/nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold:   0 running   task Add_symbols /build/ccnaDnPl.o
+/nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold:   0 completed task Add_symbols /build/ccnaDnPl.o (user: 0.000000 sys: 0.000000 wall: 0.000000)
 /nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold:   0 running   task Read_symbols /nix/store/4k79vw3p32n2w7z11d9rh4788chq73ja-gcc-7.3.0/lib/gcc/x86_64-unknown-linux-musl/7.3.0/crtendS.o
 /nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold: Opened new descriptor 10 for "/nix/store/4k79vw3p32n2w7z11d9rh4788chq73ja-gcc-7.3.0/lib/gcc/x86_64-unknown-linux-musl/7.3.0/crtendS.o"
 /nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold: Attempt to open /nix/store/4k79vw3p32n2w7z11d9rh4788chq73ja-gcc-7.3.0/lib/gcc/x86_64-unknown-linux-musl/7.3.0/crtendS.o succeeded
@@ -388,7 +388,25677 @@
 /nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold: Attempt to open /nix/store/4k79vw3p32n2w7z11d9rh4788chq73ja-gcc-7.3.0/lib/gcc/x86_64-unknown-linux-musl/7.3.0/libgcc.a succeeded
 /nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold: Unlocking file "/nix/store/4k79vw3p32n2w7z11d9rh4788chq73ja-gcc-7.3.0/lib/gcc/x86_64-unknown-linux-musl/7.3.0/libgcc.a"
 /nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold: Released descriptor 11 for "/nix/store/4k79vw3p32n2w7z11d9rh4788chq73ja-gcc-7.3.0/lib/gcc/x86_64-unknown-linux-musl/7.3.0/libgcc.a"
-/nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold:   0 completed task Read_symbols -lgcc (user: 0.000000 sys: 0.010000 wall: 0.000000)
+/nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold:   0 completed task Read_symbols -lgcc (user: 0.000000 sys: 0.000000 wall: 0.000000)
+/nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold:   0 running   task Add_archive_symbols /nix/store/4k79vw3p32n2w7z11d9rh4788chq73ja-gcc-7.3.0/lib/gcc/x86_64-unknown-linux-musl/7.3.0/libgcc.a
+/nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold:   0 completed task Add_archive_symbols /nix/store/4k79vw3p32n2w7z11d9rh4788chq73ja-gcc-7.3.0/lib/gcc/x86_64-unknown-linux-musl/7.3.0/libgcc.a (user: 0.000000 sys: 0.000000 wall: 0.000000)
+/nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold:   0 running   task Read_symbols -lgcc_s
+/nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold: Attempt to open /nix/store/35mb6imifzx9316y7nknhh6apx1k7a24-musl-1.1.19/lib/libgcc_s.so failed
+/nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold: Attempt to open /nix/store/35mb6imifzx9316y7nknhh6apx1k7a24-musl-1.1.19/lib/libgcc_s.a failed
+/nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold: Closed descriptor 11 for "/nix/store/4k79vw3p32n2w7z11d9rh4788chq73ja-gcc-7.3.0/lib/gcc/x86_64-unknown-linux-musl/7.3.0/libgcc.a"
+/nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold: Opened new descriptor 12 for "/nix/store/4dw6k7sdkw8r2b5rgka8r659451l3xx9-gcc-7.3.0-lib/lib/libgcc_s.so"
+/nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold: Attempt to open /nix/store/4dw6k7sdkw8r2b5rgka8r659451l3xx9-gcc-7.3.0-lib/lib/libgcc_s.so succeeded

[matthewbauer]

"pie" hardening is not enabled by default. Maybe it should be?

[nh2]

"pie" hardening is not enabled by default. Maybe it should be?

@matthewbauer The way I read the links is that gold for unknown reasons (likely a bug) doesn't work with musl unless pie is activated.

Perhaps it should be enabled by default for this combination (gold + musl)?

[matthewbauer]

Yeah sounds like a good idea. We don't know whether you will use ld.gold ahead of time, so better to just do it with musl everywhere.

[nh2]

I've filed an upstream issue for gold at https://sourceware.org/bugzilla/show_bug.cgi?id=23856

I recommend everyone involved subscribing to it.