search

NixOS / nixpkgs

Nix Packages collection & NixOS
MIT License
18.19k stars 14.2k forks source link

Can't build linux-headers on armv6l #107386

Closed Gaelan closed 3 years ago

Gaelan commented 3 years ago

Describe the bug stdenv.__bootPackages.linuxHeaders (and, therefore, pretty much any package) fails to build on armv6l.

To Reproduce Steps to reproduce the behavior:

  1. Get an armv6l machine. (I actually used an aarch64 Raspberry Pi 4B.)
  2. cd path/to/nixpkgs; nix-build . --arg system '"armv6l-linux"' -A stdenv.__bootPackages.linuxHeaders
these derivations will be built:
  /nix/store/6s0a9wyc446qpsn2svia7c9sc91rxbsr-linux-headers-5.9.8.drv
building '/nix/store/6s0a9wyc446qpsn2svia7c9sc91rxbsr-linux-headers-5.9.8.drv' on 'ssh://gaelan@pi4b-00.local'...
unpacking sources
unpacking source archive /nix/store/cai15i0m74ihfajdqfz56xgayc9bf9m3-linux-5.9.8.tar.xz
source root is linux-5.9.8
setting SOURCE_DATE_EPOCH to timestamp 1605039377 of file linux-5.9.8/virt/lib/irqbypass.c
patching sources
applying patch /nix/store/69lyjyca86317hdsc1rwf3ahn7s5kiyn-no-relocs.patch
patching file arch/x86/Makefile
Hunk #1 succeeded at 212 (offset -19 lines).
configuring
no configure script, doing nothing
building
  HOSTCC  scripts/basic/fixdep
bash: line 1:  4020 Segmentation fault      (core dumped) scripts/basic/fixdep scripts/basic/.fixdep.d scripts/basic/fixdep 'gcc -Wp,-MMD,scripts/basic/.fixdep.d -Wall -Wmissing-prototypes -Wstrict-prototypes -O2 -fomit-frame-pointer -std=gnu89         -o scripts/basic/fixdep scripts/basic/fixdep.c   ' > scripts/basic/.fixdep.cmd
make[1]: *** [scripts/Makefile.host:95: scripts/basic/fixdep] Error 139
make[1]: *** Deleting file 'scripts/basic/fixdep'
make: *** [Makefile:539: scripts_basic] Error 2
error: build of '/nix/store/6s0a9wyc446qpsn2svia7c9sc91rxbsr-linux-headers-5.9.8.drv' on 'ssh://gaelan@pi4b-00.local' failed: error: --- Error --- nix-daemon
builder for '/nix/store/6s0a9wyc446qpsn2svia7c9sc91rxbsr-linux-headers-5.9.8.drv' failed with exit code 2; last 10 log lines:
  patching file arch/x86/Makefile
  Hunk #1 succeeded at 212 (offset -19 lines).
  configuring
  no configure script, doing nothing
  building
    HOSTCC  scripts/basic/fixdep
  bash: line 1:  4020 Segmentation fault      (core dumped) scripts/basic/fixdep scripts/basic/.fixdep.d scripts/basic/fixdep 'gcc -Wp,-MMD,scripts/basic/.fixdep.d -Wall -Wmissing-prototypes -Wstrict-prototypes -O2 -fomit-frame-pointer -std=gnu89         -o scripts/basic/fixdep scripts/basic/fixdep.c   ' > scripts/basic/.fixdep.cmd
  make[1]: *** [scripts/Makefile.host:95: scripts/basic/fixdep] Error 139
  make[1]: *** Deleting file 'scripts/basic/fixdep'
  make: *** [Makefile:539: scripts_basic] Error 2
builder for '/nix/store/6s0a9wyc446qpsn2svia7c9sc91rxbsr-linux-headers-5.9.8.drv' failed with exit code 1
error: build of '/nix/store/6s0a9wyc446qpsn2svia7c9sc91rxbsr-linux-headers-5.9.8.drv' failed

Additional context

The segfault comes from fixdep, which is invoked by the Linux kernel build system.

Here's a minimal C program that segfaults, when compiled with the cc from nix-shell . --arg system '"armv6l-linux"' -A stdenv.__bootPackages.linuxHeaders:

#include <stdio.h>
int main(int argc, char** argv)
{
    fprintf(stderr, "Usage: fixdep <depfile> <target> <cmdline>\n");
    return 0;
}

A git bisect reveals that the faulty commit is b3640e024f01453b3c4f720135dc6cff529da8ab (#103183).

Notify maintainers

@Ericson2314, maintainer of binutils (since that's what seems to be the culprit here) @expipiplus1, author of #103183

Metadata Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute:
# a list of nixos modules affected by the problem
module:
expipiplus1 commented 3 years ago

I guess that this patch should only be enabled on armv7l then. Truthfully that PR was just changes proposed/written by others and someone else is probably more qualified wrt binutils to say what the correct thing to do would be.

Gaelan commented 3 years ago

I don't think it's an armv6l vs. armv7l thing: nix-build . --arg system '"armv7l-linux"' -A stdenv.__bootPackages.linuxHeaders fails in the same way.

There's a possibility that the issue is that I'm trying to build on a Pi 4, which is actually a 64-bit machine, with a 64-bit kernel. To test this, I copied the linux-headers closure, and all of its deps, to a Pi 0 W (which is actually armv6l), and got the same bug. It is still possible that the issue arises due to the Pi 4 miscompiling one of the dependences. I'm doing a full build on the Pi 0 to rule this out, but of course that could take a while.

Gaelan commented 3 years ago

Alright, nix-build . --arg system '"armv6l-linux"' -A stdenv.__bootPackages.linuxHeaders also fails when building from scratch on a Pi 0. At this point, I'm pretty sure this patch just breaks building anything natively on aarch32.

expipiplus1 commented 3 years ago

Thanks for the testing @Gaelan. I'm not sure what's more important, cross compiling to arm32 or native compiling! Hopefully @Ericson2314 can shed some light here.

Ericson2314 commented 3 years ago

What a mess! I guess the patch fixes some relocations breaks others. If we could make minimal programs for both that would be nice, but that's a lot of work.

Ericson2314 commented 3 years ago

Ah I see @expipiplus1 already minimized in https://sourceware.org/bugzilla/show_bug.cgi?id=16177#c9, and we can cross-reference Jessica's patch on this issue too. So maybe once we minimize this issue we'll have enough combinations to try that it shouldn't be so bad.

tgunnoe commented 3 years ago

I have this problem on armv7l as well, on the novena board. 5.9.8 headers required for building nix fail.

stefandeml commented 3 years ago

Thanks for the testing @Gaelan. I'm not sure what's more important, cross compiling to arm32 or native compiling! Hopefully @Ericson2314 can shed some light here.

I'm running into the exact same issue using arm emulation via qemu-arm-static on my x64 machine.

Atemu commented 3 years ago

IIRC GHC was the only package affected by the bug in my NixOS closure when I had it running on an armv7 machine back in 2019 and it was also happening on native builds; had nothing to do with musl or cross.

Couldn't we just make a special binutils with the patch for GHC and the (at worst) handful of other programs which need it?

stefandeml commented 3 years ago

Hello everyone, I'm trying to make use of the recent changes.

Here what I've done: I'm using the nix installer from here. I guess this should actually not be needed any more as we now have hydra jobs for this merged 🎉.

Using my Pi4 with 32bit kernel: pi@raspberrypi:~ $ uname -a Linux raspberrypi 5.10.17-v7l+ #1414 SMP Fri Apr 30 13:20:47 BST 2021 armv7l GNU/Linux on the unstable channel (should not be relevant) and on current master running: cd path/to/nixpkgs; nix-build . --arg system '"armv7l-linux"' -A stdenv.__bootPackages.linuxHeaders now passes. W-h-o-o 🎉.

However, compiling our friendly greeting: cd path/to/nixpkgs; nix-build . --arg system '"armv7l-linux"' -A hello results in the following segfault:

(cd /build/build/elf/.; /nix/store/1ikhs23qaiw1zq793iac3s1m4307dwvk-binutils-2.35.1/bin/objdump -h dl-load.o dl-lookup.o dl-object.o dl-reloc.o dl-deps.o dl-hwcaps.o dl-runtime.o dl-init.o dl-fini.o dl-debug.o dl-misc.o dl-version.o dl-profile.o dl-tls.o dl-origin.o dl-scope.o dl-execstack.o dl-open.o dl-close.o dl-trampoline.o dl-exception.o dl-sort-maps.o dl-lookup-direct.o dl-call-libc-early-init.o dl-cache.o dl-tunables.o tlsdesc.o dl-tlsdesc.o dl-support.o dl-iteratephdr.o dl-addr.o dl-addr-obj.o enbl-secure.o dl-profstub.o dl-libc.o dl-sym.o dl-sysdep.o dl-error.o dl-reloc-static-pie.o libc_early_init.o find_exidx.o dl-iteratephdr.os dl-addr.os dl-addr-obj.os dl-profstub.os dl-libc.os dl-sym.os dl-error.os libc_early_init.os aeabi_unwind_cpp_pr1.os find_exidx.os rtld.os dl-load.os dl-lookup.os dl-object.os dl-reloc.os dl-deps.os dl-hwcaps.os dl-runtime.os dl-init.os dl-fini.os dl-debug.os dl-misc.os dl-version.os dl-profile.os dl-tls.os dl-origin.os dl-scope.os dl-execstack.os dl-open.os dl-close.os dl-trampoline.os dl-exception.os dl-sort-maps.os dl-lookup-direct.os dl-call-libc-early-init.os dl-cache.os dl-tunables.os tlsdesc.os dl-tlsdesc.os dl-sysdep.os dl-environ.os dl-minimal.os dl-error-minimal.os dl-conflict.os aeabi_read_tp.os libc-do-syscall.os dl-brk.os dl-sbrk.os dl-getcwd.os dl-openat64.os dl-opendir.os dl-fxstatat64.os sofini.os interp.os static-stubs.o cache.o readlib.o xmalloc.o xstrdup.o chroot_canon.o sotruss-lib.os sotruss-lib.so) | \
gawk '/\.gnu\.glibc-stub\./ { \
          sub(/\.gnu\.glibc-stub\./, "", $2); \
          stubs[$2] = 1; } \
        END { for (s in stubs) print "#define __stub_" s }' > /build/build/elf/stubsT
mv -f /build/build/elf/stubsT /build/build/elf/stubs
make[2]: Leaving directory '/build/glibc-2.32/elf'
(sed '/^@/d' include/stubs-prologue.h; LC_ALL=C sort /build/build/csu/stubs /build/build/iconv/stubs /build/build/locale/stubs /build/build/localedata/stubs /build/build/iconvdata/stubs /build/build/assert/stubs /build/build/ctype/stubs /build/build/intl/stubs /build/build/catgets/stubs /build/build/math/stubs /build/build/setjmp/stubs /build/build/signal/stubs /build/build/stdlib/stubs /build/build/stdio-common/stubs /build/build/libio/stubs /build/build/dlfcn/stubs /build/build/nptl/stubs /build/build/malloc/stubs /build/build/string/stubs /build/build/wcsmbs/stubs /build/build/timezone/stubs /build/build/time/stubs /build/build/dirent/stubs /build/build/grp/stubs /build/build/pwd/stubs /build/build/posix/stubs /build/build/io/stubs /build/build/termios/stubs /build/build/resource/stubs /build/build/misc/stubs /build/build/socket/stubs /build/build/sysvipc/stubs /build/build/gmon/stubs /build/build/gnulib/stubs /build/build/wctype/stubs /build/build/manual/stubs /build/build/shadow/stubs /build/build/gshadow/stubs /build/build/po/stubs /build/build/argp/stubs /build/build/rt/stubs /build/build/conform/stubs /build/build/debug/stubs /build/build/mathvec/stubs /build/build/support/stubs /build/build/crypt/stubs /build/build/nptl_db/stubs /build/build/inet/stubs /build/build/resolv/stubs /build/build/nss/stubs /build/build/hesiod/stubs /build/build/sunrpc/stubs /build/build/nis/stubs /build/build/nscd/stubs /build/build/login/stubs /build/build/elf/stubs) > /build/build/stubs.h
if test -r /nix/store/ail4z80i8dfd17h35r00rvqgshgaz43f-glibc-2.32-48-dev/include/gnu/stubs-hard.h && cmp -s /build/build/stubs.h /nix/store/ail4z80i8dfd17h35r00rvqgshgaz43f-glibc-2.32-48-dev/include/gnu/stubs-hard.h; \
then echo 'stubs.h unchanged'; \
else /nix/store/cvzil995dqk628n017d02s1zaa6fjh4i-bootstrap-tools/bin/install -c -m 644 /build/build/stubs.h /nix/store/ail4z80i8dfd17h35r00rvqgshgaz43f-glibc-2.32-48-dev/include/gnu/stubs-hard.h; fi
rm -f /build/build/stubs.h
/build/build/elf/sln /build/build/elf/symlink.list
make[1]: *** [Makefile:115: install-symbolic-link] Segmentation fault
make[1]: Leaving directory '/build/glibc-2.32'
make: *** [Makefile:12: install] Error 2
builder for '/nix/store/hrrfkbfywf8awmj84gkasj3zck5p4s3l-glibc-2.32-48.drv' failed with exit code 2
cannot build derivation '/nix/store/f3a805kyq0aa6yix4r5a2rh9c6xli2wz-bootstrap-stage3-gcc-wrapper-.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/ivlrj5yxxx9a7z7ixs8faq1gszs4iydx-bootstrap-stage4-gcc-wrapper-10.3.0.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/v1gijiqkgzwkr7n534mypjfdrg2sj951-stdenv-linux.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/nzkrvarvqqnm5wqwdxyj3y4vplcs76dw-hello-2.10.drv': 1 dependencies couldn't be built
error: build of '/nix/store/nzkrvarvqqnm5wqwdxyj3y4vplcs76dw-hello-2.10.drv' failed

Seems like this PR has not been all to fix the broken armv7l target. Anyone aware of the issue here and how to continue?

Happy to update the official Wiki as soon as I've things working ... also including the hydra tarball. Thanks!

misuzu commented 3 years ago

@stefandeml try cherry-picking commit from https://github.com/NixOS/nixpkgs/pull/129004

nixos-discourse commented 3 years ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/new-aarch64-and-armv7l-linux-omg-builders/1010/8

stefandeml commented 3 years ago

@stefandeml try cherry-picking commit from #129004

Thanks 🎉. That definitely helped but failing at stage4 now:

g++ -std=gnu++98 -fno-PIE -c   -g -DIN_GCC     -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wno-format -Wmissing-format-attribute -Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings -fno-common  -DHAVE_CONFIG_H -I. -I. -I../../gcc-10.3.0/gcc -I../../gcc-10.3.0/gcc/. -I../../gcc-10.3.0/gcc/../include -I../../gcc-10.3.0/gcc/../libcpp/include -I/nix/store/5ydj93x2ih7fwfaakicxgx6n1l3gbwiv-gmp-6.2.1-dev/include -I/nix/store/68jvd28xs49xcajivfnb9dn823v4wxd5-mpfr-4.1.0-dev/include -I/nix/store/4v70af3qw8ryrlghjg9pf0z4j554r28w-libmpc-1.2.1/include  -I../../gcc-10.3.0/gcc/../libdecnumber -I../../gcc-10.3.0/gcc/../libdecnumber/dpd -I../libdecnumber -I../../gcc-10.3.0/gcc/../libbacktrace -I/nix/store/j8fpc8g0sbjpsyzrwrbvb7zn84n72pij-isl-0.20/include  -o insn-extract.o -MT insn-extract.o -MMD -MP -MF ./.deps/insn-extract.TPo insn-extract.c
g++: fatal error: Killed signal terminated program cc1plus
compilation terminated.
make[3]: *** [Makefile:1117: insn-emit.o] Error 1
make[3]: *** Waiting for unfinished jobs....
g++: fatal error: Killed signal terminated program cc1plus
compilation terminated.
make[3]: *** [Makefile:1117: gimple-match.o] Error 1
rm gcc.pod
make[3]: Leaving directory '/build/build/gcc'
make[2]: *** [Makefile:4741: all-stage1-gcc] Error 2
make[2]: Leaving directory '/build/build'
make[1]: *** [Makefile:22554: stage1-bubble] Error 2
make[1]: Leaving directory '/build/build'
make: *** [Makefile:22875: bootstrap] Error 2
builder for '/nix/store/16yvw06i2ffynql1g28nkns6hkjr7w1b-gcc-10.3.0.drv' failed with exit code 2
cannot build derivation '/nix/store/x5492ifnq2w4smrdspvdxbw20z8js1hw-bootstrap-stage4-gcc-wrapper-10.3.0.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/m4gb0xhc9w847j5liga7fw2hcqwadmkm-stdenv-linux.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/62ck69fyz7gi6jbhicizwvjwjp1gqkwi-hello-2.10.drv': 1 dependencies couldn't be built
error: build of '/nix/store/62ck69fyz7gi6jbhicizwvjwjp1gqkwi-hello-2.10.drv' failed

Hmm, looks like the OS killed it? - maybe OOM? I guess there is a way to reduce memory load by reducing the number of parallel jobs? Do we know the minimal config right now?

misuzu commented 3 years ago

Hmm, looks like the OS killed it? - maybe OOM? I guess there is a way to reduce memory load by reducing the number of parallel jobs?

I had no issues building the entire system (with a few tweaks) on armv7l virtual machine with 4 cores and 4G of RAM. You can try building with -j1 --cores 1. Do you use NixOS? If not, is nix installed in multi-user mode?

vcunat commented 3 years ago

Yes, OOM often produces exactly this "killed" log line.

Atemu commented 3 years ago

Also make sure you have zram swap (perhaps even with zstd) if you're running into OOM issues. It's free additional RAM space.

yu-re-ka commented 2 years ago

Hitting the exact same bug when cross-compiling to armv7l now, in packages that have PIC disabled (zfs)