pkgsCross.aarch64-multiplatform.gcc13 (and other cross-builds) fail to build due to gcc version mismatch

trofi commented 1 year ago

Describe the bug

pkgsCross.aarch64-multiplatform.gcc13 fails to build on x86_64-linux.

Steps To Reproduce

$  nix build --no-link nixpkgs#pkgsCross.aarch64-multiplatform.gcc13 -L
...
gcc-aarch64-unknown-linux-gnu> make[2]: *** [../../../gcc-13.1.0/libgcc/shared-object.mk:14: extendbfsf2.o] Error 1
gcc-aarch64-unknown-linux-gnu> make[2]: *** Waiting for unfinished jobs....
gcc-aarch64-unknown-linux-gnu> In file included from ../../../gcc-13.1.0/libgcc/soft-fp/trunctfbf2.c:30:
gcc-aarch64-unknown-linux-gnu> ../../../gcc-13.1.0/libgcc/soft-fp/brain.h:62:1: error: unable to emulate 'BF'
gcc-aarch64-unknown-linux-gnu>    62 | typedef float BFtype __attribute__ ((mode (BF)));
gcc-aarch64-unknown-linux-gnu>       | ^~~~~~~

Expected behavior

Build should succeed.

Additional context

I think the main cause is that cross-compiler version used to cross-build gcc does not match: cross-building gcc-13 requires gcc-13 cross-compiler presence. The build uses gcc-12 (without recently added bfloat support).

Similar kind of problem happens whee we try to build nixpkgs#pkgsCross.aarch64-multiplatform.gcc11 where newly added warning in gcc12 trigger -Werror= failures on gcc-11.

Metadata

$ nix-shell -p nix-info --run "nix-info -m"

 - system: `"x86_64-linux"`
 - host os: `Linux 6.4.3, NixOS, 23.11 (Tapir), 23.11.20230722.0b411c1`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.16.1`
 - channels(root): `""`
 - channels(slyfox): `""`
 - nixpkgs: `/run/current-system/sw/share/nixos-flakes/inputs/nixpkgs`

ghost commented 1 year ago

I think the main cause is that cross-compiler version used to cross-build gcc does not match: cross-building gcc-13 requires gcc-13 cross-compiler presence.

Yes. Unfortunately pkgsCross has its own screwy bootstrapping scheme (see libcCrossChooser, glibcCross, etc). It really should work by adding extra stages to stages.nix, but we have a bunch of assumptions in nixpkgs that there will always be exactly two stages (and no more) after the "native packageset". These two stages are in pkgs/stdenv/cross/default.nix. If it wasn't for that restriction we could just use an extra copy of exactly the same bootstrap sequence in pkgs/stdenv/generic to build pkgsCross.

I started working on this, but getting rid of that "exactly two stages after the native bootstrap completes and no more" assumption is really hard to get rid of.

Anyways, I'm working on this bug; I have an idea for a temporary fix.

ghost commented 1 year ago

So apparently we have a bigger problem: there is no way to choose what compiler is used for pkgsCross.FOO.stdenv.cc.cc. Unless I'm totally missing something.

If we had that, the fix for this bug would be to set that equal to the same version of gcc that you want to build.

ghost commented 1 year ago

So, this is basically a special case of a more general problem with nixpkgs:

https://github.com/NixOS/nixpkgs/issues/245168

For this particular case, the solution -- for now -- is to not use pkgsCross for this. Instead, instantiate nixpkgs explicitly, passing an overlay. In other words, instead of

nix-instantiate . -A pkgsCross.aarch64-multiplatform.gcc13

you should use https://github.com/NixOS/nixpkgs/pull/245169 and

nix-instantiate . -A gcc13 \
  --argstr crossSystem "aarch64-linux" \
  --arg overlays '[(final: prev: final.lib.optionalAttrs (with prev.stdenv; hostPlatform!=targetPlatform) { default-gcc-version = 13; })]'

The important part here is that gcc.libc.stdenv.cc.cc.version needs to exactly match gcc.version. In prose, this means that the glibc which is passed as the libc argument to gcc must have been compiled by exactly the same version of gcc. This is because glibc is linked against the libgcc that came from the gcc that compiled it.

We meet this requirement in the native bootstrap by using the same source code for both xgcc and gcc.
We meet this requirement in a (build==host)!=target compiler by using the same gcc for both gccWithoutTargetLibc and gcc.

However for a build!=(host==target) compiler which is not the default gcc version, the final compiler will be a non-default version, but we have no way to tell pkgsCross to compile it using something other than the default version (i.e. https://github.com/NixOS/nixpkgs/issues/245168). So we don't have any way to satisfy the requirement. So the best we can do for now is to simply change the default.

ghost commented 1 year ago

The permanent, automatic fix for this is https://github.com/NixOS/nixpkgs/pull/249301

It's still a bit WIP and I need to test it more, but it's pretty close.

NixOS / nixpkgs