ghc cross-compilation and ghc-musl: notes, issues encountered, progress

dtzWill commented 6 years ago

Notes

Some of this, I just discovered, repeats items from https://github.com/NixOS/nixpkgs/issues/36200 and its discussion, but including for now while brain-dump things :).

Apologies that at this hour I'm mostly full of issues and not notes.

For now, checkmarks are issues tackled in my branch/efforts, not neccesarily upstream yet.

Issues

[ ] AFAICT specifying BuildFlavour in mk/build.mk does nothing by default, in mk/build.mk.sample (for versions I inspected, and I suspect all) there's code that goes "if buildFlavour != "" then include ./mk/flavours/${buildFlavour}.mk`. I'm not sure what "flavour" we want, this seems like something worth checking out--cross and non-cross-- to ensure we're doing what we want.
[x] CXX is wrong--there is nothing with the -cxx suffix, it should be c++
[x] dangling symlinks: gentoo has some patches regarding this, particularly this one. This behavior can be seen by inspecting a path such as /nix/store/4yjvzsnk906bm61sk2037bvwbcnlsydg-aarch64-unknown-linux-musl-ghc-8.2.2 (available via cache.nixos.org).
[x] bundled terminfo is very special (1, 2, 3) and as-is needs ncurses (and host's gcc??)-- which we aren't currently providing AFAICT. This is hopefully easy, although I'm a bit unclear what inputs are from (ghc's "confused" terminology vs our own certainly doesn't help, heh) where :) so it's taking a number of iterations to sort this out.

Minor?

[x] paxmark in postInstall assumes installation will be in $out/lib/${targetPrefix}-ghc-${version} (or so), this is not the case unless ghc "thinks" it is cross-compiling. Which it doesn't when cross-compiling to musl O:). D'oh to hit this after building all the rest, hehe.
[ ] Related to the above: At least in 8.2.2 (version I'm looking at the most since it's currently our default) terminfo dropped the --with-curses-includes flag, because of reasons. Anyway only use of setting this is to add ncurses.dev as a build dep of some sort-- which if that is all we want should probably be accomplished more directly.
[ ] GHC seems to prefer if we specified tools with --with-as=/path/to/as instead of setting AS, but I'm not sure.
[ ] Perhaps cross-GHC should be built statically? This seems to be what the "flavour" files for cross specify.
[ ] The resulting ghc is ... well, host is glibc :). This seems to be expected, but means this is mostly useful for cross-compiling things to musl not being used on musl. Is there a nixpkgs way to cross-compile GHC instead of creating a GHC cross-compiler? (shift a layer?)
[ ] libffi is parameter to expression but unused-- might be best to use system libffi?

Good news / Progress

Aarch64-musl works?!

As linked above.. apparently GHC for aarch64-musl has been working for a while now?! Forgive the lack of styling, but here's cross-trunk filtered on "musl": https://hydra.nixos.org/jobset/nixpkgs/cross-trunk/jobs-tab?filter=musl

Where you can see 8.2.2 and HEAD are building!

A copy of "hello" built against musl for aarch64 can be obtained and inspected (it's a cross-trunk job) right now:

$  nix-store -r /nix/store/lcw90icbaakys5c1vi4ixvjsyriycxk1-hello-1.0.0.2-aarch64-unknown-linux-musl/bin/hello

You can even run it with qemu-aarch64! :grin:

armv6l

Similarly: /nix/store/83w221rd46wa08ik5gx7g8nw739rmflg-hello-1.0.0.2-armv6l-unknown-linux-musleabihf/bin/hello is in the nix cache and seems to work via qemu-arm!

x86_64

I believe I have a somewhat working version, currently iterating on cleanup and investigating/resolving some of the issues mentioned above. Not entirely sure I didn't manage to just build a broken glibc-based GHC though, haha, will report as I work on this further.

EDIT: Yep, works! Largest outstanding problem is that GHC doesn't treat this as a cross-compile, and so doesn't prefix tools. This is a problem because our plumbing expects tools to be prefixed if host != target. For now just punting and manually creating links to prefixed versions to get things going :).

My dev branch (forgive any rebase/force-pushes please) is here: https://github.com/dtzWill/nixpkgs/tree/fix/ghc-cross-musl

EDIT2: https://github.com/dtzWill/nixpkgs/releases/tag/ghc-cross-musl-works works for x86_64? :grin:

dtzWill commented 6 years ago

The resulting ghc is ... well, host is glibc :). This seems to be expected, but means this is mostly useful for cross-compiling things to musl not being used on musl. Is there a nixpkgs way to cross-compile GHC instead of creating a GHC cross-compiler? (shift a layer?)

This is the result of setting stage1Only which we currently set when host != target. Here's some info about this setting and how it interacts with cross-compiling and other fun:

https://ghc.haskell.org/trac/ghc/browser/ghc/mk/config.mk.in?rev=d7c5ec0c28a45ea22c5901a152cbc9123a07564c#L565

domenkozar commented 6 years ago

@dtzWill is there a way to get native stdenv with musl (x86_64-linux)?

domenkozar commented 6 years ago

There is nix-build -A ghc --arg localSystem '{config="x86_64-unknown-linux-musl";}' but it's blocked on https://ghc.haskell.org/trac/ghc/ticket/13958?cnum_edit=1#comment:1

dtzWill commented 6 years ago

It was my understanding musl-native GHC was blocked on providing a musl-native bootstrap GHC, is that not a problem?

(resolving this was primarily why I was looking into cross-compiling GHC to musl)

That issue suggests setting --disable-ld-override might resolve the issue? Does that work for you?

domenkozar commented 6 years ago

Could we use cross-compiled musl for bootstrapping native one? --disable-ld-override didn't work :)

dtzWill commented 6 years ago

I expect so, although I think there's some packaging/build options that are used for generating a bootstrap GHC. FWIW various musl-based distributions have bootstrap GHC's we may want to use as a starting point (assuming such things are more or less usable elsewhere) --modulo concerns about "trusting trust".

But yes the idea was to explore generating a bootstrap GHC from a cross build, although perhaps upstream can comment on how this is usually done.

dtzWill commented 6 years ago

@Ericson2314 hey whatcha doing? Are you intentionally pushing to my branch? Apparently broke some things, but mostly just wanna know what's going on. Rebasing onto latest?

nh2 commented 6 years ago

With that push, GHC 8.2.2 doesn't build any more for me, I get a failure in the terminfo package:

Configuring terminfo-0.4.1.0...
configure: WARNING: unrecognized options: --with-compiler
checking for gcc... /nix/store/vhg71q6zx8xbb851kyah0flyjjglapq7-gcc-wrapper-7.3.0/bin/gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether /nix/store/vhg71q6zx8xbb851kyah0flyjjglapq7-gcc-wrapper-7.3.0/bin/gcc accepts -g... yes
checking for /nix/store/vhg71q6zx8xbb851kyah0flyjjglapq7-gcc-wrapper-7.3.0/bin/gcc option to accept ISO C89... none needed
checking for setupterm in -ltinfo... no
checking for setupterm in -lncursesw... no
checking for setupterm in -lncurses... no
checking for setupterm in -lcurses... no
configure: error: in `/tmp/nix-build-x86_64-unknown-linux-musl-ghc-8.2.2.drv-0/ghc-8.2.2/libraries/terminfo':
configure: error: curses library not found, so this package cannot be built

nh2 commented 6 years ago

Rebasing onto latest?

Looks like that is the answer, see https://github.com/NixOS/nixpkgs/pull/37598#issuecomment-397041800

Ericson2314 commented 6 years ago

@nh2 Sorry :(. I assume that is probably because cross is in general broken until https://github.com/NixOS/nixpkgs/pull/41429 hit's master (or is re-merged directly in; I'm on the fence whether that is justifiable).

nh2 commented 6 years ago

@Ericson2314 Should we just rebase it on top of the commit before the broken one (2110c0b~), or pick that patch in the musl branch?

Ericson2314 commented 6 years ago

@nh2 I'd merge it in. It's a 100% whole world mass rebuild unfortuantely since it changes cc-wrapper.

nh2 commented 6 years ago

@Ericson2314 However you prefer -- I'm OK with mass rebuilds, better rebuilds than broken builds.

nh2 commented 6 years ago

Definitely keep working on this -- based on this, I now managed to build a Haskell package with lots of dependencies (stack) fully statically: https://github.com/nh2/static-haskell-nix

NixOS / nixpkgs