search

NixOS / nixpkgs

Nix Packages collection & NixOS
MIT License
17.63k stars 13.78k forks source link

Cross-compiling haskell executables for Android #56493

Open nomeata opened 5 years ago

nomeata commented 5 years ago

Issue description

I was under the impression that cross-compiling Haskell executables for Android should work (if that is wrong and it actually never worked, then sorry for the noise). An example for a most simple program is hello.

Steps to reproduce

Unfortunately, it does not:

~/build/nixpkgs $ git log  --oneline -n 1
e416d2f8b9a (HEAD -> staging, origin/staging) Merge pull request #55650 from dtzWill/update/plasma-5-5.15.0
~/build/nixpkgs $ nix-build -A pkgsCross.aarch64-android-prebuilt.haskellPackages.hello
these derivations will be built:
  /nix/store/0w5gdsxj03sz68davdx2an1qlxzi6mj0-gmp-6.1.2-aarch64-unknown-linux-android.drv
  /nix/store/3lvmr6di73ryppv6ih7hdrzcrkk98nzx-libiconv-1.15-aarch64-unknown-linux-android.drv
  /nix/store/6y8ikfq1c04fs3cx9ps9xj42mkvn3cpd-libffi-3.2.1-aarch64-unknown-linux-android.drv
  /nix/store/l9pvmi893m49yb5yd22vy6wsl5qk8vpx-aarch64-unknown-linux-android-ghc-8.6.3.drv
  /nix/store/8zqay0rqn17xsnwbng1cghsraz3p23zc-hello-1.0.0.2-aarch64-unknown-linux-android.drv
building '/nix/store/0w5gdsxj03sz68davdx2an1qlxzi6mj0-gmp-6.1.2-aarch64-unknown-linux-android.drv'...
unpacking sources
unpacking source archive /nix/store/8lz4k7gnsj4mj44y7vwcrhxw92m00xwp-gmp-6.1.2.tar.bz2
source root is gmp-6.1.2
setting SOURCE_DATE_EPOCH to timestamp 1481903147 of file gmp-6.1.2/mini-gmp/tests/testutils.h
patching sources
updateAutotoolsGnuConfigScriptsPhase
Updating Autotools / GNU config script to a newer upstream version: ./config.sub
Updating Autotools / GNU config script to a newer upstream version: ./config.guess
configuring
fixing libtool script ./ltmain.sh
configure flags: --disable-static --prefix=/nix/store/hsj86fcih1m95rmkvrkrkkkis7p7hc7a-gmp-6.1.2-aarch64-unknown-linux-android --bindir=/nix/store/hsj86fcih1m95rmkvrkrkkkis7p7hc7a-gmp-6.1.2-aarch64-unknown-linux-android/bin --sbindir=/nix/store/hsj86fcih1m95rmkvrkrkkkis7p7hc7a-gmp-6.1.2-aarch64-unknown-linux-android/sbin --includedir=/nix/store/n30rs13m9nbfy895qq58sii3c9vn04yf-gmp-6.1.2-aarch64-unknown-linux-android-dev/include --oldincludedir=/nix/store/n30rs13m9nbfy895qq58sii3c9vn04yf-gmp-6.1.2-aarch64-unknown-linux-android-dev/include --mandir=/nix/store/hsj86fcih1m95rmkvrkrkkkis7p7hc7a-gmp-6.1.2-aarch64-unknown-linux-android/share/man --infodir=/nix/store/q7d3hfbdkdvw7pm2yjhwq9ln4sr42jp6-gmp-6.1.2-aarch64-unknown-linux-android-info/share/info --docdir=/nix/store/hsj86fcih1m95rmkvrkrkkkis7p7hc7a-gmp-6.1.2-aarch64-unknown-linux-android/share/doc/gmp --libdir=/nix/store/hsj86fcih1m95rmkvrkrkkkis7p7hc7a-gmp-6.1.2-aarch64-unknown-linux-android/lib --libexecdir=/nix/store/hsj86fcih1m95rmkvrkrkkkis7p7hc7a-gmp-6.1.2-aarch64-unknown-linux-android/libexec --localedir=/nix/store/hsj86fcih1m95rmkvrkrkkkis7p7hc7a-gmp-6.1.2-aarch64-unknown-linux-android/share/locale --with-pic --disable-cxx --disable-fat --build=x86_64-unknown-linux-gnu --disable-assembly --build=x86_64-unknown-linux-gnu --host=aarch64-unknown-linux-android
checking build system type... x86_64-unknown-linux-gnu
checking host system type... aarch64-unknown-linux-android
checking for a BSD-compatible install... /nix/store/d9s1kq1bnwqgxwcvv4zrc36ysnxg8gv7-coreutils-8.30/bin/install -c
checking whether build environment is sane... yes
checking for aarch64-unknown-linux-android-strip... aarch64-unknown-linux-android-strip
checking for a thread-safe mkdir -p... /nix/store/d9s1kq1bnwqgxwcvv4zrc36ysnxg8gv7-coreutils-8.30/bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking whether to enable maintainer-specific portions of Makefiles... no
checking ABI=64
checking whether aarch64-unknown-linux-android-gcc is gcc... yes
checking compiler aarch64-unknown-linux-android-gcc -O2 -pedantic ... no
checking ABI=32
checking whether aarch64-unknown-linux-android-gcc is gcc... yes
checking compiler aarch64-unknown-linux-android-gcc -O2 -pedantic -fomit-frame-pointer ... no
configure: error: could not find a working compiler, see config.log for details
builder for '/nix/store/0w5gdsxj03sz68davdx2an1qlxzi6mj0-gmp-6.1.2-aarch64-unknown-linux-android.drv' failed with exit code 1
cannot build derivation '/nix/store/l9pvmi893m49yb5yd22vy6wsl5qk8vpx-aarch64-unknown-linux-android-ghc-8.6.3.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/8zqay0rqn17xsnwbng1cghsraz3p23zc-hello-1.0.0.2-aarch64-unknown-linux-android.drv': 1 dependencies couldn't be built
error: build of '/nix/store/8zqay0rqn17xsnwbng1cghsraz3p23zc-hello-1.0.0.2-aarch64-unknown-linux-android.drv' failed

This is staging with the Android-related fixes from https://github.com/NixOS/nixpkgs/pull/56197 (thanks for them!) already present.

Technical details

Please run nix-shell -p nix-info --run "nix-info -m" and paste the results.

 - system: `"x86_64-linux"`
 - host os: `Linux 4.19.0-2-amd64, Debian GNU/Linux, noversion`
 - multi-user?: `yes`
 - sandbox: `no`
 - version: `nix-env (Nix) 2.0.4`
find: '/nix/var/nix/profiles/per-user': No such file or directory

I am opening this issues for “simple Haskell programs”, as a blocker for the more ambitious issues about OpenGl (#56121) and SDL (#56120).

matthewbauer commented 5 years ago

The background on this is that it worked before December, but then the NDK got updated and it broke some stuff. My suspicion is the NDK is doing some voodoo on the flags that is breaking things. The NDK used to use (gcc, bfd) but has now moved to (clang, bfd) and is planning to move to (clang, lld) after that. Ideally we could start using our own compilers and just use the ndk for libraries to avoid this brittleness. I would advise using either reflex-platform or Nixpkgs 18.09 for best results.

/cc @luigy

nomeata commented 5 years ago

Thanks for the background, and being patient with a nix beginner like me :-)

nomeata commented 5 years ago

JFTR, release-18.09 (commit 1374ba6080b9ea85e78ea39d62b08eb64278119b) does not work either, it fails to compile GHC:

"inplace/bin/ghc-stage1" -static  -O -H64m -Wall   -Iincludes -Iincludes/dist -Iincludes/dist-derivedconstants/header -Iincludes/dist-ghcconstants/header -Irts -Irts/dist/build -DCOMPILING_RTS -this-unit-id rts -dcmm-lint      -i -irts -irts/dist/build -Irts/dist/build -irts/dist/build/./autogen -Irts/dist/build/./autogen            -O2 -fPIC -Wcpp-undef    -Wnoncanonical-monad-instances  -c rts/Apply.cmm -o rts/dist/build/Apply.o
ghc-stage1: panic! (the 'impossible' happened)
  (GHC version 8.4.4 for aarch64-unknown-linux-android):
        padLiveArgs -- i > regNum ??
CallStack (from HasCallStack):
  error, called at compiler/llvmGen/LlvmCodeGen/Base.hs:194:27 in ghc:LlvmCodeGen.Base

Please report this as a GHC bug:  http://www.haskell.org/ghc/reportabug

make[1]: *** [rts/ghc.mk:295: rts/dist/build/StgMiscClosures.o] Error 1
make[1]: *** Waiting for unfinished jobs....
ghc-stage1: panic! (the 'impossible' happened)
  (GHC version 8.4.4 for aarch64-unknown-linux-android):
        padLiveArgs -- i > regNum ??
CallStack (from HasCallStack):
  error, called at compiler/llvmGen/LlvmCodeGen/Base.hs:194:27 in ghc:LlvmCodeGen.Base

Please report this as a GHC bug:  http://www.haskell.org/ghc/reportabug

make[1]: *** [rts/ghc.mk:295: rts/dist/build/HeapStackCheck.o] Error 1
make: *** [Makefile:127: all] Error 2
builder for '/nix/store/3r1mddr4fhmd9aqfvn9kb2q4gyj8g9sn-aarch64-unknown-linux-android-ghc-8.4.4.drv' failed with exit code 2
matthewbauer commented 5 years ago

Yeah that's a known issue in ghc 8.4.4:

https://ghc.haskell.org/trac/ghc/ticket/15780

I would try ghc 8.4.3 or wait for ghc 8.4.5 to come out.

/cc @bgamari

nomeata commented 5 years ago

Indeed,

nix-build -A pkgsCross.aarch64-android-prebuilt.haskell.packages.ghc843.hello

works on release-18.09. Great! I wonder if I can manage to backport the fixes from #56197 onto that branch, so that I can also do

nix-build -A pkgsCross.aarch64-android-prebuilt.haskell.packages.ghc843.SDL

… not trivially. Guess I just have to wait for all the starts to align on master again :-)

nomeata commented 5 years ago

Just checked the status of this again, with current master (i.e. 58213cd9ca6), and I get, now with GHC-8.6.5:

~/build/nixpkgs $ git log --oneline -n 1
58213cd9ca6 (HEAD -> master, origin/master, origin/HEAD) Merge #60740: ffmpeg*: 4.1.2 -> 4.1.3
~/build/nixpkgs $ nix-build -A pkgsCross.aarch64-android-prebuilt.haskellPackages.hello
…
inplace/bin/deriveConstants --gen-haskell-type -o includes/dist-derivedconstants/header/GHCConstantsHaskellType.hs --tmpdir includes/dist-derivedconstants/header/ --gcc-program "/nix/store/9bkrc4s13jjbig895gl7fbyq4a5lfyr2-aarch64-unknown-linux-android-ndk-gcc-binutils-wrapper/bin/aarch64-unknown-linux-android-cc" --gcc-flag -Wall --gcc-flag -Wno-unknown-pragmas --gcc-flag -fno-stack-protector --gcc-flag -Iincludes --gcc-flag -Iincludes/dist --gcc-flag -Iincludes/dist-derivedconstants/header --gcc-flag -Iincludes/dist-ghcconstants/header --gcc-flag -Irts --gcc-flag -fcommon --nm-program "/nix/store/jbx7bdyzmfbrcdm50g68vq8ljw3yd7nr-ndk-gcc-binutils/bin/aarch64-unknown-linux-android-nm" --objdump-program "aarch64-unknown-linux-android-objdump" --target-os "linux_android"
inplace/bin/deriveConstants --gen-haskell-wrappers -o includes/dist-derivedconstants/header/GHCConstantsHaskellWrappers.hs --tmpdir includes/dist-derivedconstants/header/ --gcc-program "/nix/store/9bkrc4s13jjbig895gl7fbyq4a5lfyr2-aarch64-unknown-linux-android-ndk-gcc-binutils-wrapper/bin/aarch64-unknown-linux-android-cc" --gcc-flag -Wall --gcc-flag -Wno-unknown-pragmas --gcc-flag -fno-stack-protector --gcc-flag -Iincludes --gcc-flag -Iincludes/dist --gcc-flag -Iincludes/dist-derivedconstants/header --gcc-flag -Iincludes/dist-ghcconstants/header --gcc-flag -Irts --gcc-flag -fcommon --nm-program "/nix/store/jbx7bdyzmfbrcdm50g68vq8ljw3yd7nr-ndk-gcc-binutils/bin/aarch64-unknown-linux-android-nm" --objdump-program "aarch64-unknown-linux-android-objdump" --target-os "linux_android"
inplace/bin/deriveConstants --gen-haskell-exports -o includes/dist-derivedconstants/header/GHCConstantsHaskellExports.hs --tmpdir includes/dist-derivedconstants/header/ --gcc-program "/nix/store/9bkrc4s13jjbig895gl7fbyq4a5lfyr2-aarch64-unknown-linux-android-ndk-gcc-binutils-wrapper/bin/aarch64-unknown-linux-android-cc" --gcc-flag -Wall --gcc-flag -Wno-unknown-pragmas --gcc-flag -fno-stack-protector --gcc-flag -Iincludes --gcc-flag -Iincludes/dist --gcc-flag -Iincludes/dist-derivedconstants/header --gcc-flag -Iincludes/dist-ghcconstants/header --gcc-flag -Irts --gcc-flag -fcommon --nm-program "/nix/store/jbx7bdyzmfbrcdm50g68vq8ljw3yd7nr-ndk-gcc-binutils/bin/aarch64-unknown-linux-android-nm" --objdump-program "aarch64-unknown-linux-android-objdump" --target-os "linux_android"
inplace/bin/deriveConstants --gen-header -o includes/dist-derivedconstants/header/DerivedConstants.h --tmpdir includes/dist-derivedconstants/header/ --gcc-program "/nix/store/9bkrc4s13jjbig895gl7fbyq4a5lfyr2-aarch64-unknown-linux-android-ndk-gcc-binutils-wrapper/bin/aarch64-unknown-linux-android-cc" --gcc-flag -Wall --gcc-flag -Wno-unknown-pragmas --gcc-flag -fno-stack-protector --gcc-flag -Iincludes --gcc-flag -Iincludes/dist --gcc-flag -Iincludes/dist-derivedconstants/header --gcc-flag -Iincludes/dist-ghcconstants/header --gcc-flag -Irts --gcc-flag -fcommon --nm-program "/nix/store/jbx7bdyzmfbrcdm50g68vq8ljw3yd7nr-ndk-gcc-binutils/bin/aarch64-unknown-linux-android-nm" --objdump-program "aarch64-unknown-linux-android-objdump" --target-os "linux_android"
In file included from includes/dist-derivedconstants/header/tmp.c:13:
In file included from includes/Rts.h:168:
includes/rts/OSThreads.h:157:4: error: "Threads not supported"
#  error "Threads not supported"
   ^
includes/rts/OSThreads.h:165:8: error: unknown type name 'OSThreadId'
extern OSThreadId osThreadId      ( void );
       ^
includes/rts/OSThreads.h:169:31: error: expected ';' after top level declarator
typedef void* OSThreadProcAttr OSThreadProc(void *);
                              ^
                              ;
includes/rts/OSThreads.h:171:37: error: unknown type name 'OSThreadId'
extern int  createOSThread        ( OSThreadId* tid, char *name,
                                    ^
includes/rts/OSThreads.h:172:37: error: unknown type name 'OSThreadProc'; did you mean 'OSThreadProcAttr'?
                                    OSThreadProc *startProc, void *param);
                                    ^~~~~~~~~~~~
                                    OSThreadProcAttr
includes/rts/OSThreads.h:169:15: note: 'OSThreadProcAttr' declared here
typedef void* OSThreadProcAttr OSThreadProc(void *);
              ^
includes/rts/OSThreads.h:173:37: error: unknown type name 'OSThreadId'
extern bool osThreadIsAlive       ( OSThreadId id );
                                    ^
includes/rts/OSThreads.h:174:36: error: unknown type name 'OSThreadId'
extern void interruptOSThread     (OSThreadId id);
                                   ^
includes/rts/OSThreads.h:179:37: error: unknown type name 'Condition'
extern void initCondition         ( Condition* pCond );
                                    ^
includes/rts/OSThreads.h:180:37: error: unknown type name 'Condition'
extern void closeCondition        ( Condition* pCond );
                                    ^
includes/rts/OSThreads.h:181:37: error: unknown type name 'Condition'
extern bool broadcastCondition    ( Condition* pCond );
                                    ^
includes/rts/OSThreads.h:182:37: error: unknown type name 'Condition'
extern bool signalCondition       ( Condition* pCond );
                                    ^
includes/rts/OSThreads.h:183:37: error: unknown type name 'Condition'
extern bool waitCondition         ( Condition* pCond, Mutex* pMut );
                                    ^
includes/rts/OSThreads.h:183:55: error: unknown type name 'Mutex'
extern bool waitCondition         ( Condition* pCond, Mutex* pMut );
                                                      ^
includes/rts/OSThreads.h:188:37: error: unknown type name 'Mutex'
extern void initMutex             ( Mutex* pMut );
                                    ^
includes/rts/OSThreads.h:189:37: error: unknown type name 'Mutex'
extern void closeMutex            ( Mutex* pMut );
                                    ^
includes/rts/OSThreads.h:194:26: error: unknown type name 'ThreadLocalKey'
void  newThreadLocalKey (ThreadLocalKey *key);
                         ^
includes/rts/OSThreads.h:195:26: error: unknown type name 'ThreadLocalKey'
void *getThreadLocalVar (ThreadLocalKey *key);
                         ^
includes/rts/OSThreads.h:196:26: error: unknown type name 'ThreadLocalKey'
void  setThreadLocalVar (ThreadLocalKey *key, void *value);
                         ^
includes/rts/OSThreads.h:197:27: error: unknown type name 'ThreadLocalKey'
void  freeThreadLocalKey (ThreadLocalKey *key);
                          ^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.
Executing "/nix/store/9bkrc4s13jjbig895gl7fbyq4a5lfyr2-aarch64-unknown-linux-android-ndk-gcc-binutils-wrapper/bin/aarch64-unknown-linux-android-cc" failed
libraries/hpc/ghc.mk:3: libraries/hpc/dist-boot/build/.depend-v.haskell: No such file or directory
make[1]: *** [includes/ghc.mk:193: includes/dist-derivedconstants/header/DerivedConstants.h] Error 1
make[1]: *** Waiting for unfinished jobs....
make: *** [Makefile:125: all] Error 2
builder for '/nix/store/6blaw2jybinmdz14x1gx4cmvdim30m2g-aarch64-unknown-linux-android-ghc-8.6.5.drv' failed with exit code 2

So things are changing…

stale[bot] commented 4 years ago

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

  1. Search for maintainers and people that previously touched the related code and @ mention them in a comment.
  2. Ask on the NixOS Discourse.
  3. Ask on the #nixos channel on irc.freenode.net.
573 commented 3 years ago

I'm still interested in this.

jtacoma commented 3 years ago

Just taking a peek at Haskell cross-compilation support on NixOS, I found this bug and repeated the steps to reproduce. The nix-build [...] command isn't working for me either right now, but I get a different error message:

$ git log  --oneline -n 1
4138cbd913f (HEAD -> nixos-unstable, origin/nixos-unstable) Merge pull request #133535 from ylh/patch-1
$ nix-build -A pkgsCross.aarch64-android-prebuilt.haskellPackages.hello
[...]
error: builder for '/nix/store/[...]-aarch64-unknown-linux-android-ghc-8.10.4.drv' failed with exit code 2;
[...]
> checking build system type... x86_64-pc-linux-gnu
> checking host system type... Invalid configuration `aarch64-android': system `android' not recognized
$ nix-shell -p nix-info --run "nix-info -m" 
 - system: `"x86_64-linux"`
 - host os: `Linux 5.10.52, NixOS, 21.05.2478.a445f582988 (Okapi)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.4pre20210802_47e96bb`
 - channels(jtacoma): `"nixpkgs-21.11pre300649.3e0ce8c5d47, home-manager, hydra, color, critical-false-brace"`
 - channels(root): `"nixos-21.05.2132.733682c3292, hydra"`
 - nixpkgs: `/home/jtacoma/.nix-defexpr/channels/nixpkgs`
stale[bot] commented 2 years ago

I marked this as stale due to inactivity. → More info

exarkun commented 1 year ago

This is now broken even earlier because aarch64-android-prebuilt has an infinite recursion somewhere in stdenv (maybe):

nix-repl> inputs.nixpkgs.legacyPackages.x86_64-linux.pkgsCross.aarch64-android-prebuilt.haskell.packages.ghc8107.SDL
error: infinite recursion encountered

       at /nix/store/jwk9dm1kwaplwac6smjl9dx145yrm744-source/pkgs/stdenv/generic/default.nix:133:14:

          132|
          133|       inherit initialPath shell
             |              ^
          134|         defaultNativeBuildInputs defaultBuildInputs;
exarkun commented 1 year ago

And in the non-prebuilt:

nix-repl> inputs.nixpkgs.legacyPackages.x86_64-linux.pkgsCross.aarch64-android.haskell.packages.ghc8107.SDL
error: attribute 'targetPrefix' missing

       at /nix/store/jwk9dm1kwaplwac6smjl9dx145yrm744-source/pkgs/development/compilers/ghc/8.10.7.nix:256:46:

          255|     export READELF="${targetCC.bintools.bintools}/bin/${targetCC.bintools.targetPrefix}readelf"
          256|     export STRIP="${bintoolsFor.strip}/bin/${bintoolsFor.strip.targetPrefix}strip"
             |                                              ^
          257|   '' + lib.optionalString (stdenv.targetPlatform.linker == "cctools") ''
exarkun commented 1 year ago

(And by "now" I mean nixpkgs 5528350186a9e826588cee1329640899ca44a0cf)