search

nh2 / static-haskell-nix

easily build most Haskell programs into fully static Linux executables
388 stars 36 forks source link

Upgrading to nixpkgs master #6

Closed nh2 closed 3 years ago

nh2 commented 6 years ago

The latest nixpkgs master has all fixes I had in my nixpkgs commit 50677e464359947f2a71cfd497c4022e3cdf8c7d.

So I'd like to just switch to a nixpkgs master commit.

nh2 commented 6 years ago

Trying https://github.com/nh2/nixpkgs/commit/efb033ae16841cd9da8349f81c4c50dabd1d431f, I get an issue compiling e2fsprogs because apparently the musl-based GCC doesn't understand __GNUC_PREREQ:

niklas@ares:/tmp/nix-build-e2fsprogs-1.44.4.drv-0/e2fsprogs-1.44.4/lib/e2p$ gcc -I. -I../../lib -I../../lib -I../../intl -I../../intl -g -O2 -DHAVE_CONFIG_H -c feature.c -o feature.o
In file included from ../../lib/ext2fs/ext2fs.h:97:0,
                 from feature.c:20:
../../lib/ext2fs/hashmap.h:20:19: error: missing binary operator before token "("
 #if __GNUC_PREREQ (4, 8)
                   ^
../../lib/ext2fs/hashmap.h:25:19: error: missing binary operator before token "("
 #if __GNUC_PREREQ (4, 8)
                   ^

My system gcc understands it:

niklas@ares:/tmp/nix-build-e2fsprogs-1.44.4.drv-0/e2fsprogs-1.44.4/lib/e2p$ /usr/bin/gcc -I. -I../../lib -I../../lib -I../../intl -I../../intl -g -O2 -DHAVE_CONFIG_H -c feature.c -o feature.o

CC @dtzWill

nh2 commented 6 years ago

gcc -I. -I../../lib -I../../lib -I../../intl -I../../intl -g -O2 -DHAVE_CONFIG_H -c feature.c -o feature.o in the build environment reproduces the issue immediately.

nh2 commented 6 years ago

Pretty sure the reason is the upgrade to musl 1.1.20 (https://github.com/NixOS/nixpkgs/pull/46064); the commit https://github.com/NixOS/nixpkgs/commit/e37c21ce950e8975887f2beb5e648a3ad521a1a6 itself fails with that message but the commit before it, https://github.com/NixOS/nixpkgs/commit/88ae8f7d55efa457c95187011eb410d097108445, doesn't.

dtzWill commented 6 years ago

Oh, I don't see this locally because I'm using the fix I committed to staging that addresses this :P.

Sorry about that. Can you try using that patch (however works best for you) and LMK if that fixes the issue for you (especially if it doesn't!).

This PR: https://github.com/NixOS/nixpkgs/pull/45703

nh2 commented 6 years ago

@dtzWill I see. Why did it work before though? Why does the musl update make this difference?

nh2 commented 6 years ago

On https://github.com/NixOS/nixpkgs/commit/88ae8f7d55efa457c95187011eb410d097108445 I also get a Cython build failure due to failing tests (when building sphinx, a dependency of servant):

======================================================================
ERROR: runTest (__main__.CythonRunTestCase)
compiling (cpp) and running parallel
----------------------------------------------------------------------
Traceback (most recent call last):
  File "runtests.py", line 1117, in run
    self.run_tests(result, ext_so_path)
  File "runtests.py", line 1132, in run_tests
    self.run_doctests(self.module, result, ext_so_path)
  File "runtests.py", line 1142, in run_doctests
    run_forked_test(result, run_test, self.shortDescription(), self.fork)
  File "runtests.py", line 1202, in run_forked_test
    (module_name, result_code & 255))
Exception: Tests in module 'parallel' were unexpectedly killed by signal 11

----------------------------------------------------------------------
Ran 2589 tests in 938.163s

FAILED (errors=1)
Following tests excluded because of missing dependencies on your system:
   Cython.Coverage
   run.asyncio_generators
   run.coverage_api
   run.coverage_cmd
   run.coverage_nogil
   run.numpy_pythran
   run.py35_asyncio_async_def

Note this takes quite long to run: 3066.69s user 191.27s system 220% cpu 24:36.07 total

nh2 commented 6 years ago

Perhaps related: https://github.com/NixOS/nixpkgs/blob/288f9c648f2e145f6af913b88eba0696aa288df7/pkgs/development/python-modules/Cython/default.nix#L15-L24

dtzWill commented 6 years ago

Don't suppose bumping to latest cython fixes things? O:)

Given the comments in the nix expression you linked disabling tests doesn't seem so bad BUT it'd be good to check the reasons for the failures first... :/.

nh2 commented 6 years ago

I can reproduce it in the build environment; it looks a bit cleaner there:

niklas@ares:/tmp/nix-build-python2.7-Cython-0.28.3.drv-0/Cython-0.28.3$ HOME="$NIX_BUILD_TOP" /nix/store/z722f9n43493n99xwjr545s0h39x2b33-python-2.7.15/bin/python2.7 runtests.py -j1

======================================================================
ERROR: runTest (__main__.CythonRunTestCase)
compiling (c) and running longlongindex
----------------------------------------------------------------------
Traceback (most recent call last):
  File "runtests.py", line 1117, in run
    self.run_tests(result, ext_so_path)
  File "runtests.py", line 1132, in run_tests
    self.run_doctests(self.module, result, ext_so_path)
  File "runtests.py", line 1142, in run_doctests
    run_forked_test(result, run_test, self.shortDescription(), self.fork)
  File "runtests.py", line 1195, in run_forked_test
    cid, result_code = os.waitpid(child_id, 0)
OSError: [Errno 4] Interrupted system call

======================================================================
ERROR: runTest (__main__.CythonRunTestCase)
compiling (cpp) and running parallel
----------------------------------------------------------------------
Traceback (most recent call last):
  File "runtests.py", line 1117, in run
    self.run_tests(result, ext_so_path)
  File "runtests.py", line 1132, in run_tests
    self.run_doctests(self.module, result, ext_so_path)
  File "runtests.py", line 1142, in run_doctests
    run_forked_test(result, run_test, self.shortDescription(), self.fork)
  File "runtests.py", line 1202, in run_forked_test
    (module_name, result_code & 255))
Exception: Tests in module 'parallel' were unexpectedly killed by signal 139

----------------------------------------------------------------------
Ran 10798 tests in 2871.950s

FAILED (errors=2, skipped=50)
Following tests excluded because of missing dependencies on your system:
   Cython.Coverage
   run.asyncio_generators
   run.coverage_api
   run.coverage_cmd
   run.coverage_nogil
   run.numpy_pythran
   run.py35_asyncio_async_def

ALL DONE

I get 2 failures here though, and now the signal number is different.

nh2 commented 6 years ago

The longlongindex fails nondeterministically.

The other one can be reproduced quickly like this (the last CLI argument is a match on a test name):

niklas@ares:/tmp/nix-build-python2.7-Cython-0.28.3.drv-0/Cython-0.28.3$ HOME="$NIX_BUILD_TOP" /usr/bin/time /nix/store/z722f9n43493n99xwjr545s0h39x2b33-python-2.7.15/bin/python2.7 runtests.py -j1 --verbose parallel 
...
compiling (cpp) and running parallel
----------------------------------------------------------------------
Traceback (most recent call last):
  File "runtests.py", line 1117, in run
    self.run_tests(result, ext_so_path)
  File "runtests.py", line 1132, in run_tests
    self.run_doctests(self.module, result, ext_so_path)
  File "runtests.py", line 1142, in run_doctests
    run_forked_test(result, run_test, self.shortDescription(), self.fork)
  File "runtests.py", line 1202, in run_forked_test
    (module_name, result_code & 255))
Exception: Tests in module 'parallel' were unexpectedly killed by signal 139

----------------------------------------------------------------------
Ran 110 tests in 45.450s

That's much quicker than waiting 40 minutes for all the tests.

nh2 commented 6 years ago

Workaround for the Cython issue: Disabling tests with dontCheck.

And this other workaround for another issue also works: https://github.com/pyca/pyopenssl/issues/768#issuecomment-420494198

nh2 commented 6 years ago

Getting some opendjk issues when trying to evaluate allStackageExecutables, because of:

assert stdenv.hostPlatform.libc == "glibc";

(CC @Ericson2314 @dtzWill)

niklas@ares ~/src/haskell/static-haskell-nix (git)-[master] % NIX_PATH=nixpkgs=https://github.com/NixOS/nixpkgs/archive/88ae8f7d.tar.gz nix-build --no-link survey/default.nix -A allStackageExecutables --show-trace
error: while evaluating the attribute 'nativeBuildInputs' of the derivation 'sparkle-0.7.4' at /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/pkgs/stdenv/generic/make-derivation.nix:179:11:
while evaluating the attribute 'buildInputs' of the derivation 'inline-java-0.8.4' at /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/pkgs/stdenv/generic/make-derivation.nix:179:11:
while evaluating 'getOutput' at /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/lib/attrsets.nix:452:23, called from undefined position:
while evaluating anonymous function at /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/pkgs/stdenv/generic/make-derivation.nix:127:17, called from undefined position:
while evaluating 'overrideCabal' at /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/pkgs/development/haskell-modules/lib.nix:37:24, called from /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/pkgs/development/haskell-modules/configuration-nix.nix:83:9:
while evaluating 'override' at /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/lib/customisation.nix:73:20, called from /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/pkgs/development/haskell-modules/lib.nix:37:28:
while evaluating 'makeOverridable' at /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/lib/customisation.nix:67:24, called from /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/lib/customisation.nix:73:29:
while evaluating 'drvScope' at /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/pkgs/development/haskell-modules/make-package-set.nix:88:18, called from /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/lib/customisation.nix:69:12:
while evaluating anonymous function at /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/pkgs/development/haskell-modules/hackage-packages.nix:122015:6, called from /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/pkgs/development/haskell-modules/make-package-set.nix:88:27:
while evaluating 'mkDerivation' at /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/pkgs/development/haskell-modules/lib.nix:38:20, called from /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/pkgs/development/haskell-modules/hackage-packages.nix:122018:6:
while evaluating 'override' at /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/lib/customisation.nix:73:20, called from /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/pkgs/development/haskell-modules/lib.nix:38:25:
while evaluating 'makeOverridable' at /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/lib/customisation.nix:67:24, called from /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/lib/customisation.nix:73:29:
while evaluating anonymous function at /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/pkgs/development/haskell-modules/generic-builder.nix:13:1, called from /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/lib/customisation.nix:69:12:
while evaluating 'optionalAttrs' at /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/lib/attrsets.nix:322:25, called from /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/pkgs/development/haskell-modules/generic-builder.nix:474:4:
while evaluating the attribute 'buildInputs' of the derivation 'openjdk-8u181b13' at /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/pkgs/stdenv/generic/make-derivation.nix:179:11:
while evaluating 'getOutput' at /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/lib/attrsets.nix:452:23, called from undefined position:
while evaluating anonymous function at /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/pkgs/stdenv/generic/make-derivation.nix:127:17, called from undefined position:
while evaluating 'callPackageWith' at /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/lib/customisation.nix:108:35, called from /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/pkgs/top-level/all-packages.nix:6823:13:
while evaluating 'makeOverridable' at /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/lib/customisation.nix:67:24, called from /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/lib/customisation.nix:112:8:
while evaluating anonymous function at /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/pkgs/development/compilers/openjdk/bootstrap.nix:1:1, called from /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/lib/customisation.nix:69:12:
assertion failed at /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/pkgs/development/compilers/openjdk/bootstrap.nix:7:1
nh2 commented 6 years ago

I'm also having a trouble the tests for python3.6-Cython-0.28.3, which seems to be some dependency of static-stack (perhaps via sphinx for GHC 8.2?), hangs forever in its test suite with musl.

As opposed to the above Cython test issues with Python 2.7, it doesn't show test failures, only hangs forever.

nh2 commented 6 years ago

I've now worked around all of these issues affecting the upgrade to musl-1.1.19 on my branch upstream-nixpkgs-musl-1.1.19, commit 2fe0db90677d63879f5ee0a2c2660d142758875a as of writing.

I can build the working set, most of stackage exacutables (see here) and Stack 1.7 and 1.9 prerelease.

I haven't put in time for support for newer nixpkgs with musl-1.1.20 based on nixpkgs staging yet.

Ericson2314 commented 6 years ago

@nh2 you can just remove that assert. I think i was generalizing whatever was there before. A git blame should answer conclusively.

More generally, any chance of being able to get this onto 18.09 too? I might have a few cycles to help now, finally.

dtzWill commented 6 years ago

I'm not sure openjdk was working with musl previously--something must be requiring it now that didn't previously (not at all related to 1.1.19 vs 1.1.20). If you have any additional problems you're seeing that appear related to musl (or if you're not sure, of course!) please liberally cc me so I can take a look. Hope to get things working for you soon!

nh2 commented 5 years ago

More generally, any chance of being able to get this onto 18.09 too? I might have a few cycles to help now, finally.

@Ericson2314 Which part would you like to get into 18.09? An overlay ala https://github.com/nh2/static-haskell-nix/blob/7ad63b1ab4168e4a97396dddfe59a11a5e3c6f43/survey/default.nix so that Hyda would build at least some static executables from the working attrset?

If yes, sounds like a good idea to ensure that things keep working as stuff like musl is upgraded. Where in nixpkgs should that be placed?

Ericson2314 commented 5 years ago

@nh2 Well even better than that overlay incorperate things into the packages proper; I guess I just disagree with those saying it isn't worth it to make a "static nixpkgs": there's tons of enableStatic arguments and the like already, it makes no sense not to hook them up to e.g. hostPlatform.linkage == "static" or something.

nh2 commented 5 years ago

@Ericson2314 I agree with you, and am happy to help with that effort. I think it would be awesome if I could request static exes from all kinds of executables easily via nix, and have cache.nixos.org deliver them trivially. But I don't know how to kickstart that or where it should be placed into nixpkgs.

Ericson2314 commented 5 years ago

@nh2

  1. Go to https://github.com/NixOS/nixpkgs/blob/master/lib/systems/default.nix and add some sort of linkage with "static" or "shared". (Feel free to bike shed.)

  2. Grep for enableShared options and make them enableShared ? stdenv.hostPlatform.linkage.

  3. There's this --disabled-shared logic we should probably move out of stdenv/generic/setup.sh into a setup hook (that would prepend a preConfigureHook or something). (Honestly we could maybe remove it altogether, but that's a larger job.)

nh2 commented 5 years ago

Note for myself: Probably also want https://github.com/NixOS/nixpkgs/pull/47278 via https://github.com/NixOS/nixpkgs/issues/46923#issuecomment-424036597.

nh2 commented 5 years ago

Over the last days I've been working on an update to the current nixpkgs master, on branches starting with nixpkgs-master-2019-08-10.

Problems I encountered (to be updated as I go along) and dealt with:

Extra abilities unlocked:

nh2 commented 4 years ago

I did another big master-pin update in #61.

nh2 commented 3 years ago

Another big update to nixpkgs-unstable in #107.