search

nh2 / static-haskell-nix

easily build most Haskell programs into fully static Linux executables
388 stars 36 forks source link

PostgREST static build #92

Closed monacoremo closed 4 years ago

monacoremo commented 4 years ago

Thank you again @nh2 for this awesome project!

Working on getting static builds of PostgREST to work here: https://github.com/monacoremo/postgrest-static-test , towards https://github.com/PostgREST/postgrest/issues/1295

Progress so far:

Did this first to make sure I got the haskellPackages overrides right (fix broken packages, add dependencies, skip tests where required).

This is basically the 'approach 1' from your #88 comment. It required adding the libpq linking fix for the postgrest package, see PR #91. This PR is of limited use by itself, as several package needed to be overridden in order to get there, including postgrest.

The tests for happy took a long time to run and never finished for me, so I disabled them in the haskellPackages overlay. Not sure if that is an overall issue or if it worked before for you. I saw that you disabled several other test suites because of this issue.

No issues - some test suites took a very long time to run. I skipped and disabled those, to be investigated.

No issues, and this allowed to remove all haskellPackages overrides but the disabled slow test suites. Going much smoother than expected! (Edit: next step is more difficult, please see comment below)

Working solution here: https://github.com/monacoremo/postgrest/blob/e6eb7750d738267e5e9cd2422c642a82957cab98/nix/static/ - bit of a hack, see the comment below.

To do:

monacoremo commented 4 years ago

@nh2 got a first real issue when switching to "ghc883":

nix-build -A dynamic works fine, but nix-build -A static errors out with: 

Installing library in /nix/store/ihj6a7rfphjkswx8l3xymqc8hj7m06fi-ghc-8.6.5-binary/lib/ghc-8.6.5/ghc-8.6.5
"/nix/store/ihj6a7rfphjkswx8l3xymqc8hj7m06fi-ghc-8.6.5-binary/lib/ghc-8.6.5/bin/ghc-pkg" --force --global-package-db "/nix/store/ihj6a7rfphjkswx8l3xymqc8hj7m06fi-ghc-8.6.5-binary/lib/ghc-8.6.5/package.conf.d" update rts/dist/package.conf.install
/nix/store/ihj6a7rfphjkswx8l3xymqc8hj7m06fi-ghc-8.6.5-binary/lib/ghc-8.6.5/bin/ghc-pkg: /nix/store/1i32nv0sm71qr3qak4jg9bqmlq4pq1la-ncurses-6.2-abi5-compat/lib/libtinfo.so.5: no version information available (required by /nix/store/ihj6a7rfphjkswx8l3xymqc8hj7m06fi-ghc-8.6.5-binary/lib/ghc-8.6.5/bin/ghc-pkg)
/nix/store/ihj6a7rfphjkswx8l3xymqc8hj7m06fi-ghc-8.6.5-binary/lib/ghc-8.6.5/bin/ghc-pkg: /nix/store/1i32nv0sm71qr3qak4jg9bqmlq4pq1la-ncurses-6.2-abi5-compat/lib/libtinfo.so.5: no version information available (required by /nix/store/ihj6a7rfphjkswx8l3xymqc8hj7m06fi-ghc-8.6.5-binary/lib/ghc-8.6.5/bin/../terminfo-0.4.1.2/libHSterminfo-0.4.1.2-ghc8.6.5.so)                                                                       
Inconsistency detected by ld.so: dl-lookup.c: 111: check_match: Assertion `version->filename == NULL || ! _dl_name_match_p (version->filename, map)' failed!                                                        
make[1]: *** [ghc.mk:991: install_packages] Error 127
make: *** [Makefile:51: install] Error 2
builder for '/nix/store/fsblpvnc3f8vsnkszx60xpn38knc1jh2-ghc-8.6.5-binary.drv' failed with exit code 2
building '/nix/store/ywiy47vvr8b9blq32kw2zcqqqpqrs7kc-ghc-8.6.5-binary.drv'...
cannot build derivation '/nix/store/baj1xbsjljr7ssad942c0h8p73wjhqwr-ghc-8.8.3.drv': 1 dependencies couldn't be built                                                                                               
cannot build derivation '/nix/store/qc575sjpkf27hlfsdagcgfnqmm0ngcac-postgrest-7.0.0.drv': 1 dependencies couldn't be built                                                                                         
error: build of '/nix/store/qc575sjpkf27hlfsdagcgfnqmm0ngcac-postgrest-7.0.0.drv' failed

Code used is on this branch: https://github.com/monacoremo/postgrest-static-test/tree/ghc883

It's a mystery to me why 8.8.3 depends on 8.6.5 at all - do you have an idea maybe? EDIT: seems to be for bootstrapping the build of GHC 8.8.3.

Is it actually required that we build a GHC on Musl? Would the regular one not also be able to generate static executables? See e.g. https://github.com/NixOS/nixpkgs/issues/57238#issuecomment-473513922

It's an upstream issue probably, submitted here: https://github.com/NixOS/nixpkgs/issues/85924

monacoremo commented 4 years ago

Second issue is with callCabal2nix - using it yields:

building '/nix/store/h05dw09rwx2hvsz1m8ah6mzcc2ngxvhs-cabal2nix-2.15.1.drv'...
setupCompilerEnvironmentPhase
Build with /nix/store/yijb2l0ajxb876993azg13hblyzlysv9-ghc-8.6.5.
unpacking sources
unpacking source archive /nix/store/4sff2d3dva935c4chid85clchsxg5kss-cabal2nix-2.15.1.tar.gz
source root is cabal2nix-2.15.1
setting SOURCE_DATE_EPOCH to timestamp 1000000000 of file cabal2nix-2.15.1/test/golden-test-cases/zlib.nix.golden
patching sources
compileBuildDriverPhase
Determined cabalPackageId as Cabal-2.4.1.0-9MZFDeNrcJI10bcroa6pq8
setupCompileFlags: -package-db=/build/setup-package.conf.d -j4 -threaded -package-id Cabal-2.4.1.0-9MZFDeNrcJI10bcroa6pq8
[1 of 1] Compiling Main             ( Setup.hs, /build/Main.o )
Linking Setup ...
updateAutotoolsGnuConfigScriptsPhase
configuring
configureFlags: --verbose --prefix=/nix/store/2jfbrdj1b9izsrfbggm3gs6rh8d5smzr-cabal2nix-2.15.1 --libdir=$prefix/lib/$compiler --libsubdir=$abi/$libname --with-gcc=gcc --package-db=/build/package.conf.d --ghc-option=-j4 --disable-split-objs --disable-library-profiling --disable-profiling --enable-shared --disable-coverage --enable-static --disable-executable-dynamic --enable-tests --disable-benchmarks --enable-library-vanilla --disable-library-for-ghci --ghc-option=-split-sections --enable-executable-static --extra-lib-dirs=/nix/store/f6caqk7a39wmgf12rx6vr2wc9nanaalf-ncurses-6.2/lib --extra-lib-dirs=/nix/store/405xvq8mjw4ykb68rhncfx2246qkqzly-ncurses-6.2/lib --extra-lib-dirs=/nix/store/l3a7r4f3riqdk4ffgfg303z0vx4i1vb8-libffi-3.3/lib --extra-include-dirs=/nix/store/mf49l865c0vqyp48dz5wi71zx2b1d00x-musl-iconv-1.1.24/include
Using Parsec parser
Configuring cabal2nix-2.15.1...
CallStack (from HasCallStack):
  die', called at ./Distribution/Simple/Configure.hs:960:20 in Cabal-2.4.1.0-9MZFDeNrcJI10bcroa6pq8:Distribution.Simple.Configure
  configureFinalizedPackage, called at ./Distribution/Simple/Configure.hs:467:12 in Cabal-2.4.1.0-9MZFDeNrcJI10bcroa6pq8:Distribution.Simple.Configure
  configure, called at ./Distribution/Simple.hs:596:20 in Cabal-2.4.1.0-9MZFDeNrcJI10bcroa6pq8:Distribution.Simple
  confHook, called at ./Distribution/Simple/UserHooks.hs:67:5 in Cabal-2.4.1.0-9MZFDeNrcJI10bcroa6pq8:Distribution.Simple.UserHooks
  configureAction, called at ./Distribution/Simple.hs:178:19 in Cabal-2.4.1.0-9MZFDeNrcJI10bcroa6pq8:Distribution.Simple
  defaultMainHelper, called at ./Distribution/Simple.hs:115:27 in Cabal-2.4.1.0-9MZFDeNrcJI10bcroa6pq8:Distribution.Simple
  defaultMain, called at Setup.hs:6:8 in main:Main
Setup: Encountered missing dependencies:
Cabal ==3.0.*

builder for '/nix/store/h05dw09rwx2hvsz1m8ah6mzcc2ngxvhs-cabal2nix-2.15.1.drv' failed with exit code 1
cannot build derivation '/nix/store/1bjyp3bjnprvxcz7xq2wr525l5i1gl3k-cabal2nix-postgrest.drv': 1 dependencies couldn't be built
error: build of '/nix/store/1bjyp3bjnprvxcz7xq2wr525l5i1gl3k-cabal2nix-postgrest.drv' failed
(use '--show-trace' to show detailed location information)

Seems that cabal2nix is not compatible with the older Cabal library - @nh2, do you have an idea on how we could fix this elegantly?

Edit: this is where we are trying to use it: https://github.com/monacoremo/postgrest/blob/c239f4f77be2e01a8612b71d96ed7225c12b6b5d/nix/static.nix#L34

Edit 2: Kind of solved this one with https://github.com/monacoremo/postgrest/blob/e6eb7750d738267e5e9cd2422c642a82957cab98/nix/static/default.nix#L33 - but not sure why this behaved differently from the dynamic build

nh2 commented 4 years ago

First: This is great progress, congrats!

Seems that cabal2nix is not compatible with the older Cabal library - @nh2, do you have an idea on how we could fix this elegantly?

We might want to ask @peti on whether cabal2nix can conceptually work only with those versions, or whether the constraint can be relaxed a bit so that you don' thave to add tihs override.

peti commented 4 years ago

Can cabal2nix conceptually work only with those versions?

cabal2nix can probably work fine with older versions of Cabal, but the package would require CPP conditionals in several places to compile because the Cabal API has changed a bit recently. Since I don't want to bother with CPP, I just specify a lower bound on the latest version (which is what I'm using anyway). Note that the latest version of cabal2nix requires Cabal-3.2.x, even.

nh2 commented 4 years ago

I forgot to post here: Yesterday I made a small PR to cabal2nix (https://github.com/NixOS/cabal2nix/pull/453) that required only a very small amount of CPP to make it compatible with 2 versions, so it seems conceptually possible, and this type of problem might disappear in the future, making it easier to use cabal2nix across compiler versions.

monacoremo commented 4 years ago

PostgREST now has a working static build in CI, which solves a lot of problems. Many thanks for this awesome project @nh2 !

It works great on the current nixpkgs-unstable and with ghc883 (the latter currently requires a patch that reverts ghc865-binary to ghc863-binary). Cabal2nix is still a bit tricky, but we got it to work by pinning Cabal 3.0.0.0. Not sure why it does not work with the default Cabal.

nh2 commented 4 years ago

That's awesome congrats!