search

tweag / rules_haskell

Haskell rules for Bazel.
https://haskell.build
Apache License 2.0
263 stars 79 forks source link

Fully static binaries #379

Closed mboes closed 4 years ago

mboes commented 6 years ago

GHC by default creates "static" libraries. As explained in #378, this really means "mostly static" in Bazel terms, because some dependencies are still dynamically linked. It is possible, however, to create fully static binaries, provided there are static versions of all external libraries available. We should implement this link mode as well, and make it configurable per-target. We should probably stick to a similar API for this to what the CC rules already provide.

cc @lunaris

Profpatsch commented 5 years ago

nixpkgs has made major progress in this territory, meta issue: https://github.com/NixOS/nixpkgs/issues/43795

Here’s an example how dhall uses that feature: https://github.com/dhall-lang/dhall-haskell/commit/3e305ac373d7c8b2cda14d0b0b53437a3979343c

They add support to cabal to make this possible:

We don’t use cabal, but we can infer the GHC flags we need from these patches.

jkachmar commented 5 years ago

Is there anything that interested parties like myself can help contribute here?

Fully statically linked executables (in addition to everything else that's part of the rules_haskell project) would be a huge, huge win for me both personally and professionally so I'm significantly interested in helping this along.

aherrmann commented 5 years ago

It's worth noting that since https://github.com/tweag/rules_haskell/pull/587 rules_haskell links binaries "mostly static" in the sense that both haskell_library and cc_library targets will be linked statically, if possible.

It seems that what's missing to achieve full parity with the cc rules is the fully_static_link feature. Maybe this could be achieved by passing -optl-static to GHC. However, this fails if any dependency is only available as dynamic library.

mboes commented 5 years ago

Yes but a fully static mode would enable trivially deployable Haskell binaries, which is what I'm assuming @jkachmar is shooting for. I'd say the way to go about this is to enable -optl-static in compiler_flags, see what breaks, and take it from there (we might need to thread more information through the CC providers).

jkachmar commented 5 years ago

Just so, @mboes.

The added wrinkle is that it would also entail something like what @nh2 has done in the linked NixOS thread.

Special care would need to be taken to address some of the requirements that statically linked Haskell applications bring (e.g. GHC w/ musl support, integer-simple instead of integer-gmp for closed source distribution, etc.).

mboes commented 5 years ago

Right. glibc isn't designed for static linking (and neither is integer-gmp).

nh2 commented 5 years ago

and neither is integer-gmp

Does integer-gmp also have some functional problems when statically linked, or is it only the license topic on that one?

Profpatsch commented 5 years ago

Afaik only licensing (You can only link against LGPL dynamically from non-copyleft).

mboes commented 5 years ago

Does integer-gmp also have some functional problems when statically linked, or is it only the license topic on that one?

That's a question I would have asked you. ;)

jkachmar commented 5 years ago

@nh2 I'm not aware of any functional issues, I'm just concerned about it from the perspective of a commercial user not wanting to run afoul of any OSS licenses.

I just raised it as a point here because I imagine some of this would need to be "baked in" to rules_haskell, since using integer-simple requires a different GHC toolchain and some flag overrides in a few common libraries.

nh2 commented 5 years ago

@jkachmar Makes total sense, I wasn't questioning your proposal.

I just want to be very aware of anything that technically breaks under static linking, so that I don't link it in even in my fully open-source projects.

Also note there's integer-openssl as an upcoming, likely faster alternative to integer-simple.

lunaris commented 4 years ago

So, after a long hiatus, I've had some time to play with this, and I think I've got something working. At https://github.com/lunaris/minirepo I've got a minimal repository that sets up:

I've tested:

Tricks I've played:

I think this is "OK" (I'm not doing anything I shouldn't be) but fully suspect it isn't (I'm definitely doing something I shouldn't be). Take a look and let me know.

It's definitely worth saying -- this is all thanks to the works of others (@nh2, DAML, Tweag) and I'm hugely excited by it. Thanks so much for getting us to this point!

P.S. I'm aware that I'm still (I believe) using libgmp. Another thing to address.

hanshoglund commented 4 years ago

Very cool. I'll take a closer look at your repo and see if there is anything we can do upstream to make this nicer.

lunaris commented 4 years ago

Of course I spoke too soon -- now trying to get something meatier (postgresql-libpq and postgresql-simple) to build and running into some transitive dependency issues, but hopefully nothing that can't be fixed. Will post more as I have updates.

nh2 commented 4 years ago

@lunaris That's cool!

P.S. I'm aware that I'm still (I believe) using libgmp. Another thing to address.

Not sure if you've seen it already, but the use of my integer-simple flag may give some hints on what needs to be done for it. (Also, of course, my understanding of the GPL part of the LGPL is that statically linking in GPL code is fine if you only use it for deploying the linked programs internally, and that the GPL effect will only apply when you give that program to somebody else. But for the use of rules_haskell, it would be cool to be able to swap it out.)

(postgresql-libpq and postgresql-simple) to build and running into some transitive dependency issues

Do you mean something like this?

lunaris commented 4 years ago

@nh2 Alas no -- I am using your postgresql package, so that's fine (I believe :smile:). The issue is now that rules_haskell (because I've broken it, or its assumptions at least, no doubt) is missing linkable things when invoking GHC (which look to be transitive dependencies of postgresql-simple and company, like scientific or primitive).

hanshoglund commented 4 years ago

rules_haskell (because I've broken it, or its assumptions at least, no doubt) is missing linkable things

I'd like to try reproducing this, do you have a commit with the error?

lunaris commented 4 years ago

Sure -- try https://github.com/lunaris/minirepo/tree/pg and then:

$ bazel run //example-service:impl

You should get something like:

<command line>: can't load .so/.DLL for: bazel-out/k8-fastbuild/bin/example-service/../external/stackage/scientific-0.3.6.2/_install/lib/libHSscientific-0.3.6.2-ghc8.6.5.so (Error loading shared library libHSprimitive-0.6.4.0-ghc8.6.5.so: No such file or directory (needed by bazel-out/k8-fastbuild/bin/example-service/../external/stackage/scientific-0.3.6.2/_install/lib/libHSscientific-0.3.6.2-ghc8.6.5.so))

among the noise.

lunaris commented 4 years ago

@hanshoglund Not sure where you're at but from what I can tell:

Again, still not sure why GHC wants to see the dynamic libraries when being told -static. I'm probably missing something obvious. In any case, I will flesh out the full LD_LIBRARY_PATH hack to see what kind of binary I end up with (or what the next issue is).

lunaris commented 4 years ago

This appears to be an issue with the toolchain (i.e. what is being pulled from Nix) as opposed to rules_haskell. At least, if I switch out the static-haskell-nix GHC for a "vanilla" one, I get something that builds OK (although it's a totally different artifact). I wonder if something about the Musl-linked GHC means that the linking behaviour of GHC is different/missing places to look?

lunaris commented 4 years ago

Doing some more investigation on this, it seems that the RPATH being generated in my libraries was referring to the Bazel sandbox in which the library was built, which was not great. Bumping the version of rules_haskell seems to have solved this (it now points to $ORIGIN/...) but I'm still not there. For the time being I'm trying to just build a dynamic binary/library set using stack_snapshot atop libraries with C dependencies. PostgreSQL is evading me in this vein and I was wondering -- are there any examples of postgresql-simple / postgresql-libpq being used with stack_snapshot and (presumably) some nixpkgs_package of postgresql / postgresql.dev? Hopefully if I can crack that nut then I can move back to experimenting with static binaries.

As it is for now, I'll see about putting together something in which we revert to using Nix for the Hackage package set, which is not ideal but would still be an improvement for us (or anyone who values static binaries more, perhaps).

aherrmann commented 4 years ago

Doing some more investigation on this, it seems that the RPATH being generated in my libraries was referring to the Bazel sandbox in which the library was built, which was not great. Bumping the version of rules_haskell seems to have solved this (it now points to $ORIGIN/...) but I'm still not there.

Indeed, that was an issue that was resolved recently, see https://github.com/tweag/rules_haskell/issues/1130.

For the time being I'm trying to just build a dynamic binary/library set using stack_snapshot atop libraries with C dependencies. PostgreSQL is evading me in this vein and I was wondering -- are there any examples of postgresql-simple / postgresql-libpq being used with stack_snapshot and (presumably) some nixpkgs_package of postgresql / postgresql.dev? Hopefully if I can crack that nut then I can move back to experimenting with static binaries.

What issues are you encountering? In the past we had issues where pulling in libpgcommon.a caused issues that could be avoided by simply ignoring that library, as it was not required by Haskell. See https://github.com/tweag/rules_haskell/issues/1299.

I'm not terribly familiar with the static-haskell-nix GHC, does it use the static RTS? If not, maybe it could be configured to do so? rules_haskell supports GHC with static RTS on Unix, however, it needs to be explicitly configured using the is_static attribute. An example for a Nix provided GHC with static RTS is described here.

lunaris commented 4 years ago

@aherrmann Thanks for your reply -- I've tried a number of things but I don't think it's the issue you report. Bumping my Nixpkgs and rules_haskell appears to have made some things better -- zlib now works as expected -- but the errors I'm seeing from PostgreSQL (again, just trying to get dynamic binaries working here) seem unrelated to pgcommon.a. master on https://github.com/lunaris/minirepo has a commit that ensures only .so* and .dylibs are pulled in but still fails to build with messages like:

Use --sandbox_debug to see verbose messages from the sandbox                                                                       
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -l_int                                           
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -ladminpack                                        
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lamcheck                            
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lascii_and_mic                            
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lauth_delay                            
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lauto_explain                      
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lautoinc                                           
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lbloom                                             
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lbtree_gin                                     
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lbtree_gist                                     
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lcitext                             
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lcube                             
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lcyrillic_and_mic                                    
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -ldblink                                      
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -ldict_int                          
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -ldict_snowball
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -ldict_xsyn
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -learthdistance
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -leuc2004_sjis2004
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -leuc_cn_and_mic
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -leuc_jp_and_sjis
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -leuc_kr_and_mic
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -leuc_tw_and_big5
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lfile_fdw
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lfuzzystrmatch
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lhstore
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -linsert_username
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lisn
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -llatin2_and_win1250
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -llatin_and_mic
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -llo
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lltree
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lmoddatetime
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lpageinspect
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lpasswordcheck
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lpg_buffercache
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lpg_freespacemap
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lpg_prewarm
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lpg_stat_statements
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lpg_trgm
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lpg_visibility
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lpgcrypto
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lpgoutput
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lpgrowlocks
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lpgstattuple
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lpgxml
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lplpgsql
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lpostgres_fdw
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lrefint
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lseg
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lsslinfo
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -ltablefunc
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -ltcn
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -ltest_decoding
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -ltimetravel
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -ltsm_system_rows
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -ltsm_system_time
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lunaccent
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lutf8_and_ascii
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lutf8_and_big5
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lutf8_and_cyrillic
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lutf8_and_euc2004
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lutf8_and_euc_cn
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lutf8_and_euc_jp
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lutf8_and_euc_kr
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lutf8_and_euc_tw
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lutf8_and_gb18030
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lutf8_and_gbk
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lutf8_and_iso8859
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lutf8_and_iso8859_1
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lutf8_and_johab
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lutf8_and_sjis
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lutf8_and_sjis2004
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lutf8_and_uhc
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lutf8_and_win
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -luuid-ossp
/nix/store/ila0br2fsvnp60zhl652lgrjg6zhsqh8-postgresql-11.8-lib/lib/libpqwalreceiver.so: error: undefined reference to 'errcode'
/nix/store/ila0br2fsvnp60zhl652lgrjg6zhsqh8-postgresql-11.8-lib/lib/libpqwalreceiver.so: error: undefined reference to 'pfree'
/nix/store/ila0br2fsvnp60zhl652lgrjg6zhsqh8-postgresql-11.8-lib/lib/libpqwalreceiver.so: error: undefined reference to 'MemoryContextReset'
/nix/store/ila0br2fsvnp60zhl652lgrjg6zhsqh8-postgresql-11.8-lib/lib/libpqwalreceiver.so: error: undefined reference to 'TupleDescInitEntry'
/nix/store/ila0br2fsvnp60zhl652lgrjg6zhsqh8-postgresql-11.8-lib/lib/libpqwalreceiver.so: error: undefined reference to 'MyDatabaseId'
/nix/store/ila0br2fsvnp60zhl652lgrjg6zhsqh8-postgresql-11.8-lib/lib/libpqwalreceiver.so: error: undefined reference to 'DirectFunctionCall1Coll'
/nix/store/ila0br2fsvnp60zhl652lgrjg6zhsqh8-postgresql-11.8-lib/lib/libpqwalreceiver.so: error: undefined reference to 'appendStringInfoString'
/nix/store/ila0br2fsvnp60zhl652lgrjg6zhsqh8-postgresql-11.8-lib/lib/libpqwalreceiver.so: error: undefined reference to 'tuplestore_begin_heap'
/nix/store/ila0br2fsvnp60zhl652lgrjg6zhsqh8-postgresql-11.8-lib/lib/libpqwalreceiver.so: error: undefined reference to 'GetDatabaseEncodingName'
/nix/store/ila0br2fsvnp60zhl652lgrjg6zhsqh8-postgresql-11.8-lib/lib/libpqwalreceiver.so: error: undefined reference to 'errfinish'
/nix/store/ila0br2fsvnp60zhl652lgrjg6zhsqh8-postgresql-11.8-lib/lib/libpqwalreceiver.so: error: undefined reference to 'WalReceiverFunctions'
/nix/store/ila0br2fsvnp60zhl652lgrjg6zhsqh8-postgresql-11.8-lib/lib/libpqwalreceiver.so: error: undefined reference to 'errdetail'
/nix/store/ila0br2fsvnp60zhl652lgrjg6zhsqh8-postgresql-11.8-lib/lib/libpqwalreceiver.so: error: undefined reference to 'AllocSetContextCreateExtended'
/nix/store/ila0br2fsvnp60zhl652lgrjg6zhsqh8-postgresql-11.8-lib/lib/libpqwalreceiver.so: error: undefined reference to 'errmsg'
...

Any thoughts? I suspect I'm doing something silly here.

lunaris commented 4 years ago

Yes, I was being silly -- in practice it's even more specific than that -- you only want libpq.so. Restricting the c_lib rules in my example fixes it. Apologies! That said, this is still all only dynamic, so will now attempt to edge back towards fully-static binaries :smile:

lunaris commented 4 years ago

Huzzah! I think it all works -- check out https://github.com/lunaris/minirepo/tree/b0c0ca440bcbb73e76afbbdb640b23c11cf28b00

In short, and documenting many pitfalls in the hope that Google indexing helps those who come later (links and comments in the repo too):

Now I have something approaching featureful, I plan to try and integrate it into our (much larger) repository and see how it goes. But perhaps this offers a route to fully-static binaries in rules_haskell in a manner that we can make generic/offer to other users of the rules.

aherrmann commented 4 years ago

This is wonderful news, thank you @lunaris for pushing this forward and documenting your progress and the required steps! I tried out minirepo locally and it's working very well.

It's very reassuring to see that this works with a recent but otherwise unpatched rules_haskell. Though, there seem to be quite a few pitfalls to set this up correctly.

I think a good next step to improve support in rules_haskell would be to add setup instructions to the use-cases guide and to add an example to rules_haskell CI to ensure that this keeps working.

lunaris commented 4 years ago

For sure -- happy to help with documentation and the like. That said, I don't think we're quite there yet. Some more gotchas I've encountered since:

diff --git a/haskell/private/cc_wrapper.py.tpl b/haskell/private/cc_wrapper.py.tpl
index 7540ddf8..d188f04d 100644
--- a/haskell/private/cc_wrapper.py.tpl
+++ b/haskell/private/cc_wrapper.py.tpl
@@ -239,8 +239,9 @@ class Args:

         if consumed:
             # Remember the required libraries.
-            self.libraries.append(library)
-            out.append("-l{}".format(library))
+            if library != "ffi" and library != "gmp":
+                self.libraries.append(library)
+                out.append("-l{}".format(library))
/nix/store/qy56pj9kw2q0s397jl3w1csf639jqah9-binutils-2.31.1/bin/ld.gold: warning: discarding version information for __register_frame_info@GCC_3.0, defined in unused shared library /nix/store/8by85zgjmvkjaf7l8vvj1p5imb85v3v9-gcc-9.2.0-lib/lib/libgcc_s.so.1 (linked with --as-needed)
/nix/store/qy56pj9kw2q0s397jl3w1csf639jqah9-binutils-2.31.1/bin/ld.gold: warning: discarding version information for __deregister_frame_info@GCC_3.0, defined in unused shared library /nix/store/8by85zgjmvkjaf7l8vvj1p5imb85v3v9-gcc-9.2.0-lib/lib/libgcc_s.so.1 (linked with --as-needed)

which surely shouldn't happen if your GHC is using musl. This could be because my configuration of nixpkgs_cc_configure is wrong and somehow a GCC-linked ld.gold is getting through, or it could be because dynamically-linked libraries are still present (though again, I think those should be linked against musl).

Any thoughts on all this welcome and as I said, as we get to the bottom of this for sure sign me up for documentation -- it's sorely needed and would love to pitch in.

lunaris commented 4 years ago

OK, so a less hacky way of fixing the hsc2hs option is having GCC (which will be called by Cabal/Setup.hs during in the case of hslua [example hsc2hs-using library]) use static linking. E.g. in cc_wrapper.py.tpl we check if we're linking an hsc_make-ending filename and add -static to the arguments of GCC in this case.

I can't think of a condition which we'd check for this -- is_static isn't used a lot in the rules_haskell codebase and it's not clear what its use is to me (apart from stopping the generation of dynamic libraries, etc.). Might this be a time to discuss whether fully-static building is a "toolchain-level" thing? And if so, would is_static be the trigger? If so, and this kind of patch makes sense in the general case, this could be the condition we check in e.g. cc_wrapper.py.tpl.

Of course this proposal in general might be horrible -- feedback welcome. Example diff to be clear:

diff --git a/haskell/private/cc_wrapper.py.tpl b/haskell/private/cc_wrapper.py.tpl
index 7540ddf8..6e24d3eb 100644
--- a/haskell/private/cc_wrapper.py.tpl
+++ b/haskell/private/cc_wrapper.py.tpl
@@ -138,6 +138,19 @@ class Args:
                 # not speficy the required runpaths on the command-line in the
                 # context of Bazel.
                 self.rpaths.extend(self.library_paths)
+
+            # hsc2hs will call us (its configured C compiler) when generating
+            # binaries which output Haskell code. Such binaries end in the
+            # suffix "hsc_make". When we are building in a static configuration
+            # (as indicated by TODO_WHAT_DO_WE_CHECK), we want these binaries
+            # too to be statically linked, since the dynamic libraries they
+            # attempt to load won't be available at runtime when we're done
+            # building. It should be that everything we are linking together is
+            # statically linked (e.g. GHC-compiled Haskell libraries, etc.; the
+            # Haskell toolchain), and so we just need to pass `-static` to the
+            # real C compiler to produce the final binary.
+            if self.output.endswith("hsc_make"):
+                self.args.extend(["-static"])
         else:
             # We don't expect rpath arguments if not linking, however, just in
             # case, forward them if we don't mean to modify them.

(TODO_WHAT_DO_WE_CHECK could be is_static, making the condition if self.output ... and is_static, for example)

I will continue working on the other two points.

lunaris commented 4 years ago

What a journey this has been. Yet again, I think I'm "there" but I'm sure I'll be proven wrong in the coming days. Update for now for those following/landing here from a Google search.

Disclaimer: all diffs/hacks are WIPs -- lots of cleaning up and design discussions needed before these become bonafide patches

Context

You need to end up at https://github.com/tweag/rules_haskell/pull/970 and https://github.com/tweag/rules_haskell/issues/728 at some point, which both talk about pieces of the puzzle somewhat separately. In short -- GHCi and Template Haskell are going to do their best to load dynamic libraries, and you need to pull several tricks to stop this happening. These two threads talk about several of them. Thanks so much to those who wrote them!

I'm focussing on a GHC-from-Nixpkgs use case and (luckily) am only concerned with Linux right now. Bindists are likely still an open problem/I've no idea about them or Windows/macOS. In my scenario at least, there are broadly three "groups" of problems:

  1. Libraries that come bundled with GHC (base, containers, etc.) -- these are coming from Nixpkgs.

  2. Libraries that we get from Hackage/Stackage/etc. -- these are built with _ruleshaskell's Cabal support.

  3. Libraries/executables that we define and built -- these are built with _ruleshaskell's "raw" support.

Bundled libraries from Nixpkgs

  staticHaskellNixpkgs = builtins.fetchTarball
    https://github.com/nh2/static-haskell-nix/archive/dbce18f4808d27f6a51ce31585078b49c86bd2b5.tar.gz;

  staticHaskellPkgs =
    let
      p = import (staticHaskellNixpkgs + "/survey/default.nix") {};
    in
      p.approachPkgs;

  overlay = self: super: {
    staticHaskell = staticHaskellPkgs.extend (selfSH: superSH: {
      ghc = (superSH.ghc.override {
        enableRelocatedStaticLibs = true;
        enableShared = false;
      }).overrideAttrs (oldAttrs: {
        preConfigure = ''
          ${oldAttrs.preConfigure or ""}
          echo "GhcLibHcOpts += -fPIC -fexternal-dynamic-refs" >> mk/build.mk
          echo "GhcRtsHcOpts += -fPIC -fexternal-dynamic-refs" >> mk/build.mk
        '';
        postInstall = ''
          ${oldAttrs.postInstall or ""}
          # Patch the package configs shipped with GHC so that it treats Haskell
          # libraries like C libraries which allows linking against static Haskell libs from
          # TH and GHCi.
          local packageConfDir="$out/lib/${superSH.ghc.name}/package.conf.d";
          for f in $packageConfDir/*.conf; do
            filename="$(basename $f)"
              if [ "$filename" != "rts.conf" ]; then
                cp $f $f-tmp
                rm $f
                sed -e "s/hs-libraries/extra-libraries/g" $f-tmp > $f
                rm $f-tmp
              fi
          done
          $out/bin/ghc-pkg recache
        '';
      });
    });
  };

Note that here I'm overriding the GHC from @nh2's static-haskell-nix -- at this point I'm not sure how necessary this is given that I think Bazel will end up doing most of the "statification" work but it either a. doesn't hurt or b. will be easy to replace with just pkgsMusl.haskell.compiler.ghcXXX (for instance). I am pretty sure that pkgsMusl is key though -- glibc is not built for dynamic linking.

Libraries from Hackage/Stackage -- built with Cabal/Setup.hs

@@ -215,6 +220,10 @@ with tmpdir() as distdir:
 libraries=glob(os.path.join(libdir, "libHS*.a"))
 package_conf_file = os.path.join(package_database, name + ".conf")

+def make_hs_as_extra(line):
+    line = re.sub(r"hs-libraries:(.*)", r"extra-libraries:\1", line)
+    return line
+
 def make_relocatable_paths(line):
     line = re.sub("library-dirs:.*", "library-dirs: ${pkgroot}/lib", line)

@@ -235,7 +244,7 @@ if libraries != [] and os.path.isfile(package_conf_file):
     with open(package_conf_file, 'r', errors='surrogateescape') as package_conf:
         with open(tmp_package_conf_file, 'w', errors='surrogateescape') as tmp_package_conf:
             for line in package_conf.readlines():
-                print(make_relocatable_paths(line), file=tmp_package_conf)
+                print(make_hs_as_extra(make_relocatable_paths(line)), file=tmp_package_conf)
     os.remove(package_conf_file)
     os.rename(tmp_package_conf_file, package_conf_file)
     recache_db()
diff --git a/haskell/private/cabal_wrapper.py.tpl b/haskell/private/cabal_wrapper.py.tpl
index 89d0dbb7..917e20f8 100755
--- a/haskell/private/cabal_wrapper.py.tpl
+++ b/haskell/private/cabal_wrapper.py.tpl
@@ -163,6 +163,8 @@ with tmpdir() as distdir:
         "--with-gcc=" + cc,
         "--with-strip=" + strip,
         "--enable-deterministic", \
+        "--ghc-option=-fPIC", \
+        "--ghc-option=-fexternal-dynamic-refs", \
         ] +
         [ "--ghc-option=" + flag.replace("$CC", cc) for flag in %{ghc_cc_args} ] +
         enable_relocatable_flags + \
@@ -191,7 +193,10 @@ with tmpdir() as distdir:
         [ arg.replace("=", "=" + execroot + "/") for arg in path_args ] + \
         [ "--package-db=" + package_database ], # This arg must come last.
         )
-    run([runghc] + runghc_args + [setup, "build", "--verbose=0", "--builddir=" + distdir])
+    run([runghc] + runghc_args + [setup, "build", "--verbose=0", "--builddir=" +
+    distdir,
+        "--ghc-option=-fPIC",
+        "--ghc-option=-fexternal-dynamic-refs"])
     if haddock:
         run([runghc] + runghc_args + [setup, "haddock", "--verbose=0", "--builddir=" + distdir])
     run([runghc] + runghc_args + [setup, "install", "--verbose=0", "--builddir=" + distdir])

Libraries/executables that we build (finally!) using rules_haskell

@@ -97,8 +102,8 @@ def package(
         "import-dirs": [import_dir],
         "library-dirs": ["${pkgroot}"] + extra_lib_dirs,
         "dynamic-library-dirs": ["${pkgroot}"] + extra_dynamic_lib_dirs,
-        "hs-libraries": [pkg_id.library_name(hs, my_pkg_id)] if has_hs_library else [],
-        "extra-libraries": extra_libs,
+        "hs-libraries": [],
+        "extra-libraries": [pkg_id.library_name(hs, my_pkg_id)] if has_hs_library else [] + extra_libs,
         "depends": hs.package_ids,
     })

Again, no consideration for conditions here yet.

Where does this leave us

I think we're pretty comprehensive at this point. Don't forget the hsc_make hack from an earlier comment too. Next steps (perhaps):

lunaris commented 4 years ago

PR #1390 is a first stab at implementing all these things cleanly. It's just the bare bones so we can discuss whether or not we want to support all these hacks/this use case; if it's a goer we can look at tests and docs (though I've tried to make a start with some fairly heavy comments).

lunaris commented 4 years ago

I've also updated https://github.com/lunaris/minirepo with a bigger set of working examples that tests GHCi, Template Haskell, GHCIDE support, Haddocks, etc. Additionally, our large (several tens of binaries, hundreds of libraries, plenty of C-linking Hackage packages) work repository appears to all build just fine (and the executables and Docker images actually work) but it'll take a bit more time to make sure we've tested all the cases before it lands in production :smile:

aherrmann commented 4 years ago

@lunaris Thanks a lot for documenting all the steps in detail! Sorry for the late reply, I've been meaning to look into this in more depth but haven't gotten around to it, yet. I can give comments/answers to some of the points raised.

Building GHC with a static RTS has worked well in daml and sounds like the right approach to enable static only builds, i.e. what https://github.com/tweag/rules_haskell/pull/970 enables in rules_haskell and what's done in daml in #1515. This includes, as you point out, the flags -fPIC -fexternal-dynamic-refs.

Regarding the hs-libraries --> extra-libraries patching of the package configurations. I'm surprised this is necessary. In daml we disabled these patches when we switched to GHC with a static RTS and are able to build the fully Haskell project without having to build dynamic Haskell libraries. Have you tried with a GHC with a static RTS but without these patches? If so, what goes wrong without these patches?

Regarding the hs.toolchain.is_static value and the corresponding toolchain parameter. That is used to distinguish a GHC with a static RTS from one with a dynamic RTS. Admittedly, the name is imprecise, we could instead have is_static_rts and is_static if needed. Though, I'm not sure if the distinction is still necessary if it is possible to avoid the hs-libraries --> extra-libraries patching.

lunaris commented 4 years ago

That is weird -- it does seem to work without the extra-libraries hacks, which is great news! I'm not sure why I thought this wasn't the case; perhaps my GHC derivation wasn't quite right until I (coincidentally) applied the extra-libraries hacks. I'm running some tests on the larger repository now to confirm that Template Haskell and GHCi and a multitude of libraries with C bits work and if so I'll simplify the patch. That said, I still think we'll be left with the bits that pass -fexternal-dynamic-refs to Cabal and also the bit that handles hsc2hs. Maybe. All thoughts and feedback on those changes welcome!

aherrmann commented 4 years ago

That is weird -- it does seem to work without the extra-libraries hacks, which is great news!

:tada: That's great news!

That said, I still think we'll be left with the bits that pass -fexternal-dynamic-refs to Cabal

Yes, that sounds reasonable. I'm actually surprised that we don't need this in daml for the static RTS use-case. The fact that we don't do this there is, I think, just an oversight from when we switched from Hazel to stack_snapshot. So, I think adding this functionality to the is_static flag is perfectly reasonable.

That said, I still think we'll be left with [...] also the bit that handles hsc2hs.

This might also be a good thing to integrate into the is_static flag, as the static RTS use-case is also aimed at minimizing dynamic linking. I saw that the current patch addresses this with a heuristic on the output filename in cc_wrapper. Would it be possible to achieve the same at a higher level, e.g. by adding to the hcs2hs flags in haskell/private/actions/compile.bzl and by passing --hsc2hs-options to Setup.hs configure? That would seem like a cleaner and more robust approach.

lunaris commented 4 years ago

I will give the --hsc2hs-options thing a try -- good shout! One other thing which I can't really wrap my head around is the patch to haskell_repl that basically disables linking all C libraries when invoking GHCi (https://github.com/tweag/rules_haskell/pull/1390/files#diff-61e0caeb5be56438636ea710c9625111R304) -- does this make sense? In my mind I've rationalised by saying "those things are already being pulled in by virtue of the static linkage to Haskell libraries" but I'm not sure if this is right. That said, I can't get it to fail and testing code that uses C libraries (e.g. PostgreSQL connections, zlib compression) in the REPL works, so it seems OK.

aherrmann commented 4 years ago

One other thing which I can't really wrap my head around is the patch to haskell_repl that basically disables linking all C libraries when invoking GHCi (https://github.com/tweag/rules_haskell/pull/1390/files#diff-61e0caeb5be56438636ea710c9625111R304) -- does this make sense? In my mind I've rationalised by saying "those things are already being pulled in by virtue of the static linkage to Haskell libraries" but I'm not sure if this is right.

Thanks for the pointer. That doesn't look right. With static linking each library archive only contains symbols introduced by that library, symbols from transitive dependencies are not pulled in. Only when statically linking the final executable will all required symbols be bundled together in that executable. I.e. all transitive static archive dependencies need to be available. (If this wasn't the case it would lead to lots of duplicate symbols and unnecessarily inflate the size of intermediate archives.)

In the context of haskell_repl that means that all static Haskell libraries of from_binary targets and all transitive C library dependencies need to be available for GHCi's linker to load. Ideally the logic in haskell_repl should already cover the fully static use-case and not require any patching. What happens if you don't patch it?

That said, I can't get it to fail and testing code that uses C libraries (e.g. PostgreSQL connections, zlib compression) in the REPL works, so it seems OK.

That may well be by chance. The REPL is invoked via bazel run outside the sandbox, which means that it has access to outputs from prior Bazel build steps. If a prior build (either a prior bazel build or a build of a dependency of the REPL target) produced the required C library artifacts, then they will be visible to the REPL even if they are not declared dependencies.

A more precise test would be to run bazel clean; bazel run //my:repl --remote_download_toplevel with a pre-populated disk or remote cache as described in https://github.com/tweag/rules_haskell/issues/1334.

lunaris commented 4 years ago

Thanks for the pointer -- I'll run these tests and report back. That said, even if the artifacts were lying around on disk (believable), how would GHCi know to try and pick them up without appropriate -l flags (which are the things I'm removing)?

lunaris commented 4 years ago

Indeed, this is completely broken. The question then, is how to solve this error (from the example in minirepo):

➜ bazel run //example-service:impl@repl
INFO: Invocation ID: c085a8c6-8ed3-4904-ab8e-51c274873cbd
INFO: Analyzed target //example-service:impl@repl (49 packages loaded, 3089 targets configured).
INFO: Found 1 target...
Target //example-service:impl@repl up-to-date:
  bazel-bin/example-service/impl@repl@repl
INFO: Elapsed time: 1.699s, Critical Path: 0.17s
INFO: 32 processes: 30 remote cache hit, 2 linux-sandbox.
INFO: Build completed successfully, 1053 total actions
INFO: Build completed successfully, 1053 total actions
GHCi, version 8.6.5: http://www.haskell.org/ghc/  :? for help
<command line>: user specified .o/.so/.DLL could not be loaded (Error loading shared library libssl.so: No such file or directory)
Whilst trying to load:  (dynamic) ssl
Additional directories searched:   external/haskell_nixpkgs_openssl/lib
   external/haskell_nixpkgs_crypto/lib
   external/haskell_nixpkgs_postgresql/lib
   external/haskell_nixpkgs_zlib/lib
   bazel-out/k8-fastbuild/bin/_solib_k8/_U@haskell_Unixpkgs_Uopenssl_S_S_Cc_Ulib___Uexternal_Shaskell_Unixpkgs_Uopenssl_Slib
   bazel-out/k8-fastbuild/bin/_solib_k8/_U@haskell_Unixpkgs_Ucrypto_S_S_Cc_Ulib___Uexternal_Shaskell_Unixpkgs_Ucrypto_Slib
   bazel-out/k8-fastbuild/bin/_solib_k8/_U@haskell_Unixpkgs_Upostgresql_S_S_Cc_Ulib___Uexternal_Shaskell_Unixpkgs_Upostgresql_Slib
   bazel-out/k8-fastbuild/bin/_solib_k8/_U@haskell_Unixpkgs_Uzlib_S_S_Cc_Ulib___Uexternal_Shaskell_Unixpkgs_Uzlib_Slib

Does GHCi work in DAML without any issues? I know DAML is using zlib; does it use other libraries? Note that I don't think this is libssl-specific -- if I remove the -l arguments passed to GHCi one by one, it bites all C dependencies I believe (z, pq, etc.). Perhaps the issue is search directories, or some other configuration?

aherrmann commented 4 years ago

That said, even if the artifacts were lying around on disk (believable), how would GHCi know to try and pick them up without appropriate -l flags (which are the things I'm removing)?

If they are dependencies of some from_binary packages as well, e.g. anything in @stackage, then GHC will find them in these package's package-db entries.

Does GHCi work in DAML without any issues?

Yes, GHCi works in DAML. There is a static libz dependency and a dynamic libgpr/libgrpc dependency.

I haven't reproduced the issue, yet (building GHC takes a while). But, I have a suspicion. GHC lists two search paths that seem relevant for ssl:

external/haskell_nixpkgs_openssl/lib
bazel-out/k8-fastbuild/bin/_solib_k8/_U@haskell_Unixpkgs_Uopenssl_S_S_Cc_Ulib___Uexternal_Shaskell_Unixpkgs_Uopenssl_Slib

The latter would only contain the dynamic library. The former would contain the static library, but that path is relative to the execroot and won't work relative to the repository root. However, haskell_repl changes directory into the repository root. We probably need some extra path mangling in haskell_repl to fix this sort of thing.

aherrmann commented 4 years ago

Implemented in https://github.com/tweag/rules_haskell/pull/1390 :tada: