Closed mboes closed 4 years ago
nixpkgs has made major progress in this territory, meta issue: https://github.com/NixOS/nixpkgs/issues/43795
Here’s an example how dhall uses that feature: https://github.com/dhall-lang/dhall-haskell/commit/3e305ac373d7c8b2cda14d0b0b53437a3979343c
They add support to cabal to make this possible:
We don’t use cabal, but we can infer the GHC flags we need from these patches.
Is there anything that interested parties like myself can help contribute here?
Fully statically linked executables (in addition to everything else that's part of the rules_haskell
project) would be a huge, huge win for me both personally and professionally so I'm significantly interested in helping this along.
It's worth noting that since https://github.com/tweag/rules_haskell/pull/587 rules_haskell links binaries "mostly static" in the sense that both haskell_library
and cc_library
targets will be linked statically, if possible.
It seems that what's missing to achieve full parity with the cc rules is the fully_static_link
feature. Maybe this could be achieved by passing -optl-static
to GHC. However, this fails if any dependency is only available as dynamic library.
Yes but a fully static mode would enable trivially deployable Haskell binaries, which is what I'm assuming @jkachmar is shooting for. I'd say the way to go about this is to enable -optl-static
in compiler_flags
, see what breaks, and take it from there (we might need to thread more information through the CC providers).
Just so, @mboes.
The added wrinkle is that it would also entail something like what @nh2 has done in the linked NixOS thread.
Special care would need to be taken to address some of the requirements that statically linked Haskell applications bring (e.g. GHC w/ musl
support, integer-simple
instead of integer-gmp
for closed source distribution, etc.).
Right. glibc
isn't designed for static linking (and neither is integer-gmp
).
and neither is
integer-gmp
Does integer-gmp
also have some functional problems when statically linked, or is it only the license topic on that one?
Afaik only licensing (You can only link against LGPL dynamically from non-copyleft).
Does
integer-gmp
also have some functional problems when statically linked, or is it only the license topic on that one?
That's a question I would have asked you. ;)
@nh2 I'm not aware of any functional issues, I'm just concerned about it from the perspective of a commercial user not wanting to run afoul of any OSS licenses.
I just raised it as a point here because I imagine some of this would need to be "baked in" to rules_haskell
, since using integer-simple
requires a different GHC toolchain and some flag overrides in a few common libraries.
@jkachmar Makes total sense, I wasn't questioning your proposal.
I just want to be very aware of anything that technically breaks under static linking, so that I don't link it in even in my fully open-source projects.
Also note there's integer-openssl
as an upcoming, likely faster alternative to integer-simple
.
So, after a long hiatus, I've had some time to play with this, and I think I've got something working. At https://github.com/lunaris/minirepo I've got a minimal repository that sets up:
Bazel, rules_nixpkgs
, rules_haskell
, rules_docker
and other supporting stuff (rules_go
, rules_python
, rules_sh
-- I've not played with these in anger though as I say; they're just there to make the others work/hermetic).
Two Nixpkgs pins -- one for a shell.nix
/ direnv
/ some of the support (Python, Go) Bazel tooling and one for the Haskell package set used by Bazel. The first uses a "vanilla" Nixpkgs. The second uses @nh2's static-haskell-nix
to provide a GHC and package set compiled statically/supporting static compilation through Musl.
rules_haskell
is set up to use a toolchain backed by the static-haskell-nix
GHC.
stack_snapshot
is used to pull Hackage dependencies in. Libraries needing C bits get their C bits from the approachPkgs
(nominally pkgsMusl
) provided by the static-haskell-nix
pin. So far I've only tested zlib
but will move on to PostgreSQL shortly.
C bits build both static (building) and dynamic (GHCi) bits. The same is true for haskell_library
calls. This is why I've not set is_static
on the toolchain. This may be incorrect but it seems to work for now.
haskell_binary
can then be passed the classic one-two of -optl-static
, -optl-pthread
to get a fully-static executable (including C bits). This can also be passed to cc_image
(for example) to get a working distroless
Docker container with no faffing about base layers.
I've tested:
-optl-static
, -optl-pthread
from haskell_binary
)zlib
).haskell_binary
target (even with C bits)haskell_binary
target (even if that TH targets a C-bit thing e.g. zlib
).haskell_library
target (even with C bits)Tricks I've played:
nixpkgs_cc_configure
to use the Musl GCC toolchain. If you don't do this, no dice. I think this is correct behaviour (it's not hermetic otherwise) but just calling it out.zlib_both
for just this case!) -- this is a hack to appease GHCi, which will e.g. always pass -lz
even though the zlib
Haskell library has the C bits statically linked in (so it links in something it doesn't use). This could probably be cleaned up but seems to work fine and as a bonus doesn't require me patching rules_haskell
immediately :)is_static = True
? Not sure but it seems to work.I think this is "OK" (I'm not doing anything I shouldn't be) but fully suspect it isn't (I'm definitely doing something I shouldn't be). Take a look and let me know.
It's definitely worth saying -- this is all thanks to the works of others (@nh2, DAML, Tweag) and I'm hugely excited by it. Thanks so much for getting us to this point!
P.S. I'm aware that I'm still (I believe) using libgmp
. Another thing to address.
Very cool. I'll take a closer look at your repo and see if there is anything we can do upstream to make this nicer.
Of course I spoke too soon -- now trying to get something meatier (postgresql-libpq
and postgresql-simple
) to build and running into some transitive dependency issues, but hopefully nothing that can't be fixed. Will post more as I have updates.
@lunaris That's cool!
P.S. I'm aware that I'm still (I believe) using
libgmp
. Another thing to address.
Not sure if you've seen it already, but the use of my integer-simple
flag may give some hints on what needs to be done for it. (Also, of course, my understanding of the GPL part of the LGPL is that statically linking in GPL code is fine if you only use it for deploying the linked programs internally, and that the GPL effect will only apply when you give that program to somebody else. But for the use of rules_haskell
, it would be cool to be able to swap it out.)
(
postgresql-libpq
andpostgresql-simple
) to build and running into some transitive dependency issues
Do you mean something like this?
@nh2 Alas no -- I am using your postgresql
package, so that's fine (I believe :smile:). The issue is now that rules_haskell
(because I've broken it, or its assumptions at least, no doubt) is missing linkable things when invoking GHC (which look to be transitive dependencies of postgresql-simple
and company, like scientific
or primitive
).
rules_haskell (because I've broken it, or its assumptions at least, no doubt) is missing linkable things
I'd like to try reproducing this, do you have a commit with the error?
Sure -- try https://github.com/lunaris/minirepo/tree/pg and then:
$ bazel run //example-service:impl
You should get something like:
<command line>: can't load .so/.DLL for: bazel-out/k8-fastbuild/bin/example-service/../external/stackage/scientific-0.3.6.2/_install/lib/libHSscientific-0.3.6.2-ghc8.6.5.so (Error loading shared library libHSprimitive-0.6.4.0-ghc8.6.5.so: No such file or directory (needed by bazel-out/k8-fastbuild/bin/example-service/../external/stackage/scientific-0.3.6.2/_install/lib/libHSscientific-0.3.6.2-ghc8.6.5.so))
among the noise.
@hanshoglund Not sure where you're at but from what I can tell:
-static
, GHC still wants the dynamic libraries for the transitive dependency set.compile-package_env-impl
(where impl
is the target name) contains the correct package-db
entries for the full transitive dependency set, in order.-L
, perhaps?) for GHC. If I try something like LD_LIBRARY_PATH=bazel-out/.../..._install/lib bazel-out/...ghc-wrapper ...
for a set of libraries, I can push the build along manually.Again, still not sure why GHC wants to see the dynamic libraries when being told -static
. I'm probably missing something obvious. In any case, I will flesh out the full LD_LIBRARY_PATH
hack to see what kind of binary I end up with (or what the next issue is).
This appears to be an issue with the toolchain (i.e. what is being pulled from Nix) as opposed to rules_haskell. At least, if I switch out the static-haskell-nix
GHC for a "vanilla" one, I get something that builds OK (although it's a totally different artifact). I wonder if something about the Musl-linked GHC means that the linking behaviour of GHC is different/missing places to look?
Doing some more investigation on this, it seems that the RPATH
being generated in my libraries was referring to the Bazel sandbox in which the library was built, which was not great. Bumping the version of rules_haskell seems to have solved this (it now points to $ORIGIN/...
) but I'm still not there. For the time being I'm trying to just build a dynamic binary/library set using stack_snapshot
atop libraries with C dependencies. PostgreSQL is evading me in this vein and I was wondering -- are there any examples of postgresql-simple
/ postgresql-libpq
being used with stack_snapshot
and (presumably) some nixpkgs_package
of postgresql
/ postgresql.dev
? Hopefully if I can crack that nut then I can move back to experimenting with static binaries.
As it is for now, I'll see about putting together something in which we revert to using Nix for the Hackage package set, which is not ideal but would still be an improvement for us (or anyone who values static binaries more, perhaps).
Doing some more investigation on this, it seems that the
RPATH
being generated in my libraries was referring to the Bazel sandbox in which the library was built, which was not great. Bumping the version of rules_haskell seems to have solved this (it now points to$ORIGIN/...
) but I'm still not there.
Indeed, that was an issue that was resolved recently, see https://github.com/tweag/rules_haskell/issues/1130.
For the time being I'm trying to just build a dynamic binary/library set using
stack_snapshot
atop libraries with C dependencies. PostgreSQL is evading me in this vein and I was wondering -- are there any examples ofpostgresql-simple
/postgresql-libpq
being used withstack_snapshot
and (presumably) somenixpkgs_package
ofpostgresql
/postgresql.dev
? Hopefully if I can crack that nut then I can move back to experimenting with static binaries.
What issues are you encountering? In the past we had issues where pulling in libpgcommon.a
caused issues that could be avoided by simply ignoring that library, as it was not required by Haskell. See https://github.com/tweag/rules_haskell/issues/1299.
I'm not terribly familiar with the static-haskell-nix
GHC, does it use the static RTS? If not, maybe it could be configured to do so? rules_haskell supports GHC with static RTS on Unix, however, it needs to be explicitly configured using the is_static
attribute. An example for a Nix provided GHC with static RTS is described here.
@aherrmann Thanks for your reply -- I've tried a number of things but I don't think it's the issue you report. Bumping my Nixpkgs and rules_haskell appears to have made some things better -- zlib
now works as expected -- but the errors I'm seeing from PostgreSQL (again, just trying to get dynamic binaries working here) seem unrelated to pgcommon.a
. master
on https://github.com/lunaris/minirepo has a commit that ensures only .so*
and .dylib
s are pulled in but still fails to build with messages like:
Use --sandbox_debug to see verbose messages from the sandbox
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -l_int
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -ladminpack
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lamcheck
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lascii_and_mic
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lauth_delay
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lauto_explain
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lautoinc
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lbloom
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lbtree_gin
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lbtree_gist
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lcitext
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lcube
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lcyrillic_and_mic
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -ldblink
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -ldict_int
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -ldict_snowball
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -ldict_xsyn
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -learthdistance
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -leuc2004_sjis2004
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -leuc_cn_and_mic
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -leuc_jp_and_sjis
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -leuc_kr_and_mic
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -leuc_tw_and_big5
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lfile_fdw
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lfuzzystrmatch
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lhstore
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -linsert_username
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lisn
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -llatin2_and_win1250
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -llatin_and_mic
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -llo
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lltree
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lmoddatetime
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lpageinspect
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lpasswordcheck
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lpg_buffercache
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lpg_freespacemap
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lpg_prewarm
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lpg_stat_statements
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lpg_trgm
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lpg_visibility
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lpgcrypto
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lpgoutput
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lpgrowlocks
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lpgstattuple
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lpgxml
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lplpgsql
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lpostgres_fdw
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lrefint
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lseg
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lsslinfo
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -ltablefunc
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -ltcn
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -ltest_decoding
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -ltimetravel
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -ltsm_system_rows
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -ltsm_system_time
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lunaccent
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lutf8_and_ascii
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lutf8_and_big5
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lutf8_and_cyrillic
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lutf8_and_euc2004
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lutf8_and_euc_cn
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lutf8_and_euc_jp
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lutf8_and_euc_kr
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lutf8_and_euc_tw
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lutf8_and_gb18030
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lutf8_and_gbk
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lutf8_and_iso8859
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lutf8_and_iso8859_1
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lutf8_and_johab
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lutf8_and_sjis
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lutf8_and_sjis2004
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lutf8_and_uhc
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -lutf8_and_win
/nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/ld.gold: error: cannot find -luuid-ossp
/nix/store/ila0br2fsvnp60zhl652lgrjg6zhsqh8-postgresql-11.8-lib/lib/libpqwalreceiver.so: error: undefined reference to 'errcode'
/nix/store/ila0br2fsvnp60zhl652lgrjg6zhsqh8-postgresql-11.8-lib/lib/libpqwalreceiver.so: error: undefined reference to 'pfree'
/nix/store/ila0br2fsvnp60zhl652lgrjg6zhsqh8-postgresql-11.8-lib/lib/libpqwalreceiver.so: error: undefined reference to 'MemoryContextReset'
/nix/store/ila0br2fsvnp60zhl652lgrjg6zhsqh8-postgresql-11.8-lib/lib/libpqwalreceiver.so: error: undefined reference to 'TupleDescInitEntry'
/nix/store/ila0br2fsvnp60zhl652lgrjg6zhsqh8-postgresql-11.8-lib/lib/libpqwalreceiver.so: error: undefined reference to 'MyDatabaseId'
/nix/store/ila0br2fsvnp60zhl652lgrjg6zhsqh8-postgresql-11.8-lib/lib/libpqwalreceiver.so: error: undefined reference to 'DirectFunctionCall1Coll'
/nix/store/ila0br2fsvnp60zhl652lgrjg6zhsqh8-postgresql-11.8-lib/lib/libpqwalreceiver.so: error: undefined reference to 'appendStringInfoString'
/nix/store/ila0br2fsvnp60zhl652lgrjg6zhsqh8-postgresql-11.8-lib/lib/libpqwalreceiver.so: error: undefined reference to 'tuplestore_begin_heap'
/nix/store/ila0br2fsvnp60zhl652lgrjg6zhsqh8-postgresql-11.8-lib/lib/libpqwalreceiver.so: error: undefined reference to 'GetDatabaseEncodingName'
/nix/store/ila0br2fsvnp60zhl652lgrjg6zhsqh8-postgresql-11.8-lib/lib/libpqwalreceiver.so: error: undefined reference to 'errfinish'
/nix/store/ila0br2fsvnp60zhl652lgrjg6zhsqh8-postgresql-11.8-lib/lib/libpqwalreceiver.so: error: undefined reference to 'WalReceiverFunctions'
/nix/store/ila0br2fsvnp60zhl652lgrjg6zhsqh8-postgresql-11.8-lib/lib/libpqwalreceiver.so: error: undefined reference to 'errdetail'
/nix/store/ila0br2fsvnp60zhl652lgrjg6zhsqh8-postgresql-11.8-lib/lib/libpqwalreceiver.so: error: undefined reference to 'AllocSetContextCreateExtended'
/nix/store/ila0br2fsvnp60zhl652lgrjg6zhsqh8-postgresql-11.8-lib/lib/libpqwalreceiver.so: error: undefined reference to 'errmsg'
...
Any thoughts? I suspect I'm doing something silly here.
Yes, I was being silly -- in practice it's even more specific than that -- you only want libpq.so
. Restricting the c_lib
rules in my example fixes it. Apologies! That said, this is still all only dynamic, so will now attempt to edge back towards fully-static binaries :smile:
Huzzah! I think it all works -- check out https://github.com/lunaris/minirepo/tree/b0c0ca440bcbb73e76afbbdb640b23c11cf28b00
In short, and documenting many pitfalls in the hope that Google indexing helps those who come later (links and comments in the repo too):
Use static-haskell-nix
, which ensures that GHC is patched to use static builds of things like libgmp
, libffi
, etc. This is key. I tried for a while to use latest Nixpkgs and pkgsMusl
, which goes a lot of the way, but eventually ran into GMP errors that suggested I needed to patch the ghc
derivation as @nh2 had already done. Thus, I dropped back to this :+1:
For packages that come in for linking in e.g. extra_deps
of stack_snapshot
, ensure that they have both dynamic and static libraries. The former make it possible to do GHCi/Template Haskell without patching the GHC build further (e.g. as I think DAML have done). The latter are for building fully-static binaries in the end. I imagine, as demonstrated by DAML that it would be possible to only use static libraries here, but I've not tested it and am inclined to say this is "low risk" given the provenance of both static and dynamic libraries is quite tightly controlled. Open to feedback on this though. static-haskell-nix
already provides zlib_both
in this vein; I patched in openssl_both
for similar reasons.
Make sure we're using a recent version of rules_haskell
that makes RPATH
s work correctly -- this held me up for a bit.
libssl.a
and libcrypto.a
must be linked in a specific order. There are probably many ways to make this happen, but I resorted to having two Bazel rules for pulling them in (though both from the same Nixpkgs derivation) and relying on the fact that extra_deps
takes ordered lists to get the right behaviour.
Be very careful with libraries you pull in from a C dependency -- e.g. for PostgreSQL you really only want libpq.*
-- anything else creates spurious linker dependencies that will fail when it comes to execute them.
Now I have something approaching featureful, I plan to try and integrate it into our (much larger) repository and see how it goes. But perhaps this offers a route to fully-static binaries in rules_haskell
in a manner that we can make generic/offer to other users of the rules.
This is wonderful news, thank you @lunaris for pushing this forward and documenting your progress and the required steps! I tried out minirepo
locally and it's working very well.
It's very reassuring to see that this works with a recent but otherwise unpatched rules_haskell. Though, there seem to be quite a few pitfalls to set this up correctly.
I think a good next step to improve support in rules_haskell would be to add setup instructions to the use-cases guide and to add an example to rules_haskell CI to ensure that this keeps working.
For sure -- happy to help with documentation and the like. That said, I don't think we're quite there yet. Some more gotchas I've encountered since:
hsc2hs
I believe fall foul of the fact that hsc2hs
produces code-generating executables that are dynamically linked (or at least, are being passed flags as though they were). For instance, trying to build @stackage//:hslua
can fail because a generated _hsc_make
executable can't load libffi
or libgmp
. I think the executable is actually statically linked (ldd
says so at least) but rules_haskell
is passing flags that upset it; this completely hacky diff seemed to work at least:diff --git a/haskell/private/cc_wrapper.py.tpl b/haskell/private/cc_wrapper.py.tpl
index 7540ddf8..d188f04d 100644
--- a/haskell/private/cc_wrapper.py.tpl
+++ b/haskell/private/cc_wrapper.py.tpl
@@ -239,8 +239,9 @@ class Args:
if consumed:
# Remember the required libraries.
- self.libraries.append(library)
- out.append("-l{}".format(library))
+ if library != "ffi" and library != "gmp":
+ self.libraries.append(library)
+ out.append("-l{}".format(library))
It's still ugly that we have both static and dynamic libraries to appease GHCi and Template Haskell (at least, these are the use cases I understand need to be appeased). It seems that DAML are avoiding this with something like https://github.com/digital-asset/daml/blob/master/nix/with-packages-wrapper.nix#L2, but copying this technique doesn't appear to be sufficient on its own (perhaps I am missing something).
Even in the working bits of minirepo
, you get warnings during build about libgcc
like:
/nix/store/qy56pj9kw2q0s397jl3w1csf639jqah9-binutils-2.31.1/bin/ld.gold: warning: discarding version information for __register_frame_info@GCC_3.0, defined in unused shared library /nix/store/8by85zgjmvkjaf7l8vvj1p5imb85v3v9-gcc-9.2.0-lib/lib/libgcc_s.so.1 (linked with --as-needed)
/nix/store/qy56pj9kw2q0s397jl3w1csf639jqah9-binutils-2.31.1/bin/ld.gold: warning: discarding version information for __deregister_frame_info@GCC_3.0, defined in unused shared library /nix/store/8by85zgjmvkjaf7l8vvj1p5imb85v3v9-gcc-9.2.0-lib/lib/libgcc_s.so.1 (linked with --as-needed)
which surely shouldn't happen if your GHC is using musl. This could be because my configuration of nixpkgs_cc_configure
is wrong and somehow a GCC-linked ld.gold
is getting through, or it could be because dynamically-linked libraries are still present (though again, I think those should be linked against musl).
Any thoughts on all this welcome and as I said, as we get to the bottom of this for sure sign me up for documentation -- it's sorely needed and would love to pitch in.
OK, so a less hacky way of fixing the hsc2hs
option is having GCC (which will be called by Cabal/Setup.hs
during in the case of hslua
[example hsc2hs
-using library]) use static linking. E.g. in cc_wrapper.py.tpl
we check if we're linking an hsc_make
-ending filename and add -static
to the arguments of GCC in this case.
I can't think of a condition which we'd check for this -- is_static
isn't used a lot in the rules_haskell
codebase and it's not clear what its use is to me (apart from stopping the generation of dynamic libraries, etc.). Might this be a time to discuss whether fully-static building is a "toolchain-level" thing? And if so, would is_static
be the trigger? If so, and this kind of patch makes sense in the general case, this could be the condition we check in e.g. cc_wrapper.py.tpl
.
Of course this proposal in general might be horrible -- feedback welcome. Example diff to be clear:
diff --git a/haskell/private/cc_wrapper.py.tpl b/haskell/private/cc_wrapper.py.tpl
index 7540ddf8..6e24d3eb 100644
--- a/haskell/private/cc_wrapper.py.tpl
+++ b/haskell/private/cc_wrapper.py.tpl
@@ -138,6 +138,19 @@ class Args:
# not speficy the required runpaths on the command-line in the
# context of Bazel.
self.rpaths.extend(self.library_paths)
+
+ # hsc2hs will call us (its configured C compiler) when generating
+ # binaries which output Haskell code. Such binaries end in the
+ # suffix "hsc_make". When we are building in a static configuration
+ # (as indicated by TODO_WHAT_DO_WE_CHECK), we want these binaries
+ # too to be statically linked, since the dynamic libraries they
+ # attempt to load won't be available at runtime when we're done
+ # building. It should be that everything we are linking together is
+ # statically linked (e.g. GHC-compiled Haskell libraries, etc.; the
+ # Haskell toolchain), and so we just need to pass `-static` to the
+ # real C compiler to produce the final binary.
+ if self.output.endswith("hsc_make"):
+ self.args.extend(["-static"])
else:
# We don't expect rpath arguments if not linking, however, just in
# case, forward them if we don't mean to modify them.
(TODO_WHAT_DO_WE_CHECK
could be is_static
, making the condition if self.output ... and is_static
, for example)
I will continue working on the other two points.
What a journey this has been. Yet again, I think I'm "there" but I'm sure I'll be proven wrong in the coming days. Update for now for those following/landing here from a Google search.
Disclaimer: all diffs/hacks are WIPs -- lots of cleaning up and design discussions needed before these become bonafide patches
You need to end up at https://github.com/tweag/rules_haskell/pull/970 and https://github.com/tweag/rules_haskell/issues/728 at some point, which both talk about pieces of the puzzle somewhat separately. In short -- GHCi and Template Haskell are going to do their best to load dynamic libraries, and you need to pull several tricks to stop this happening. These two threads talk about several of them. Thanks so much to those who wrote them!
I'm focussing on a GHC-from-Nixpkgs use case and (luckily) am only concerned with Linux right now. Bindists are likely still an open problem/I've no idea about them or Windows/macOS. In my scenario at least, there are broadly three "groups" of problems:
Libraries that come bundled with GHC (base
, containers
, etc.) -- these are coming from Nixpkgs.
Libraries that we get from Hackage/Stackage/etc. -- these are built with _ruleshaskell's Cabal support.
Libraries/executables that we define and built -- these are built with _ruleshaskell's "raw" support.
These are going to get compiled as part of the GHC derivation you target, so you need to patch such a derivation to make sure that the libraries are a. compiled statically with the right flags (-fPIC
, -fexternal-dynamic-refs
) and b. described in a manner that means GHCi will think they are C libraries (using extra-libraries
instead of hs-libraries
, which is the trick to getting static loading).
@aherrmann's snippet in #970 sorts much of this out. Passing enableRelocatedStaticLibs = true
and enableShared = false
sets the scene for GHCi/Template Haskell being willing to load static libraries; the cheeky preConfigure
patch adds the -fexternal-dynamic-refs
flag that is necessary to avoid some nasty R_X86_64_PC32
relocation errors which you'll get if your GHC uses -fPIC
alone. You'll want to go one step further and hack the package.conf
files for the libraries coming from Nixpkgs so that they use extra-libraries
instead of hs-libraries
(this is (3) in #728).
If you are using stack_snapshot
(as I am) then it's probably the case that the only libraries coming from Nixpkgs are those bundled with GHC. You can thus get away with a postInstall
step in your custom GHC derivation that patches the package.conf
files and recaches packages.
If you are using packages other than those bundled with GHC from Nixpkgs (e.g. you're going to use ghcWithPackages
instead of e.g. stack_snapshot
), then you're going to need something more complicated -- https://github.com/digital-asset/daml/blob/master/nix/with-packages-wrapper.nix and its use in https://github.com/digital-asset/daml/blob/master/nix/ghc.nix are I think of interest (in which a complete haskellPackages
replacement is constructed with a hacked withPackages
wrapper that auto-patches any library that goes through it). I've not tried this though.
That should be it! In the "simple" postInstall
case (so you're using stack_snapshot
and a bare-minimum amount of Nixpkgs GHC tooling):
staticHaskellNixpkgs = builtins.fetchTarball
https://github.com/nh2/static-haskell-nix/archive/dbce18f4808d27f6a51ce31585078b49c86bd2b5.tar.gz;
staticHaskellPkgs =
let
p = import (staticHaskellNixpkgs + "/survey/default.nix") {};
in
p.approachPkgs;
overlay = self: super: {
staticHaskell = staticHaskellPkgs.extend (selfSH: superSH: {
ghc = (superSH.ghc.override {
enableRelocatedStaticLibs = true;
enableShared = false;
}).overrideAttrs (oldAttrs: {
preConfigure = ''
${oldAttrs.preConfigure or ""}
echo "GhcLibHcOpts += -fPIC -fexternal-dynamic-refs" >> mk/build.mk
echo "GhcRtsHcOpts += -fPIC -fexternal-dynamic-refs" >> mk/build.mk
'';
postInstall = ''
${oldAttrs.postInstall or ""}
# Patch the package configs shipped with GHC so that it treats Haskell
# libraries like C libraries which allows linking against static Haskell libs from
# TH and GHCi.
local packageConfDir="$out/lib/${superSH.ghc.name}/package.conf.d";
for f in $packageConfDir/*.conf; do
filename="$(basename $f)"
if [ "$filename" != "rts.conf" ]; then
cp $f $f-tmp
rm $f
sed -e "s/hs-libraries/extra-libraries/g" $f-tmp > $f
rm $f-tmp
fi
done
$out/bin/ghc-pkg recache
'';
});
});
};
Note that here I'm overriding the GHC from @nh2's static-haskell-nix
-- at this point I'm not sure how necessary this is given that I think Bazel will end up doing most of the "statification" work but it either a. doesn't hurt or b. will be easy to replace with just pkgsMusl.haskell.compiler.ghcXXX
(for instance). I am pretty sure that pkgsMusl
is key though -- glibc
is not built for dynamic linking.
Setup.hs
rules_haskell
is going to need some patching here -- I hit quite a few gotchas that I think (happy to be wrong!) are features we could support. Most commonly -- we need to ensure that when Cabal calls GHC, it is playing the same tricks that our custom derivation hack and raw Bazel calls are playing (that is -fexternal-dynamic-refs
, rewriting hs-libraries
to extra-libraries
, etc.). Based on my previous comment, I feel like hanging this all off is_static
(or something similarly "toolchain global") is the way to go here, but would love to hear others' thoughts.
Patch 1 -- cabal_wrapper.py.tpl
already rewrites generated package.conf
files, so it's relatively simple to slip in another transformation to pull off the hs-libraries
to extra-libraries
trick:
@@ -215,6 +220,10 @@ with tmpdir() as distdir:
libraries=glob(os.path.join(libdir, "libHS*.a"))
package_conf_file = os.path.join(package_database, name + ".conf")
+def make_hs_as_extra(line):
+ line = re.sub(r"hs-libraries:(.*)", r"extra-libraries:\1", line)
+ return line
+
def make_relocatable_paths(line):
line = re.sub("library-dirs:.*", "library-dirs: ${pkgroot}/lib", line)
@@ -235,7 +244,7 @@ if libraries != [] and os.path.isfile(package_conf_file):
with open(package_conf_file, 'r', errors='surrogateescape') as package_conf:
with open(tmp_package_conf_file, 'w', errors='surrogateescape') as tmp_package_conf:
for line in package_conf.readlines():
- print(make_relocatable_paths(line), file=tmp_package_conf)
+ print(make_hs_as_extra(make_relocatable_paths(line)), file=tmp_package_conf)
os.remove(package_conf_file)
os.rename(tmp_package_conf_file, package_conf_file)
recache_db()
cabal_wrapper.py.tpl
, we need to make sure that when Setup.hs
calls GHC, it's got our back when it comes to hitting -fPIC
and -fexternal-dynamic-refs
. I don't think we want to forward compiler flags in general (these are third-party packages, after all) but these two are important. Again, conditions to be decided but here's the ugly, ugly, ugly hard-code for my use case:diff --git a/haskell/private/cabal_wrapper.py.tpl b/haskell/private/cabal_wrapper.py.tpl
index 89d0dbb7..917e20f8 100755
--- a/haskell/private/cabal_wrapper.py.tpl
+++ b/haskell/private/cabal_wrapper.py.tpl
@@ -163,6 +163,8 @@ with tmpdir() as distdir:
"--with-gcc=" + cc,
"--with-strip=" + strip,
"--enable-deterministic", \
+ "--ghc-option=-fPIC", \
+ "--ghc-option=-fexternal-dynamic-refs", \
] +
[ "--ghc-option=" + flag.replace("$CC", cc) for flag in %{ghc_cc_args} ] +
enable_relocatable_flags + \
@@ -191,7 +193,10 @@ with tmpdir() as distdir:
[ arg.replace("=", "=" + execroot + "/") for arg in path_args ] + \
[ "--package-db=" + package_database ], # This arg must come last.
)
- run([runghc] + runghc_args + [setup, "build", "--verbose=0", "--builddir=" + distdir])
+ run([runghc] + runghc_args + [setup, "build", "--verbose=0", "--builddir=" +
+ distdir,
+ "--ghc-option=-fPIC",
+ "--ghc-option=-fexternal-dynamic-refs"])
if haddock:
run([runghc] + runghc_args + [setup, "haddock", "--verbose=0", "--builddir=" + distdir])
run([runghc] + runghc_args + [setup, "install", "--verbose=0", "--builddir=" + distdir])
rules_haskell
hs-libraries
to extra-libraries
trick when we generate package.conf
files for the packages we define with haskell_library
, haskell_binary
, etc. private/actions/package.bzl
is the place to be:@@ -97,8 +102,8 @@ def package(
"import-dirs": [import_dir],
"library-dirs": ["${pkgroot}"] + extra_lib_dirs,
"dynamic-library-dirs": ["${pkgroot}"] + extra_dynamic_lib_dirs,
- "hs-libraries": [pkg_id.library_name(hs, my_pkg_id)] if has_hs_library else [],
- "extra-libraries": extra_libs,
+ "hs-libraries": [],
+ "extra-libraries": [pkg_id.library_name(hs, my_pkg_id)] if has_hs_library else [] + extra_libs,
"depends": hs.package_ids,
})
Again, no consideration for conditions here yet.
I think we're pretty comprehensive at this point. Don't forget the hsc_make
hack from an earlier comment too. Next steps (perhaps):
Naturally we don't want all these hacks in all use cases. It's not even certain that they are foolproof in this use case, but as I said I'll be trying to work them into a pretty large codebase that builds lots of targets and also Docker images (currently with ldd
hackery to get dynamic dependencies). If it works there I'll feel pretty good about it. Regardless -- is is_static
the thing to switch on here or do we want a beta_is_fully_static
(or similarly warning-level-ugliness name) flag, for instance?
I'll patch up the example repository with something much cleaner, perhaps collating these patches into a diff that gets applied to rules_haskell
to keep the progress/ideas in one place.
Happy to work on the PR to get this into rules_haskell
if we think the use case merits upstream support.
PR #1390 is a first stab at implementing all these things cleanly. It's just the bare bones so we can discuss whether or not we want to support all these hacks/this use case; if it's a goer we can look at tests and docs (though I've tried to make a start with some fairly heavy comments).
I've also updated https://github.com/lunaris/minirepo with a bigger set of working examples that tests GHCi, Template Haskell, GHCIDE support, Haddocks, etc. Additionally, our large (several tens of binaries, hundreds of libraries, plenty of C-linking Hackage packages) work repository appears to all build just fine (and the executables and Docker images actually work) but it'll take a bit more time to make sure we've tested all the cases before it lands in production :smile:
@lunaris Thanks a lot for documenting all the steps in detail! Sorry for the late reply, I've been meaning to look into this in more depth but haven't gotten around to it, yet. I can give comments/answers to some of the points raised.
Building GHC with a static RTS has worked well in daml
and sounds like the right approach to enable static only builds, i.e. what https://github.com/tweag/rules_haskell/pull/970 enables in rules_haskell and what's done in daml
in #1515. This includes, as you point out, the flags -fPIC -fexternal-dynamic-refs
.
Regarding the hs-libraries --> extra-libraries
patching of the package configurations. I'm surprised this is necessary. In daml
we disabled these patches when we switched to GHC with a static RTS and are able to build the fully Haskell project without having to build dynamic Haskell libraries. Have you tried with a GHC with a static RTS but without these patches? If so, what goes wrong without these patches?
Regarding the hs.toolchain.is_static
value and the corresponding toolchain parameter. That is used to distinguish a GHC with a static RTS from one with a dynamic RTS. Admittedly, the name is imprecise, we could instead have is_static_rts
and is_static
if needed. Though, I'm not sure if the distinction is still necessary if it is possible to avoid the hs-libraries --> extra-libraries
patching.
That is weird -- it does seem to work without the extra-libraries
hacks, which is great news! I'm not sure why I thought this wasn't the case; perhaps my GHC derivation wasn't quite right until I (coincidentally) applied the extra-libraries
hacks. I'm running some tests on the larger repository now to confirm that Template Haskell and GHCi and a multitude of libraries with C bits work and if so I'll simplify the patch. That said, I still think we'll be left with the bits that pass -fexternal-dynamic-refs
to Cabal and also the bit that handles hsc2hs
. Maybe. All thoughts and feedback on those changes welcome!
That is weird -- it does seem to work without the
extra-libraries
hacks, which is great news!
:tada: That's great news!
That said, I still think we'll be left with the bits that pass
-fexternal-dynamic-refs
to Cabal
Yes, that sounds reasonable. I'm actually surprised that we don't need this in daml
for the static RTS use-case. The fact that we don't do this there is, I think, just an oversight from when we switched from Hazel to stack_snapshot
. So, I think adding this functionality to the is_static
flag is perfectly reasonable.
That said, I still think we'll be left with [...] also the bit that handles
hsc2hs
.
This might also be a good thing to integrate into the is_static
flag, as the static RTS use-case is also aimed at minimizing dynamic linking. I saw that the current patch addresses this with a heuristic on the output filename in cc_wrapper
. Would it be possible to achieve the same at a higher level, e.g. by adding to the hcs2hs
flags in haskell/private/actions/compile.bzl
and by passing --hsc2hs-options
to Setup.hs configure
? That would seem like a cleaner and more robust approach.
I will give the --hsc2hs-options
thing a try -- good shout! One other thing which I can't really wrap my head around is the patch to haskell_repl
that basically disables linking all C libraries when invoking GHCi (https://github.com/tweag/rules_haskell/pull/1390/files#diff-61e0caeb5be56438636ea710c9625111R304) -- does this make sense? In my mind I've rationalised by saying "those things are already being pulled in by virtue of the static linkage to Haskell libraries" but I'm not sure if this is right. That said, I can't get it to fail and testing code that uses C libraries (e.g. PostgreSQL connections, zlib compression) in the REPL works, so it seems OK.
One other thing which I can't really wrap my head around is the patch to
haskell_repl
that basically disables linking all C libraries when invoking GHCi (https://github.com/tweag/rules_haskell/pull/1390/files#diff-61e0caeb5be56438636ea710c9625111R304) -- does this make sense? In my mind I've rationalised by saying "those things are already being pulled in by virtue of the static linkage to Haskell libraries" but I'm not sure if this is right.
Thanks for the pointer. That doesn't look right. With static linking each library archive only contains symbols introduced by that library, symbols from transitive dependencies are not pulled in. Only when statically linking the final executable will all required symbols be bundled together in that executable. I.e. all transitive static archive dependencies need to be available. (If this wasn't the case it would lead to lots of duplicate symbols and unnecessarily inflate the size of intermediate archives.)
In the context of haskell_repl
that means that all static Haskell libraries of from_binary
targets and all transitive C library dependencies need to be available for GHCi's linker to load. Ideally the logic in haskell_repl
should already cover the fully static use-case and not require any patching. What happens if you don't patch it?
That said, I can't get it to fail and testing code that uses C libraries (e.g. PostgreSQL connections, zlib compression) in the REPL works, so it seems OK.
That may well be by chance. The REPL is invoked via bazel run
outside the sandbox, which means that it has access to outputs from prior Bazel build steps. If a prior build (either a prior bazel build
or a build of a dependency of the REPL target) produced the required C library artifacts, then they will be visible to the REPL even if they are not declared dependencies.
A more precise test would be to run bazel clean; bazel run //my:repl --remote_download_toplevel
with a pre-populated disk or remote cache as described in https://github.com/tweag/rules_haskell/issues/1334.
Thanks for the pointer -- I'll run these tests and report back. That said, even if the artifacts were lying around on disk (believable), how would GHCi know to try and pick them up without appropriate -l
flags (which are the things I'm removing)?
Indeed, this is completely broken. The question then, is how to solve this error (from the example in minirepo
):
➜ bazel run //example-service:impl@repl
INFO: Invocation ID: c085a8c6-8ed3-4904-ab8e-51c274873cbd
INFO: Analyzed target //example-service:impl@repl (49 packages loaded, 3089 targets configured).
INFO: Found 1 target...
Target //example-service:impl@repl up-to-date:
bazel-bin/example-service/impl@repl@repl
INFO: Elapsed time: 1.699s, Critical Path: 0.17s
INFO: 32 processes: 30 remote cache hit, 2 linux-sandbox.
INFO: Build completed successfully, 1053 total actions
INFO: Build completed successfully, 1053 total actions
GHCi, version 8.6.5: http://www.haskell.org/ghc/ :? for help
<command line>: user specified .o/.so/.DLL could not be loaded (Error loading shared library libssl.so: No such file or directory)
Whilst trying to load: (dynamic) ssl
Additional directories searched: external/haskell_nixpkgs_openssl/lib
external/haskell_nixpkgs_crypto/lib
external/haskell_nixpkgs_postgresql/lib
external/haskell_nixpkgs_zlib/lib
bazel-out/k8-fastbuild/bin/_solib_k8/_U@haskell_Unixpkgs_Uopenssl_S_S_Cc_Ulib___Uexternal_Shaskell_Unixpkgs_Uopenssl_Slib
bazel-out/k8-fastbuild/bin/_solib_k8/_U@haskell_Unixpkgs_Ucrypto_S_S_Cc_Ulib___Uexternal_Shaskell_Unixpkgs_Ucrypto_Slib
bazel-out/k8-fastbuild/bin/_solib_k8/_U@haskell_Unixpkgs_Upostgresql_S_S_Cc_Ulib___Uexternal_Shaskell_Unixpkgs_Upostgresql_Slib
bazel-out/k8-fastbuild/bin/_solib_k8/_U@haskell_Unixpkgs_Uzlib_S_S_Cc_Ulib___Uexternal_Shaskell_Unixpkgs_Uzlib_Slib
Does GHCi work in DAML without any issues? I know DAML is using zlib; does it use other libraries? Note that I don't think this is libssl
-specific -- if I remove the -l
arguments passed to GHCi one by one, it bites all C dependencies I believe (z
, pq
, etc.). Perhaps the issue is search directories, or some other configuration?
That said, even if the artifacts were lying around on disk (believable), how would GHCi know to try and pick them up without appropriate
-l
flags (which are the things I'm removing)?
If they are dependencies of some from_binary
packages as well, e.g. anything in @stackage
, then GHC will find them in these package's package-db entries.
Does GHCi work in DAML without any issues?
Yes, GHCi works in DAML. There is a static libz
dependency and a dynamic libgpr
/libgrpc
dependency.
I haven't reproduced the issue, yet (building GHC takes a while). But, I have a suspicion. GHC lists two search paths that seem relevant for ssl:
external/haskell_nixpkgs_openssl/lib
bazel-out/k8-fastbuild/bin/_solib_k8/_U@haskell_Unixpkgs_Uopenssl_S_S_Cc_Ulib___Uexternal_Shaskell_Unixpkgs_Uopenssl_Slib
The latter would only contain the dynamic library. The former would contain the static library, but that path is relative to the execroot and won't work relative to the repository root. However, haskell_repl
changes directory into the repository root. We probably need some extra path mangling in haskell_repl
to fix this sort of thing.
Implemented in https://github.com/tweag/rules_haskell/pull/1390 :tada:
GHC by default creates "static" libraries. As explained in #378, this really means "mostly static" in Bazel terms, because some dependencies are still dynamically linked. It is possible, however, to create fully static binaries, provided there are static versions of all external libraries available. We should implement this link mode as well, and make it configurable per-target. We should probably stick to a similar API for this to what the CC rules already provide.
cc @lunaris