foreign-libraries break with --enable-profiling

[DanielG]

When I compile a project with foreign-library sections with --enable-profiling ld complains about impossible relocations when compiling the flib component. I am using Cabal/cabal-install from master.

The errors look suspiciously simmilar to the ones from the GHC bug report about Debian/Ubuntu enabling PIE by default, not sure if that's actually related. I'm pretty sure my compilers aren't affected by that since I can also reproduce it in jessie, see below.

To reproduce:

$ git clone https://gist.github.com/b0c119d866799812fa2c9d367d67e84a.git
$ cd b0c119d866799812fa2c9d367d67e84a
$ cabal new-build --enable-profiling

On Debian jessie (using the ghc-7.8.4 bindist):

Resolving dependencies...
In order, the following will be built (use -v for more details):
 - foreign-lib-0 (flib:foo) --enable-profiling (first run)
Configuring component flib:foo from foreign-lib-0...
Preprocessing foreign library 'foo' for foreign-lib-0...
[1 of 1] Compiling Lib              ( Lib.hs, /home/dxld/share/dev/hs/testing/foreign-lib/dist-newstyle/build/x86_64-linux/ghc-7.8.4/foreign-lib-0/c/foo/build/foo/foo-tmp/Lib.p_o )
[1 of 1] Compiling Lib              ( Lib.hs, /home/dxld/share/dev/hs/testing/foreign-lib/dist-newstyle/build/x86_64-linux/ghc-7.8.4/foreign-lib-0/c/foo/build/foo/foo-tmp/Lib.p_o ) [flags changed]
Linking /home/dxld/share/dev/hs/testing/foreign-lib/dist-newstyle/build/x86_64-linux/ghc-7.8.4/foreign-lib-0/c/foo/build/foo/libfoo.so ...
/usr/bin/ld: /usr/local/lib/ghc-7.8.4/base-4.7.0.2/libHSbase-4.7.0.2_p.a(Base__115.p_o): relocation R_X86_64_32S against `era' can not be used when making a shared object; recompile with -fPIC
/usr/local/lib/ghc-7.8.4/base-4.7.0.2/libHSbase-4.7.0.2_p.a: error adding symbols: Bad value
collect2: error: ld returned 1 exit status

On Debian sid:

Resolving dependencies...                                                                                    
In order, the following will be built (use -v for more details):
 - foreign-lib-0 (flib:foo) --enable-profiling (first run)
Configuring component flib:foo from foreign-lib-0...
Preprocessing foreign library 'foo' for foreign-lib-0...
[1 of 1] Compiling Lib              ( Lib.hs, /home/dxld/share/dev/hs/testing/foreign-lib/dist-newstyle/build/x86_64-linux/ghc-8.0.2/foreign-lib-0/c/foo/build/foo/foo-tmp/Lib.p_o )
[1 of 1] Compiling Lib              ( Lib.hs, /home/dxld/share/dev/hs/testing/foreign-lib/dist-newstyle/build/x86_64-linux/ghc-8.0.2/foreign-lib-0/c/foo/build/foo/foo-tmp/Lib.p_o ) [flags changed]
Linking /home/dxld/share/dev/hs/testing/foreign-lib/dist-newstyle/build/x86_64-linux/ghc-8.0.2/foreign-lib-0/c/foo/build/foo/libfoo.so ...
/usr/bin/ld: /home/dxld/var/local/stow/ghc-8.0.2/lib/ghc-8.0.2/base-4.9.1.0/libHSbase-4.9.1.0_p.a(Base__165.p_o): relocation R_X86_64_32S against symbol `era' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: /home/dxld/var/local/stow/ghc-8.0.2/lib/ghc-8.0.2/base-4.9.1.0/libHSbase-4.9.1.0_p.a(Base__166.p_o): relocation R_X86_64_32S against symbol `era' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: /home/dxld/var/local/stow/ghc-8.0.2/lib/ghc-8.0.2/base-4.9.1.0/libHSbase-4.9.1.0_p.a(Base__170.p_o): relocation R_X86_64_32S against symbol `era' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: /home/dxld/var/local/stow/ghc-8.0.2/lib/ghc-8.0.2/base-4.9.1.0/libHSbase-4.9.1.0_p.a(Base__172.p_o): relocation R_X86_64_32S against symbol `era' can not be used when making a shared object; recompile with -fPIC
[...]
/usr/bin/ld: /usr/lib/ghc/ghc-prim-0.5.0.0/libHSghc-prim-0.5.0.0_p.a(Types__328.p_o): relocation R_X86_64_32S against symbol `era' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: /usr/lib/ghc/ghc-prim-0.5.0.0/libHSghc-prim-0.5.0.0_p.a(Types__327.p_o): relocation R_X86_64_32S against symbol `era' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: /usr/lib/ghc/base-4.9.0.0/libHSbase-4.9.0.0_p.a(Base__1.p_o): relocation R_X86_64_PC32 against undefined symbol `CCS_LIST' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: Bad value
collect2: error: ld returned 1 exit status
`gcc' failed in phase `Linker'. (Exit code: 1)

Verbose output with -v here: Sid log, Jessie log.

[ezyang]

CC @edsko and @abooij

[abooij]

Similarly, you get lots of linking errors when enabling library profiling on the ForeignLibs test in the test suite, with either build or new-build. So there is nothing specific about @DanielG's code or the build command. Edit: Oh, and I tested this on Arch.

[abooij]

If you add

ghc-options: -dynamic

we get a slightly more sensible error message:

[1 of 3] Compiling MyForeignLib.SomeBindings ( /tmp/ForeignLibs/dist-newstyle/build/x86_64-linux/ghc-8.0.1/my-foreign-lib-0.1.0.0/c/myforeignlib/build/myforeignlib/myforeignlib-tmp/MyForeignLib/SomeBindings.hs, /tmp/ForeignLibs/dist-newstyle/build/x86_64-linux/ghc-8.0.1/my-foreign-lib-0.1.0.0/c/myforeignlib/build/myforeignlib/myforeignlib-tmp/MyForeignLib/SomeBindings.p_o )

src/MyForeignLib/SomeBindings.hsc:2:8: error:
    Failed to load interface for ‘Prelude’
    Perhaps you haven't installed the "p_dyn" libraries for package ‘base-4.9.0.0’?
    Use -v to see a list of the files searched for.

I believe this might make sense: GHC ships with a static variant of every haskell library interface, a dynamic variant, and a profiling variant (this is assuming that the p in p_hi means "profiling"). We need to use the dynamic variants (and apparently GHC doesn't choose this correctly if we don't hint it with -dynamic). But we also need this dynamic variant to be profiling, hence p_dyn. But this variant is not shipped with GHC. Edit: added link.

[abooij]

So to rephrase this: should we add -dynamic to ghc-options automatically for all foreign libraries that are being built on unix-style OSes? And output a warning that most GHC distributions don't support profiling of dynamic objects?

[ezyang]

Well, according to the documentation and @edsko's original design, it was also intended that static foreign libraries eventually be supported.

I guess there is probably something missing in our build planning, which is when a foreign library is found, we need to plan to build shared versions of all dependencies of that library. Is that actually true? If so, I am a bit surprised this works at all today.

[abooij]

@ezyang I'm not sure I follow you entirely. In any case, on linux we are now normally using the dynamic interface files to build foreign libraries. So if we enable profiling, we'll need dynamic profiling variants. Now I don't know why GHC would choose static profiling variants instead - perhaps this is a mistake in GHC.

[ezyang]

It seems to me that the build logic for foreign libraries is interacting with some extra logic for profiling. Notice in the logs, GHC is re-building all of the modules because flags changed (you can see this in the original logs when you see [1 of 1] Compiling Lib show up twice; since the paths are long the "flags changed" notification is off the right side of the screen".

The crux of the matter is that when we invoke GHC to link the foreign library, we do so with a command line that looks like this:

/usr/bin/ghc --make -fbuilding-cabal-package -O -prof -fprof-auto-top -shared -fPIC -osuf p_o -hisuf p_hi -outputdir dist/build/myforeignlib/myforeignlib-tmp -odir dist/build/myforeignlib/myforeignlib-tmp -hidir dist/build/myforeignlib/myforeignlib-tmp -stubdir dist/build/myforeignlib/myforeignlib-tmp -i -idist/build/myforeignlib/myforeignlib-tmp -isrc -idist/build/myforeignlib/autogen -idist/build/global-autogen -Idist/build/myforeignlib/autogen -Idist/build/global-autogen -Idist/build/myforeignlib/myforeignlib-tmp -optP-include -optPdist/build/myforeignlib/autogen/cabal_macros.h -optl-Wl,--no-as-needed -lHSrts-ghc7.10.3 -lffi -L/usr/lib/ghc/rts -no-hs-main '-dynload deploy' -optl-Wl,-rpath,/srv/cabal/lib/x86_64-linux-ghc-7.10.3 -optl-Wl,-rpath,/usr/lib/ghc/base_HQfYBxpPvuw8OunzQu6JGM -optl-Wl,-rpath,/usr/lib/ghc/rts -optl-Wl,-rpath,/usr/lib/ghc/ghcpr_8TmvWUcS1U1IKHT0levwg3 -optl-Wl,-rpath,/usr/lib/ghc/integ_2aU3IZNMF9a7mQ0OzsZ0dS -hide-all-packages -package-db dist/package.conf.inplace -package-id base-4.8.2.0-0d6d1084fbc041e1cded9228e80e264d -package-id my-foreign-lib-0.1.0.0-7Cr37SBtAJVHqfpaapg81r -XHaskell2010 MyForeignLib.Hello MyForeignLib.SomeBindings dist/build/myforeignlib/myforeignlib-tmp/csrc/MyForeignLibWrapper.o -o dist/build/myforeignlib/libmyforeignlib.so

Notably, -shared is passed to this command, but not -dynamic. In contrast, look at the command we use to initially build the object file:

/usr/bin/ghc --make -no-link -fbuilding-cabal-package -O -prof -fprof-auto-top -j1 -osuf p_o -hisuf p_hi -outputdir dist/build/myforeignlib/myforeignlib-tmp -odir dist/build/myforeignlib/myforeignlib-tmp -hidir dist/build/myforeignlib/myforeignlib-tmp -stubdir dist/build/myforeignlib/myforeignlib-tmp -i -idist/build/myforeignlib/myforeignlib-tmp -isrc -idist/build/myforeignlib/autogen -idist/build/global-autogen -Idist/build/myforeignlib/autogen -Idist/build/global-autogen -Idist/build/myforeignlib/myforeignlib-tmp -optP-include -optPdist/build/myforeignlib/autogen/cabal_macros.h -hide-all-packages -package-db dist/package.conf.inplace -package-id base-4.8.2.0-0d6d1084fbc041e1cded9228e80e264d -package-id my-foreign-lib-0.1.0.0-7Cr37SBtAJVHqfpaapg81r -XHaskell2010 MyForeignLib.Hello MyForeignLib.SomeBindings

Here, there is no reference to -shared. Which seems unlikely to work.

I think what happens is that when we pass --enable-profiling, Setup tries to be clever and disable dynamic linking (because, as you observe, lack of distributed p_dyn files means this isn't really going to work.) But that doesn't play nice with foreign libraries, which MUST be built shared.

[ezyang]

(By the way, I found all this info by running cabal build -v and looking carefully at the GHC command line invocations.)

[abooij]

@ezyang: So does your analysis imply that this could be fixed by passing -shared to the ghc command that builds the object files? Where is the code that does this?

[ezyang]

The code that does this is in Distribution.Simple.GHC in gbuild. It's kind of a mess; I'm not sure passing -shared in more cases will "fix" it entirely. It could be this code needs to be written to understand how to build with BOTH profiling and dynamic; looking at the code, it seems to assume that this is an either/or situation (since GHC doesn't ship with dynamic profiling libraries, so you wouldn't be able to do this anyway).

[abooij]

If I add -shared to the command you mentioned by adding ghc-options: -shared to the foreign library, it fails with the same errors.

[ezyang]

Well, I'm doubtful this is the correct approach, as the current semantics of --enable-profiling --enable-shared seem to be (1) build static with profiling, and (2) build dynamic without profiling.

[KaneTW]

Is there any update on this? It's really annoying not being able to profile foreign libraries.