guibou
commented
5 years ago I'm still not able to generate a small reproducible example, but I now have a lot of question related to the implementation of -prof in rules_haskell.
So, rules_haskell fails building some targets in the following context:
TemplateHaskellis activated- More than one haskell file in the
srcpackage - Dependencies to other haskell libraries which have cbit
When experimenting, I found that removing the -fexternal-interpreter flag from https://github.com/tweag/rules_haskell/blob/f7b355a9c5da4e0b81fc60bc963cd78f0dc4fe0b/haskell/private/actions/compile.bzl#L202 leads to progress, i.e. targets which cannot be built previously can now be built.
I've tried the same change in rules_haskell codebase and the result is surprising. bazel test -c dbg //... fails all tests, but bazel test -c dbg //a_specific_test works for all tests. I'll try to understand that later.
There are two ways to build with profiling using ghc and TemplateHaskell, -fexternal-interpreter and a two phases build where you first build without -prof and then with. This is detailed in https://downloads.haskell.org/~ghc/latest/docs/html/users_guide/glasgow_exts.html#using-template-haskell-with-profiling
For example, this experimentation (with only one module, in this case the two phases build is not really necessary, but will be for more modules).
Foo.hs:
{-# LANGUAGE TemplateHaskell #-}
main = do
print $([| 10 |])Build without profiling:
[guillaume@paddle:/tmp]$ ghc Foo.hs
[1 of 1] Compiling Main ( Foo.hs, Foo.o )
Linking Foo ...
[guillaume@paddle:/tmp]$ ./Foo
10
[guillaume@paddle:/tmp]$ ./Foo +RTS -p
Foo: the flag -p requires the program to be built with -prof
[...]Failure with just -prof:
[guillaume@paddle:/tmp]$ ghc Foo.hs -prof
[1 of 1] Compiling Main ( Foo.hs, Foo.o )
Foo.hs:4:9: fatal:
Cannot load -prof objects when GHC is built with -dynamic
To fix this, either:
(1) Use -fexternal-interpreter, or
(2) Build the program twice: once with -dynamic, and then
with -prof using -osuf to set a different object file suffix.(Note how it discusses the two solutions)
First solution with -fexternal-interpreter:
[guillaume@paddle:/tmp]$ ghc Foo.hs -prof -fexternal-interpreter
[1 of 1] Compiling Main ( Foo.hs, Foo.o )
Linking Foo ...
[guillaume@paddle:/tmp]$ ./Foo +RTS -p
10(This build command takes 30s on my computer)
Second solution, with the two builds:
[guillaume@paddle:/tmp]$ ghc Foo.hs
[1 of 1] Compiling Main ( Foo.hs, Foo.o )
Linking Foo ...
[guillaume@paddle:/tmp]$ ghc Foo.hs -prof -osuf curryforpresident
[1 of 1] Compiling Main ( Foo.hs, Foo.curryforpresident )
Linking Foo ...
[guillaume@paddle:/tmp]$ ./Foo +RTS -p
10This solution takes 2s to build.
-fexternal-interpreter is discussed here: https://ghc.haskell.org/trac/ghc/wiki/RemoteGHCi
Now I have many discussions points to direct my work. Currently, rules_haskell uses -fexternal-interpreter AND the two build solution at the same time, see:
The -fexternal-interpreter flag was added in the middle of a pull request which seems totally unrelated:
@mrkkrp could you comment please?
There is two possible directions:
- remove
-fexternal-interpreterand fix the two steps build. - remove the two steps build and fix the
-fexternal-interpreter
Both solutions have pro and cons, mostly:
-fexternal-interpreteris slow! 30s for a one line file with TemplateHaskell. However I don't know if that's a fixed cost which won't grow with the source code complexity or if it is not.- The two step build is building two times, so may be slow for complex builds
-fexternal-interpreteris listed as more "secure" and versatile on its design page AND may lead to a simpler implementation in rules haskell.-fexternal-interpreteris new (ghc 8.0) and not broadly tested because that's not the default solution, but it "make become in the future".
cc @mboes
mboes
Profpatsch
This is an updated description which summarize the discussion
-c dbgis really slow when template haskell is used because we use-fexternal-interpreterand this GHC bug: https://ghc.haskell.org/trac/ghc/ticket/16256#ticket . In summary, profiling build is 15x times slower than normal build.-fexternal-interpreter, which is now used in rules_haskell, and another solution which build two times, once without profiling (so template haskell can execute non profiled code) and one with profiling.-fexternal-interpreterwas accidentally done in be5cf67#diff-33a8c46955e2a9917090b5258346aee2R205. Since #892 the dead code of the two build solution was removed.We have two solution here:
-fexternal-interpreterand wait for an upstream fix for GHC-fexternal-interpreterand reimplement the two steps build.