Break up JSDOM.Types

[mightybyte]

When I build jsaddle-dom I get the following error:

[  1 of 600] Compiling JSDOM.Types      ( src/JSDOM/Types.hs, dist/build/JSDOM/Types.p_o )
clang-3.7: error: unable to execute command: Killed: 9
clang-3.7: error: clang frontend command failed due to signal (use -v to see invocation)
clang version 3.7.1 (tags/RELEASE_371/final)
Target: x86_64-apple-darwin15.6.0
Thread model: posix
clang-3.7: note: diagnostic msg: PLEASE submit a bug report to  and include the crash backtrace, preprocessed source, and associated run script.
clang-3.7: note: diagnostic msg:
********************

PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT:
Preprocessed source(s) and associated run script(s) are located at:
clang-3.7: note: diagnostic msg: /private/var/folders/yw/4lyc2l_n51z601wqjnyc9vvr0000gn/T/nix-build-jsaddle-dom-0.9.1.0.drv-0/ghc_14-d6f02a.c
clang-3.7: note: diagnostic msg: /private/var/folders/yw/4lyc2l_n51z601wqjnyc9vvr0000gn/T/nix-build-jsaddle-dom-0.9.1.0.drv-0/ghc_14-d6f02a.sh
clang-3.7: note: diagnostic msg:

********************

<no location info>: error:
    `cc' failed in phase `C Compiler'. (Exit code: 254)

It sits on the JSDOM.Types module for tens of minutes, possibly more than an hour before failing. I'm running macOS on a machine with 16 gigs of RAM. The fact that I can't build jsaddle-dom on a machine this big seems like a problem. Is it possible to break this module up to make it buildable on normal size systems?

[luite]

what are the compiler version and tools/settings you use to cause this?

in a normal build on my system, GHC 8.2.2, macOS 10.13.2, latest XCode tools, JSDOM.Types takes very long to build, but GHC "only" peaks at around 2.5GB of memory.

(edit: my "normal" build meaning cabal install in a clean sandbox, with no custom options. GHC from the minimal Haskell platform)

[mightybyte]

This is with GHC 8.0.2 and ghcjs-0.2.1-8.0.2 on macOS 10.11.6.

If you look closely at the above output, it says Types.p_o, so I think this on a build with profiling turned on (probably done automatically by nix I would imagine).

[mightybyte]

Just to follow up, we've also tried to build this on a machine with 32 gigs of RAM and it still runs out of memory. IMO you should definitely consider breaking this module up. This is a huge problem.

[Ericson2314]

CC @luigy, who was running out of memory on linux-only.

[luite]

GHC's own memory usage looks reasonable. This is caused by the C compiler compiling the temporary file declaring all cost centres. GCC 5.4.0 seems to cope, clang 3.7 doesn't. The file is around 100k lines with contents like this:

 extern CostCentreStack jsaddlezmdomzm0zi9zi1zi0zm523c3ILQYauIbqrzzngOY1D_JSDOMziTypes_gTypeGObject_CAF_cc_ccs[];
 REGISTER_CCS(jsaddlezmdomzm0zi9zi1zi0zm523c3ILQYauIbqrzzngOY1D_JSDOMziTypes_gTypeGObject_CAF_cc_ccs);
 extern CostCentreStack jsaddlezmdomzm0zi9zi1zi0zm523c3ILQYauIbqrzzngOY1D_JSDOMziTypes_gTypeGObject1_CAF_cc_ccs[];
 REGISTER_CCS(jsaddlezmdomzm0zi9zi1zi0zm523c3ILQYauIbqrzzngOY1D_JSDOMziTypes_gTypeGObject1_CAF_cc_ccs);
 extern CostCentreStack jsaddlezmdomzm0zi9zi1zi0zm523c3ILQYauIbqrzzngOY1D_JSDOMziTypes_gTypeGObject2_CAF_cc_ccs[];
 REGISTER_CCS(jsaddlezmdomzm0zi9zi1zi0zm523c3ILQYauIbqrzzngOY1D_JSDOMziTypes_gTypeGObject2_CAF_cc_ccs);

This has been fixed in GHC 8.4.1. A workaround for earlier versions is removing any -fprof-auto flag from the GHC options (or appending -fno-prof-auto)

see https://ghc.haskell.org/trac/ghc/ticket/7960 https://git.haskell.org/ghc.git/commitdiff/a8da0de27e600211f04601ac737c329d6603c700

[mightybyte]

Any chance we can side step this problem easily by just breaking up the module?

[luite]

"just" breaking up the module seems like a tad more work than adding a GHC flag. Why make major changes to the code if we already have a workaround and proper upstream fix?

Now that we know the problem, I think we should first figure out if there should be a workaround in the package itself, perhaps a GHC version dependent (with CPP) OPTIONS_GHC pragma or a few lines in the cabal file.

[luigy]

This has been fixed in GHC 8.4.1. A workaround for earlier versions is removing any -fprof-auto flag from the GHC options (or appending -fno-prof-auto)

good call @luite removing -fprof-auto got rid of my out of memory issues I was hitting on linux with profiled build for jsaddle-dom

[luite]

workaround for GHC issue implemented in pull request #6

I'm not the package maintainer. @hamishmack merge if you think this looks ok.

[eskimor]

I am experiencing a similar issue with ghc 8.4.3 (not with 8.2.2). Even without profiling the build does not complete on my 8 GB machine. I filed a ghc issue: https://ghc.haskell.org/trac/ghc/ticket/15455#comment:1

Is anyone else experiencing this?

[FintanH]

This is still an issue for me on ghc-8.0.2 and using the -fno-prof-auto

[573]

@eskimor I experience a similar issue with ghc883, see link in comment here. @mightybyte guess I will repeat my build with profiling off for jsaddle-dom then.

[Lysxia]

I also have trouble compiling JSDOM.Types. I split the module by hand; with one hour work, the new modules build in 5 minutes with 8GB RAM. The main thing stopping me from just making a PR is that the split is sort of random, and maybe you'd rather fix whatever script is generating JSDOM.Types instead of messing with the generated file manually. Feel free to leave suggestions to put it into a reviewable state.

https://github.com/Lysxia/jsaddle-dom/tree/splittypes

[noinia]

I second the request to split up JSDOM.Types. Even with current GHC's this still need a huge amount of RAM, and takes quite a while to compile. Moreover, it prevents downstream packages from easily using CI systems [1,2]. This has a cost as well, since it does not easily allow authors to make sure their package builds using the latest versions of the packages on hackage. (Indeed, reflex-dom currently no longer builds, thus causing me as yet another downstream package author to figure out why).

Are there reasons why we would particularly like to keep this single huge types module?

[1] e.g. reflex-dom's testsuite fails due to out of memory errors when compiling jsaddle-dom (see e.g. https://github.com/reflex-frp/reflex-dom/runs/4629811019?check_suite_focus=true ) [2] I can't add hgeometry-web to my CI for the same reason as above.