Closed juhp closed 2 years ago
Stack builds packages in parallel by default I believe - try turning that off so only one package is built at a time?
[280 of 280] Compiling Network.AWS.EC2
gcc: error: .stack-work/dist/x86_64-linux/Cabal-3.0.0.0/build/Network/AWS/EC2/Types/Product.dyn_o: No such file or directory
`gcc' failed in phase `Linker'. (Exit code: 1)
and another time:
-- While building package amazonka-ec2-1.6.1 using:
/var/stackage/.stack/setup-exe-cache/x86_64-linux/Cabal-simple_mPHDZzAJ_3.0.0.0_ghc-8.8.1 --builddir=.stack-work/dist/x86_6
4-linux/Cabal-3.0.0.0 build lib:amazonka-ec2 test:amazonka-ec2-test --ghc-options ""
Process exited with code: ExitFailure (-9) (THIS MAY INDICATE OUT OF MEMORY)
Logs have been written to: /var/stackage/work/unpack-dir/.stack-work/logs/amazonka-ec2-1.6.1.log
Preprocessing library for amazonka-ec2-1.6.1..
Building library for amazonka-ec2-1.6.1..
[ 3 of 280] Compiling Network.AWS.EC2.Types.Product
This is fairly well trodden territory by now - there's little short term I can do to affect how much memory varying versions of GHC use when compiling the libraries. There is discussion around splitting the graph of generated types into modules based on their SCCs or some other boundary to alleviate the fact Product.hs
is too large for GHC to compile with reasonable memory.
There was a GHC ticket raised about this recently. While we will try to improve things in the long term. However, given that the current situation seems barely tenable there are a few things that you might consider doing in the short term as well:
-fno-call-arity
): As noted in the ticket cited above, the Product
(and to a lesser extent the Sum
modules` tend to hit a pathological case in this analysis. Disabling the analysis will slightly reduces residency and reduces compilation time in my measurements.{-# OPTIONS_GHC -O0 #-}
). I would not expect any of this code to be a computational bottleneck in realistic applications; there is little sense in spending cycles optimising it.hs-boot
files to break cycles (avoiding the need for any SCC cleverness). Simply move each type (and its instances and lenses) into its own module, producing an hs-boot
file for each with a simple data
declaration. Ideally each module would then need only to SOURCE
import the hs-boot
files for types that that depends upon (e.g. has fields of). However, if that is too hard to arrange, you could go one step simpler: define a single Types
module which SOURCE
imports and reexports each of the types. This can then be import
'd by each of the types' modules.I will say that even with -O0
this module is disappointingly slow. I do hope that we can improve this in 9.2. This package is clearly a great stress test for GHC. Nevertheless, I do think that it could be made significantly less painful with a bit of restructuring (suggestion (3) above); compiling a single 10kLoC with hundreds of derived instances is simply asking for trouble. If that is too much work I would probably instead try suggestion (2), as this code really doesn't seem performance-critical.
The large Types
modules are split as of https://github.com/brendanhay/amazonka/pull/591, which substantially improved our compile time. I have not analyzed memory improvements.
The large Types modules are split as of #591, which substantially improved our compile time. I have not analyzed memory improvements.
Right. I have been building amazonka
master
in my GHC measurements. However, even Product
and Sum
are very large given the how many non-trivial instances they derive. One module per type may seem excessive, but I suspect it will work quite well, especially given that this would provide plenty of module-level parallelism for GHC's parallel up-sweep to exploit.
@bgamari Is this still bad, now that the pattern synonym stuff is in develop
?
The pattern synonyms change itself has little to no impact on compilation performance - it's the quadratic core which is detailed in Edkso's Avoiding quadratic core code size with large records that we've always been subject to.
We are already generating one-type-per-module on main
, which was introduced in https://github.com/brendanhay/amazonka/pull/591 - this should be sufficient to ameliorate this particular issue. Further work/improvements to be followed up via #717
We are trying to build amazonka-ec2-1.6.1 with ghc-8.8.1 on Linux for Stackage Nightly using stack-2.1.3 and ghc is using a lot of memory: about 6.8GB, which may be causing problems. Not sure if anything can be done, but thought I would report it here.