search

mgsloan / store

Fast binary serialization in Haskell
MIT License
109 stars 35 forks source link

High memory usage during compilation using GHC 8.0.2-rc2 #91

Closed domenkozar closed 7 years ago

domenkozar commented 7 years ago

I'm building stack using GHC 8.0.2-rc2 and noticed store uses 5.17GB of (RSS) memory during Data.Store.Internal compilation step. Using GHC 8.0.1 memory goes up to only 1.6GB.

I haven't dug deeper, but something suspicious is going on.

You should be able to reproduce using installed Nix and executing nix-build -A haskell.packages.ghc802.store '<nixpkgs>' using recent channel.

Building haddock for store also goes up to 5.8GB.

domenkozar commented 7 years ago

I've submitted GHC bug report, maybe leave this open until it's resolved: https://ghc.haskell.org/trac/ghc/ticket/13059

mgsloan commented 7 years ago

Wow, good catch! Not sure why so much memory is needed to compile this module. A few possibilities:

1) Lots of TH use? Seems unlikely 2) Lots of instances, lots of them with INLINE

domenkozar commented 7 years ago

@mgsloan FYI GHC 8.0.2 has just been released so I expect to see this to pop up :)

mitchty commented 7 years ago

Yep, just hit this trying to build on an arm box with 2g of ram:

Configuring store-0.3...
Building store-0.3...
Preprocessing library store-0.3...
[ 1 of 11] Compiling System.IO.ByteBuffer ( src/System/IO/ByteBuffer.hs, dist/build/System/IO/ByteBuffer.o )
[ 2 of 11] Compiling Data.Store.Streaming.Internal ( src/Data/Store/Streaming/Internal.hs, dist/build/Data/Store/Streaming/Internal.o )
[ 3 of 11] Compiling Data.Store.Impl  ( src/Data/Store/Impl.hs, dist/build/Data/Store/Impl.o )
[ 4 of 11] Compiling Data.Store.TH    ( src/Data/Store/TH.hs, dist/build/Data/Store/TH.o )
[ 5 of 11] Compiling Data.Store.TH.Internal ( src/Data/Store/TH/Internal.hs, dist/build/Data/Store/TH/Internal.o )
[ 6 of 11] Compiling Data.Store.Internal ( src/Data/Store/Internal.hs, dist/build/Data/Store/Internal.o )
ghc: out of memory (requested 1048576 bytes)
cabal: Leaving directory '/tmp/cabal-tmp-31399/store-0.3'
cabal: Error: some packages failed to install:
stack-1.3.2 depends on store-0.3 which failed to install.
store-0.3 failed during the building phase. The exception was:
ExitFailure 251
sjakobi commented 7 years ago

A likely culprit for the excessive memory usage has been identified in https://ghc.haskell.org/trac/ghc/ticket/13059#comment:20:

Essentially, GHCs up to 8.0.2 had a bug that caused the INLINE pragma on DefaultSignatures method implementations to be ignored. Now that this bug has been fixed, the additional INLINE pragmas kick in and cause the extra memory usage.

The fix is to remove one "layer" of INLINE pragmas as shown in the comment. Apparently this shouldn't much affect performance, but it might be better to run some benchmarks to be safe.