GHC bug (possibly triggered in ide-backend, not certain)

[snoyberg]

I just got the following error message when running our test suite:

test: internal error: allocation of 3926544 bytes too large (GHC should have complained at compile-time)
    (GHC version 7.8.3.20140729 for x86_64_unknown_linux)
    Please report this as a GHC bug:  http://www.haskell.org/ghc/reportabug

@manny-fp recently saw something similar on his system as well, and I saw the same thing when running devel-manager a few days ago. This is most likely not an ide-backend issue, but I wanted to ask for any feedback you gentlemen may have on what's going on here.

We've upgraded a large number of our libraries since the last release of FPHC, which is the more likely trigger for this issue.

Also pinging @mgsloan, I (obviously) want to resolve this before the FPHC 3.2 release.

[Mikolaj]

It's probably this issue:

https://ghc.haskell.org/trac/ghc/ticket/9647

I don't know any more details, but it seems not to be fixed in 7.8.4RC1 and the only proposed fix seems to just detect the compiler deficiency at compile time, instead of runtime. (I've heard somebody muse recently about lifting a 1MB allocation ceiling, but I'm now not sure if that was said in the same context.)

[snoyberg]

Thanks @Mikolaj, that's helpful. Do you have any thoughts on isolating where this allocation is occurring?

[Mikolaj]

That's where the compile-time failure would be very helpful. I don't remember OTTOMH any place where ide-backend or the Haskell code of GHC allocates things as in the bug description.

The program compiles to code that allocates >1MB in one go, and the runtime doesn't currently support that (we support allocating >1MB only for a single object, such as an array, not for multiple objects).

I guess, for the allocation not to be divided into smaller allocations, the size of, e.g., a list or a tree has to be known statically at compilation time (though it may be nested in ifs, and so not occurring every time) and it needs to be non-lazy. That's not a common thing: e.g., a constant huge list or a huge Integer. I don't think our conversion of lazy values to strict values qualifies, because it's done gradually via a traversal and the sizes are not known statically. The constant licence texts are not big enough.

[snoyberg]

Thanks again @Mikolaj. @mgsloan and I were just discussing this as well, and we have a possible lead: we're generating a lot of data for the hlint hint database at compile time. Michael Sloan is going to try turning that off and see if it fixes the segfault.

[mgsloan]

Yup, it looks like using th-lift on the parsed versions of the hlint hint database was the culprit. When using hlint in a more ordinary way, this doesn't happen.

[snoyberg]

That's good news @mgsloan, let me know when a patch lands on master so I can test.

Thanks for the help @Mikolaj, the GHC links definitely helped us track this down.