memory leak

[zoranbosnjak]

This test program is a short example of a memory leak problem in a real application.

-- test.hs
{-# LANGUAGE BangPatterns #-}

import Control.Monad
import Pipes
import Pipes.Safe

main :: IO ()
main = do
    forever $
        runSafeT $
            runEffect (source >-> sink True)
  where
    source = forever (yield ())

    sink !arg = do
        _dummy1 <- await
        _dummy2 <- liftIO $ return ()
        sink arg

To reproduce:

$ ghc -prof -rtsopts -O2 -Wall test.hs && ./test +RTS -M10m -RTS
test: Heap exhausted;
test: Current maximum heap size is 10485760 bytes (10 MB).
test: Use `+RTS -M<size>' to increase it.

Problem was observed on:

• pipes-4.3.13
• pipes-safe-2.3.2
• ghc 8.6.5

Am I misusing any functions here?

There are several minor changes to the test program that would make a leak disappear. BangPatterns pragma is one of them. I could remove this one, but the rest of the structure is required by the real application where the problem was first observed.

If there is no easy fix, please advice how to workaround the problem with the current versions.

[Gabriella439]

So I was able to minimize the reproduction a little more to this example:

{-# LANGUAGE BangPatterns #-}

import Control.Monad
import Pipes
import Pipes.Safe

main :: IO ()
main = forever (runSafeT (runEffect (sink True)))
  where
    sink !arg = do
        liftIO (return ())
        sink arg

I'm still looking into why that is behaving weirdly by studying the core (by running ghc -O2 -ddump-simpl -dsuppress-all test.hs -fforce-recomp on variations of that program). That is giving me some leads, but nothing conclusive, yet.

[Gabriella439]

@zoranbosnjak: I was able to minimize it further down to this example:

{-# LANGUAGE BangPatterns #-}

import Control.Monad
import Control.Monad.Reader
import Pipes
import Pipes.Internal

main :: IO ()
main = forever (runReaderT (runEffect (sink True)) True)
  where
    sink !arg = M $! return (sink arg)

The issue appears to not be specific to pipes-safe and is rather the result of a surprising GHC optimization.

What I was able to determine is that the root cause appears to be something you can't easily control from the user source code and it doesn't appear to be easily fixable in pipes. Basically, what's happening is that GHC is making the M constructor of the Proxy type unnecessarily strict (which the smaller reproduction emulates explicitly), and that strictness triggers the leak. The pipes source code has no such strictness annotation anywhere in the relevant utilities.

The only thing that reliable corrects this behavior from user source code is to remove the bang pattern on the argument to sink (or more generally, to not be strict in the loop argument).

[zoranbosnjak]

Thanks for analysis. It is obviously not specific to pipes-safe, so I am closing this issue. I have work around the problem in the sink function as suggested.