Memory leak on OS X

[sethfowler]

There seems to be a memory leak when using this library on OS X. I am not sure if it comes from code in System.FSNotify or from the underlying System.OSX.FSEvents, though.

[gregwebs]

Thanks for the report. Can you give a minimal test program that creates the memory leak?

[sethfowler]

Sure. This program is enough to do it:

{-# LANGUAGE OverloadedStrings #-}

import System.FSNotify
import System.IO

main = withManager doWatch

doWatch m = do
  putStrLn "doWatch is starting. Do something that will cause large amounts of"
  putStrLn "disk activity in the current directory; my test case is to compile"
  putStrLn "Firefox. Press Enter when things settle down to terminate. This"
  putStrLn "program should easily reach hundreds of megabytes of memory"
  putStrLn "when compiling a large program like Firefox."
  watchTree m "." (const True) (\e -> putStrLn $ "Got event " ++ (show e))
  getLine >> return ()

Instructions are included. =) If you want to try with my exact test case, you can grab a copy of the Firefox source using hg clone http://hg.mozilla.org/mozilla-central and build it with make -f client.mk.

I just ran this twice and checked the final memory usage as reported by Activity Monitor. I got 442.9 MB on the first run and 650.7 MB on the second.

[gregwebs]

One thing that might be related to this: I had slow performance with const True on a large code base, but once I filtered down to the files I actually needed it worked fine.

[sethfowler]

It's very possible that they're related, yeah. In my real application I filter out everything but C++ source files and the leak does seem slower. If filtered out events don't contribute to the leak, that is quite interesting. I'm not 100% sure there's actually an effect, though, because there's so much variance in the leak size.

[sethfowler]

The hfsevents package comes with a trace program in the tests directory that does something very similar to what my test program above does. The difference is that trace does not leak memory. I'm afraid that the leak does not appear to be in System.OSX.FSEvents. It must be in this package.

[sethfowler]

@gregwebs I'm thinking of trying to get this fixed sometime in the near future. Which repo and branch should I work from?

[gregwebs]

the rc branch. Thanks!

[thomasjm]

@sethfowler I realize this was 2 years ago, but did you ever make any progress on memory leak issues? I'm running into it on Ubuntu linux as well. Setting up a watch with a broad predicate (essentially just const True) causes big memory usage.

In addition, I noticed that if I trigger a large number of files changes by invoking chown -R on the watched folder, then the memory usage becomes even bigger (like hundreds of megabytes in my case)

[sethfowler]

@thomasjm I didn't, unfortunately. I just ended up periodically killing and restarting the thread on a timer.

[thomasjm]

Note to self: try replacing Chans with strict-concurrency versions and see if it helps

[simonmichael]

We see this with hledger-ui too. [Using watchDir, with a restrictive predicate matching only a few files.]