search

DanielG / ghc-mod

Happy Haskell Hacking for editors. DEPRECATED
Other
677 stars 175 forks source link

Large memory leak when loading the Futhark compiler on Windows 10 #797

Open mrakgr opened 8 years ago

mrakgr commented 8 years ago

When I try to load futhark-pyopencl.hs or any other file from the Futhark compiler in the main source directory in Atom, ghc-mod takes up a large amount of memory and then crashes once it takes up all the available computer memory. That can range from 3.5-5Gb out of 8Gb on my system. I've tried building the latest version of ghc-mod myself and disabling autocomplete in Atom and neither of those two things helped as far as I could tell.

The above is a summary from this thread.

That having said, Futhark itself does take up 2.3-3Gb when loaded inside the Repl, so I am not completely sure whether this is a bug or whether ghc-mod really does require a large amount of memory to work. @lierdakil reports that on Linux, for him it only takes up 1Gb for him. There might also be a worst case in the Futhark parser code as reported by its author.

It would be good if somebody could clone the project and confirm the total memory usage.

Apart from that, please tell me if there is something I could do to assist debugging this issue. I'll do a Futhark related project in Haskell and I'd very much like to do it with the help of real time type checking and autocomplete services. Static languages like Haskell cannot show their full power without interactive feedback.

DanielG commented 8 years ago

Have you tried comparing the memory footprint when you load Futhark in cabal repl or ghci vs ghc-mod? If that's about the same then you ought to report this with GHC HQ otherwise we'll have to look at what ghc-mod is doing wrong here.

mrakgr commented 8 years ago

I used stack repl from the command line to load it in interactive. Is that the same? In the task manager then I observed ghc occupying 2.35Gb. It was higher for Athas. In Atom ghc-mod eats 3.5-5Gb and then aborts with an error.

In the currently released branch (not the master) ghc-mod fails when I use it from the command line to check futhark-pyopencl.hs. The master might be a bit more efficient as it does not run out of memory from the command line, but both of them do so in Atom.

lierdakil commented 8 years ago

@mrakgr, one more thing you might want to try is to run ghc-mod legacy-interactive in futhark project dir to see about memory consumption. I've got a creeping suspicion that find precache work might be responsible for some of this.

mrakgr commented 8 years ago

I've typed it in, but get something to the following effect.

C:\futhark\src>ghc-mod legacy-interactive
:load futhark-pyopencl
NG Invalid argument `:load'\n\nUsage:

I just get NG Invalid argument regardless of what I type in. What exactly am I supposed to do with that command?

lierdakil commented 8 years ago

Just let it sit there for a bit and see how much memory it eats. 26 мая 2016 г. 11:56 AM пользователь "mrakgr" notifications@github.com написал:

I've typed it in, but get something to the following effect.

C:\futhark\src>ghc-mod legacy-interactive :load futhark-pyopencl NG Invalid argument `:load'\n\nUsage:

I just get NG Invalid argument regardless of what I type in. What exactly am I supposed to do with that command?

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/DanielG/ghc-mod/issues/797#issuecomment-221815273

mrakgr commented 8 years ago

I've tried it. I am not sure what it is supposed to be doing, but I do not think this command is working properly. It appears to me as if it is doing nothing. In the task manager it only takes up 30Mb as well.

lierdakil commented 8 years ago

Okay, apparently I was wrong. Try running check src\futhark-pyopencl.hs then (i.e.

> ghc-mod legacy-interactive
check src\futhark-pyopencl.hs
mrakgr commented 8 years ago

Tried it. I see it takes up 3,584Mb, which is close to the limit of what my system can afford. The check passed successfully.

lierdakil commented 8 years ago

But plain old ghc-mod check src\futhark-pyopencl.hs works without taking up this much memory?

mrakgr commented 8 years ago

I am not sure as it terminates, so I had not bothered to keep track of it yesterday. Looking at it now, ghc-mod check src\futhark-pyopencl.hs reaches a high of ~3,300Mb before terminating so I would say the two are even.

In Atom ghc-mod definitely takes up more than this. Also I can see that Atom makes use of two ghc-mod processes even with the flag set to 1. One is much smaller (on the order of 100-300Mb) than the main one though. I am also sure that Atom takes up more memory than 3.5Gb.

lierdakil commented 8 years ago

It seems to me that Atom is using older ghc-mod installation (i.e. not one built from master). Maybe see if there are any lingering installations and try to remove/rename those?

mrakgr commented 8 years ago

No, actually, the tests done now weren't from the version built from master. It is true that I built the project yesterday, but I then removed it due to it having no effect on memory use. I think the ghc-mod checks pass now due to my computer simply having more free memory than when I tested it yesterday.

Edit: Should I try testing with the 5.6.0.0 version?

mrakgr commented 8 years ago

I just tested it with the 5.6.0.0 version. They peak around the same point in terms of memory usage. I just saw it reach 3,384Mb before finishing.

lierdakil commented 8 years ago

It's quite mysterious that Atom's instance of ghc-mod takes up more memory than one run from console, since all it actually does is run ghc-mod legacy-interactive in background and feeds commands into it...

mrakgr commented 8 years ago

I think it does take up roughly 3.5Gb at first if I just let it load and do nothing even in Atom. It actually takes up more and crashes when I move my cursor across the variables. That triggers something in the background and causes it to load even more memory after which it fails.

lierdakil commented 8 years ago

That would be info and type commands. My assumptions were repeatedly proven wrong, so I won't try to guess which one. FYI, you can disable type/info commands on mouseover by setting 'On Mouse Hover Show: Nothing' in h-g-m settings and using keybindings as described here

mrakgr commented 8 years ago

I've tried the above. I think I now have a good idea of what is going on after working on this issue for so much. When the file first loads, the two ghc-mod processes do not build the internal structure and at ~50 and ~300Mb respectively. Now when I right click and ask for type, I see it work in the background and memory usage goes to 3.5Gb for one process right away. When I ask for a different type, the other process goes up to 1.7Gb (while the main one stays at 3.5Gb) which makes the program run out of memory.

I should have mentioned this before, but I actually cannot turn off that other parallel process. When I try to by setting Max Processes to zero, the configuration automatically resets to 2 which gives me a total of 3 ghc-mod processes.

As a workaround, I think eliminating those parallel processes entirely might help. Of course, ghc-mod taking up this much space is probably a bug and should be looked into, but for now capping the total memory usage should take priority.

Edit: I actually do not intend to start the project until July so this is not that urgent for me. It is just a suggestion.

mrakgr commented 8 years ago

@DanielG Let me make a update here since the discussion moved to the other issue page for a bit. @lierdakil did as I suggested in the post above and added a low power feature to haskell-ghc-mod. I tried it it, and discovered that unlike what I'd surmised, even with only a single ghc-mod process available, it will try to eat more than 3.5Gb. Yesterday when I tried it it crashed after allocating more than 4.5Gb.

To fix this issue, it seems that somebody knowledgeable with ghc-mod will have to track down the source of the space leak and plug it. I did say that I can't do it myself, but if there is something I can do, do not hesitate to ask.

DanielG commented 8 years ago

I tried to reproduce this under windows but couldn't get it working with either stack or cabal because of a really strange build failure in futhark. First time I compile it with cabal it works just fine but after that I get a bunch of errors which I don't remember off hand.

With stack that doesn't happen because it bails out early when it sees it already built it but as soon as I run ghc-mod in either case the same errors crop up there..

Probably something broken with the cabal preprocessors or something.

lierdakil commented 8 years ago

Okay, so a bit more info. I just tested master on Linux notebook with 4 gigs of ram and 8-gig swap. Everything is done on the same ghc-mod legacy-interactive instance

  1. Type -- 3.2G
  2. Info -- 3.2G
  3. Type, then info -- 3.2G
  4. Type/info, then check -- 3.2G
  5. Check, then type/info -- over 5G !!!
  6. Check, then check -- over 5G !!!

It seems that cached typechecker information isn't stored correctly between checks?

It seems last time I checked on my desktop, my testing methodology was flawed, i.e. memory alloc/release was faster than reporting interval, so memory useage was underestimated. It seems this is not Windows-specific, although memory figures might differ a bit.

mrakgr commented 8 years ago

@DanielG I never ran into any errors when building it so I cannot help unfortunately. I am not familiar with Haskell anyway. If you cannot get it to work then I suggest opening an issue here. I can vouch for the author's responsiveness.

mrakgr commented 8 years ago

I actually I did run into errors when opening the Futhark in Atom for the first time. For some reason the editor could not find the correct dependencies and I resolved this by wiping stack and upgrading it to the latest version as can be seen in this issue. Maybe the problems you are having are related to this?