search

microscope-cockpit / cockpit

Cockpit is a microscope graphical user interface. It is a flexible and easy to extend platform aimed at life scientists using bespoke microscopes.
https://microscope-cockpit.org
GNU General Public License v3.0
37 stars 27 forks source link

Memory leak. #815

Closed iandobbie closed 2 years ago

iandobbie commented 2 years ago

I have been distracted and left cockpit running but doing nothing in full simulation mode over several days. Eventually the process was at 70GB and my OS complained about low memory. Not sure how best to diagnose what is growing so much. I am happy to start a machine and leave it running if someone can advise on the best way to find large objects in the process.

carandraug commented 2 years ago

How much is several days? Assuming a linear increase in the memory usage, maybe overnight is enough to notice something. If so, can you share the cockpit configuration? I will try to replicate that on my side too.

iandobbie commented 2 years ago

I am not entirely sure, I think it was 3 days. I was running in pure simulation mode, but I am not entirely on the master branch, I am trying to fix the camera window size issue. My plan was to try again on a machine I can just leave running with a vanilla master codebase.

iandobbie commented 2 years ago

Quick update.

Reverted to master started with "cockpit --no-config-files"

Start cockpit, give about 300 MB memory footprint.

Did no interaction with cockpit at all. The memory footprint is growing at about 3MB/sec

carandraug commented 2 years ago

Start cockpit, give about 300 MB memory footprint.

Did no interaction with cockpit at all. The memory footprint is growing at about 3MB/sec

I can't reproduce this issue. Cockpit starts with 193MB memory (this seems a lot) but it stays there. I'm on Debian 10 running Python 3.7 with WxPython "4.1.1 gtk3 (phoenix) wxWidgets 3.1.5".

iandobbie commented 2 years ago

My machine is running macOS 11.6.6 (on an M1) Python 3.9.13 and 4.1.1 osx-cocoa (phoenix) wxWidgets 3.1.5 Memory footprint now about 12 GB after about 4 hours.

mickp commented 2 years ago

Wow. It sounds like something is not getting disposed of correctly in the event loop. I remember I had to fix a bunch of wx context cleanup at some point - the code didn't follow the wx documentation and while it worked fine on MSWin, it caused problems on Mac. It might be worth checking that any custom drawing stuff is cleaning up after itself correctly.

On Thu, 2 Jun 2022 at 18:04, Ian Dobbie @.***> wrote:

My machine is running macOS 11.6.6 (on an M1) Python 3.9.13 and 4.1.1 osx-cocoa (phoenix) wxWidgets 3.1.5 Memory footprint now about 12 GB after about 4 hours.

— Reply to this email directly, view it on GitHub https://github.com/MicronOxford/cockpit/issues/815#issuecomment-1145485244, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABHGTL4SZ34NHN52UKDT6ODVNFK2PANCNFSM5XVDMK5Q . You are receiving this because you are subscribed to this thread.Message ID: @.***>

--

Mick Phillips

carandraug commented 2 years ago

Since the issue is so noticeable, you could just comment out all the extra windows at https://github.com/MicronOxford/cockpit/blob/f0a548630e1e3fbc718230e91d6c2b90d3fc66de/cockpit/__init__.py#L184-L193

and add them one by one until you can see the memory leak again.

iandobbie commented 2 years ago

Nope, comment out all this code section still ends up with a memory leak so it's not there.

juliomateoslangerak commented 2 years ago

This is very similar to issue #775, for which I found a 'fix' without really understanding what is happening.

iandobbie commented 2 years ago

Yes, except that this happens even when the cockpit process is entirely idle. On my mac when I start cockpit the process just starts to grow, eventually crashing the machine if left running long enough.

How did you generate your memory size plots? I did some searching but couldn't find an easy way to track where the memory was getting taken up within the python process. I'm sure there are tools to do this but I just couldn't find one which seemed to provide the info I needed.

juliomateoslangerak commented 2 years ago

Ah, that is indeed different. I looked for long to find a easy to use memory profiler and I went for memory-profiler :-) https://github.com/pythonprofilers/memory_profiler with the command mprof run -C cockpit\__main__.py

iandobbie commented 2 years ago

I went back to visit this again today with a view to trying Julio's suggested profiler and someweher along the line (probably a brew update) I've now moved to Python 3.10.6 and wx 4.2.0 osx-cocoa (phoenix) wxWidgets 3.2.0 and it appears as though the issue has vanished. I read this as it being a mac specific wx bug, but I'm not sure.

I will do some more testing and try to leave cockpit running for a while and see if I can see any memory creep. If it really has gone I'l close this issue.

iandobbie commented 2 years ago

This is now fixed. I have tested pretty extensively and the memory leak appears to be gone.