search

jneilliii / OctoPrint-TabOrder

22 stars 7 forks source link

(Possible) Memory Leak #33

Closed jrezin closed 4 years ago

jrezin commented 4 years ago

I'm not sure how to properly debug and provide proper data about this bug, but basically, enabling this plugin causes a huge memory leak and disabling it fixes it. I've initially posted it, before narrowing the problem TabOrder, in this issue OctoPrint/OctoPrint#3405

octoprint_memoryleak20200331 png

jneilliii commented 4 years ago

You might want to check this again while not on the control tab...that memory consumption is because of the webcam I think.

jrezin commented 4 years ago

Yes, I did that. I happens while outside the webcam too, just starting octoprint and leaving it be, causes the memory consumption to grow and keep growing you terminate the process or crashing.

I've so, 6 GiBs of extra memory... against 150-250 MiB (even with the webcam in the control tab) of normal consumption (with TabOrder disabled).

jrezin commented 4 years ago

This is Chrome's Task Manager. You can summon it with Shift+Esc in the chrome window, or by 3dots > More tools > Task manager.

As the memory leak itself, I don't know what to say. I'm consistently getting it every time I enable the plugin. If there's a procedure to better track the problem, I'd like to know how.

jneilliii commented 4 years ago

Yeah, I see it growing in the regular task manager. Not sure what has introduced this or if it's been there from the beginning. I'll try to figure this out.

jneilliii commented 4 years ago

Ok, I think I've isolated the issue down to the latest release that incorporated an external library (bootstrap-tabdrop) for the ability to push items into the drop-down list, effectively "hiding" the tab from default view. This basically overwrites the bundled version of OctoPrint. Now, figuring out why it is happening will be a challenge,

jneilliii commented 4 years ago

I think I got it. Went back the default bootstrap-tabdrop module included with OctoPrint and just patched it with the settings needed to push the hidden tabs into the drop-down. The caveat is we lose the label on the dropdown for the active tab, but I think killing a memory leak is more important. If you don't mind testing to confirm this does fix the issue that would be greatly appreciated. Use the URL below in Plugin Manager to install the updated version.

https://github.com/jneilliii/OctoPrint-TabOrder/archive/0.5.7.zip

jrezin commented 4 years ago

After finishing a print, I've installed 0.5.7, enabled it and left the UI open and running overnight. Unfortunately this morning OctoPrint is using 3.6GiBs of memory.

octoprint_memoryleak20200405

jneilliii commented 4 years ago

Did you force a browser refresh to overwrite the local cache?

jneilliii commented 4 years ago

Never mind. It's still a problem. I didn't leave it long enough before to know for sure. I'm curious if this issue has been there longer than my tabdrop modifications. Going to install an old version and see if it's the same.

jneilliii commented 4 years ago

So I'm starting to think my previous memory growth was coincidental. I've been running two tabs overnight, one with version 0.5.7 (highlighted) and one with version 0.5.5 and it's showing "normal" memory consumption.

image

jrezin commented 4 years ago

I'm testing version 0.5.4, and after few hours, it looks fine. Octoprint using about ~140MiB of memory, steady.

jneilliii commented 4 years ago

I haven't shut down my browser tab running version 0.5.7 since last night and it is steady at ~200MB now, so I think whatever issue was there before is either specific to a certain condition/steps or not related to the plugin at all.

jrezin commented 4 years ago

I didn't have problems with 0.5.4 after several hours. upgraded to 0.5.5. and after about an hour, memory usage have past 800 MiBs already.

jneilliii commented 4 years ago

Yeah, this is the strange part because both my 0.5.5 version and 0.5.7 version are still in the 200s.

image

Curious if it's a conflict with another plugin maybe or like I mentioned before some other random occurrence/conflict like mentioned in the other post in regards to update notifications maybe.

jneilliii commented 4 years ago

I did just notice something @jrezin that is different to what you have versus what I have on my 0.5.7 version instance. You appear to be using "icon only" mode by not showing any labels, but on mine my labels are visible. At this point I'm just stabbing in the dark to isolate this issue and determine if it's specific to TabOrder or not.

Edit: Doesn't appear to be icon only options. I changed my 0.5.7 instance to just icons and I'm still floating in the 200s.

jneilliii commented 4 years ago

12 hours later and my instance is still hovering in the 200s...

jrezin commented 4 years ago

well, my setup current other enabled plugins:

jneilliii commented 4 years ago

@MFHFozzy, do you have any of these same plugin's installed?

MFHFozzy commented 4 years ago

1 has GecodeEditor.

2 has Cancel objects. (#1 might have had this before a couple days ago...)

Both have Gcodebar Plugin, LayerDisplay, Themeify.

jneilliii commented 4 years ago

Common Plugins So Far and some assumptions since I don't use all these:

Once you see the higher memory usage does refreshing the page drop the memory back down?

jrezin commented 4 years ago

"OctoPrint 1.4.0 running on OctoPi 0.17.0".

LayerDisplay - https://plugins.octoprint.org/plugins/layerdisplay/

0.5.7 compared to 0.5.4, its 100MiB of extra memory usage from the start. 0.5.7 does leak memory much less/slower than 0.5.6. Refreshing does drop the memory usage back down (after a minute or two), but not all the leak (after refreshing, still a LOT more memory than Ending Tab's Process). With time, I'll leak a lot again.

jneilliii commented 4 years ago

Thanks @jrezin, the memory jump from 0.5.4 to 0.5.7 is more than likely due to the inclusion of the fontawesome icon picker. Let me pull together an update to 0.5.7 without that feature and let's see if the issue is still there or not and that will help in identifying if that is the true cause or not. Updates to come for additional testing.

MFHFozzy commented 4 years ago

I reinstalled tab order 0.5.6 yesterday (from repo) on #1 of my octopies, but so far today no leaking. On the other, i just re-installed bed leveler 0.1.13 (from repo) and restarted it to see what happens.

jrezin commented 4 years ago

a few hours in with https://github.com/jneilliii/OctoPrint-TabOrder/archive/0.5.7_iconpicker_removed.zip and it looks like 0.5.4 levels of memory usage, does not appear to be leaking for now.

jneilliii commented 4 years ago

So I think I've been able to get the full functionality from before without the memory leak. So far my testing of https://github.com/jneilliii/OctoPrint-TabOrder/archive/0.5.7_pureComputed.zip which converts the computed observables to pureComputed observables and removes the fa-lg class from being automatically added is doing well. If you do try it out, make sure to force a browser refresh.

jrezin commented 4 years ago

installed 0.5.7_pureComputed.zip last night, was using 0.5.7_iconpicker_removed.zip before. Unfortunately the memory usage this morning is 3.9 GiB. Changing tabs didn't make a difference. Refreshing OctoPrint did drop the memory usage, but to around 950 MiBs at first, and 650 MiB after 1 ou 2 minutes, which is still much more than normal.

jneilliii commented 4 years ago

That's so strange. I left mine running overnight and although it did grow, it didn't get anywhere near that much. It did double though. @MFHFozzy is your #1 printer still not showing signs of memory leak with the official 0.5.6 release?

MFHFozzy commented 4 years ago

Im trying to nail things down better, as nothing is making sense. Using/Monitoring Waterfox about:memory, i am so far unable to reproduce the leak... but the initial report was using Opera, and looking at its task manager.

In frustration, i opened octopi the other night in chrome, and i also did a print last night (small, 1 hour). This morning was at 822MB. (yay, at least i got a reproduction) Ive switched back to water fox, and am doing the same print right now. If no leak again on waterfox, ill re-test with/without plugins on chrome, unless anyone has a better idea?

Logging so far: Memory Tracking Octopi

2020-04-05 1:50pm, removed tab order, bed visualizer from both 96.69 MB (05.52%) ++ top(http://192.168.0.92/#control, id=1593) Connected 107.12 MB (06.11%) ++ top(http://192.168.0.93/#temp, id=1599) Not connected

2020-04-07 4:13pm 102.25 MB (05.90%) ++ top(http://192.168.0.92/#temp, id=1593) 44.24 MB (02.55%) ++ top(http://192.168.0.93/#temp, id=1599)

2020-04-08 2:51pm 91.15 MB (04.99%) ++ top(http://192.168.0.92/#temp, id=1593) 40.11 MB (02.20%) ++ top(http://192.168.0.93/#temp, id=1599)

2020-04-08 2:53pm reisntalled tab order 0.5.6 on .92 from repo, restarted 104.39 MB (05.68%) ++ top(http://192.168.0.92/#temp, id=1593) 39.70 MB (02.16%) ++ top(http://192.168.0.93/#temp, id=1599)

2020-04-09 5:49pm 94.27 MB (05.67%) ++ top(http://192.168.0.92/#temp, id=1593) 40.81 MB (02.46%) ++ top(http://192.168.0.93/#temp, id=1599)

2020-04-09 5:49pm reisntalled bed visualizer 0.1.13 on .93 from repo, restarted

2020-04-10 1:57pm 84.70 MB (04.91%) ++ top(http://192.168.0.92/#temp, id=1593) 87.51 MB (05.07%) ++ top(http://192.168.0.93/#temp, id=1599)

2020-04-11 2:37am 78.58 MB (04.23%) ++ top(http://192.168.0.92/#temp, id=1593) 57.06 MB (03.07%) ++ top(http://192.168.0.93/#temp, id=1599)

2020-04-11 2:07pm 80.88 MB (04.25%) ++ top(http://192.168.0.92/#temp, id=1593) 57.16 MB (03.01%) ++ top(http://192.168.0.93/#temp, id=1599)

2020-04-11 2:15pm reinstalled tab order 0.5.6 on .93, bed visualizer 0.1.13 on .92 257.40 MB (10.75%) ++ top(http://192.168.0.92/#temp, id=1593 127.34 MB (05.32%) ++ top(http://192.168.0.93/#temp, id=1599)

2020-04-11 3:19pm 106.92 MB (04.86%) ++ top(http://192.168.0.92/#temp, id=1593) 72.83 MB (03.31%) ++ top(http://192.168.0.93/#temp, id=1599)

2020-04-11 3:19pm 76.53 MB (04.05%) ++ top(http://192.168.0.92/#temp, id=1593) 70.18 MB (03.72%) ++ top(http://192.168.0.93/#temp, id=1599)

2020-04-13 1:10pm 93.93 MB (03.97%) ++ top(http://192.168.0.92/#temp, id=1593)
60.43 MB (02.55%) ++ top(http://192.168.0.93/#temp, id=1599)

Chrome: 2020-04-13 to 2020-04-14 10:07am .92 822,480K (did a print alst night) .93 101,396k (no print, still not connected) Switched back to Waterfox (10:10am) 142.08 MB (06.04%) ++ top(http://192.168.0.92/#temp, id=1593) did same print

jneilliii commented 4 years ago

Great stats @MFHFozzy, but for the most part it looks like the leak is not present until that last day in chrome, which is what I've been testing with and haven't been able to get reliable reproduction of the issue. That did make me think though, what extensions do you guys have enabled/installed in chrome? I noticed during my memory heap snapshots in developer tools that there was memory being consumed by installed extensions, not just the page content.

This may be something else that you guys could do to help diagnose. On the memory tab of developer tools take a heap snapshot, wait a couple of hours then press the little delete button followed by another snapshot, then switch the view mode to Statistics (or compare) and let's see where the consumption is happening.

image

The files end up being pretty big, but if you can upload/share them somewhere it might also help. Like I said I haven't been able to reliably reproduce. There was another user on the community forum that was discussing memory leaks with AutomaticShutdown plugin. Also, have either of you tried with just TabOrder enabled to see if the issue persists?

MFHFozzy commented 4 years ago

I have switched back to chrome, and will take some snapshots as described.

MFHFozzy commented 4 years ago

I should mention that i has also saved a bunch of the measure reports from Waterfox as well. I dont know how to use them though :) I can post them if that will help.

MFHFozzy commented 4 years ago

My printers are currently non-stop printing face shield parts for the covid-19 efforts in LA, so 1 change is that i cant let them just "sit" for a day anymore. That being said, here is some additional findings:

I switched back to chrome @ 12:01am, took snapshots and recorded task manager tab memory values, then started another 3 hour print:

2020-04-17 12:01am Started 3 hour print Task manager: .92 139 MB .93 112 MB Heap Snapshots: .92 19 MB .93 21.7 MB

~3 hours later, after the print was done: Task manager: .92 1.2 GB .93 348 MB Heap Snapshots: .92 20.6 MB .93 23.4 MB

So Chrome task manager sees a huge memory tab use by Octoprint, but the heap doesnt?

Ive attached all the log files i have so far. Both heap files and the waterfox mem dumps I did a quick look over to make sure theres no personal or identiy data in them..but if you see/know of something that should be removed, please let me know.

92-Heap-20200417T120509.zip 92-Heap-20200417T154020.zip 93-Heap-20200417T153957.zip 93-Heap-20200417T120611.zip 2020-04-05x1-50pm-memory-report.json.gz 2020-04-11x01-38am-memory-report.json.gz 2020-04-11x03-02pm-memory-report.json.gz 2020-04-15x09-55pm-memory-report.json.gz 2020-04-17x11-58am-memory-report.json.gz 2020-04-07x12-56pm-memory-report.json.gz 2020-04-08x02-49pm-memory-report.json.gz 2020-04-10x01-57pm-memory-report.json.gz

jneilliii commented 4 years ago

Thanks @MFHFozzy, not sure about what's captured in those heap dumps but from what I've seen so far I don't think there's any personal information in them.

I was seeing the same discrepancy between chrome task manager window and the heap snapshots, not sure what that means. I'm new to memory leak debugging.

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had activity in 14 days. It will be closed if no further activity occurs in 7 days.

jneilliii commented 4 years ago

Releasing version 0.5.8 that should resolve the memory leak issue and retain the icon picker by binding it only once to a popup editor form instead of every row.