TheSpaghettiDetective / OctoPrint-Obico

GNU Affero General Public License v3.0
136 stars 42 forks source link

[BUG] Janus causing memory leak on Raspberry Pi #197

Open FliesWithWind opened 1 year ago

FliesWithWind commented 1 year ago

Hey,

I've noticed today a possible memory leak in "janus" used by Obico on Raspberry Pi.

My raspberry crashed today after 3hours of printing and streaming in high quality. I wasn't sure what caused it at first, but I've kept monitoring it using "htop" during the rest of the print. Janus memory usage when from 10% (~100mb) to 70% (~700mb) after 3 hours.

Janus version used is 0.8.2 which is a legacy version now. Also, after some googling, I found a few mentions of fixed memory leaks.

Also, there is this Reddit topic about Obico memory leak on Reddit, which might be related to this. https://www.reddit.com/r/klippers/comments/xas3gb/system_memory_takes_about_15_minutes_to_get_to/

I plan to install and try out the latest "janus" version to see what happens. I'm in a middle of a print now, so will try this tomorrow and also maybe provide some screenshots as proof of the memory leak.

kennethjiang commented 1 year ago

The bug mentioned in https://www.reddit.com/r/klippers/comments/xas3gb/system_memory_takes_about_15_minutes_to_get_to/ was fixed a long time ago.

700MB is too much. Not sure why this happens. I doubt it's the version issue though.

FliesWithWind commented 1 year ago

Attaching some initial htop images. After 55 minutes of print memory usage went up from 1.6% to 5.8% and It seems to be growing exponentially, since by the time i wrote this, it grew to 6.3% in 5 mins.

I've tried using other janus versions, but I couldn't get them stable enough to work without flaws. I'll post another screenshop once I'll approach a higher number of memory usage.

htop htop2
FliesWithWind commented 1 year ago

I may have exaggerated with the exponential part but the usage keeps growing.

Since I started printing, I'm watching the prtiner using stream sharing (example URL: https://obicourl/printers/share_token/a8bfba1f179de60153217b2a02ce9484521/) and I'm pretty sure the issue is there. There were couple of fixes for memory leak in the streaming plugin. Maybe that's it https://github.com/meetecho/janus-gateway/blob/e8d1395d3de9caa5e244605ec1a7281cd7d3ecf1/CHANGELOG.md#v0112---2021-05-03 https://github.com/meetecho/janus-gateway/blob/e8d1395d3de9caa5e244605ec1a7281cd7d3ecf1/CHANGELOG.md#v091---2020-03-10

I've compiled and tested the latest janus multistream and legacy versions, but in both cases, I could not make the stream sharing to work, so I reverted back to the version in the repo here.

htop3 htop4
kennethjiang commented 1 year ago

Something is not right on your Pi. On my pi, janus never goes over 1% in CPU, and memory hovers around 110MB no matter how long I have been printing.

FliesWithWind commented 1 year ago

Have you been watching WebRTC stream while printing for a long time? This is how I manage to replicate it.

  1. Reboot Raspberry Pi
  2. Start printing
  3. Open printer stream sharing (Example: https://obicourl/printers/share_token/a8bfba1f179de60153217b2a02ce9484521/)
  4. Monitor htop for increasing memory usage.

However, I have found a kind of memory leak walkaround. Once the memory leak happens, I kill janus process and start.sh. This causes janus to restart, but with the memory leak gone.

Anyhow, you guys are planning to update janus at some point, right? :)

kennethjiang commented 1 year ago

Yes this code has been there for a long time. I'm pretty sure myself, as well as a lot of other users, have gone through this test case.

And yes we plan to upgrade to janus 1.x in the near future. The only reason why we still keep it at 0.x is because some users are still using the old mobile app that is using features only in janus 0.x