search

TheSpaghettiDetective / OctoPrint-Obico

GNU Affero General Public License v3.0
136 stars 42 forks source link

[BUG] Possible memory leak with Obico `ffmpeg` video streaming #217

Open calexandre opened 7 months ago

calexandre commented 7 months ago

I've been struggling with an instability issue on my OctoPrint instance, where, from time to time, the system runs out of memory. After some debugging, I managed to isolate the issue to the Obico plugin, specifically the ffmpeg process.

It seems that when I have the streaming enabled (which is the purpose of Obico), the ffmpeg process is automatically spawned and slowly consumes memory until OctoPrint crashes due to out-of-memory issues.

This process takes a few days before memory pressure starts causing instability in the entire system, eventually leading to OctoPrint crashing and failing the print.

I isolated this issue by letting the system run for some time with and without the Obico plugin enabled, which led me to open this issue.

To mitigate this problem, I set up a cron job to kill the ffmpeg process every day.

To Reproduce Steps to reproduce the behavior:

  1. Enable Obico streaming.
  2. Monitor the ffmpeg process and check the memory it is consuming.
  3. After a couple of days, check the ffmpeg process and notice the sudden increase in memory. Depending on the interval, you may already be experiencing swapping problems due to a lack of available memory.
  4. If you don't kill the ffmpeg process, OctoPrint will eventually crash.

Screenshots

Day 0 - Enabled Obico streaming - ffmpeg is consuming almost 4% of memory, which is fine.

ffmpeg-day0

Day 1 - After 27 hours, the ffmpeg process is already taking up almost 30% of memory.

ffmpeg-day1

Desktop (please complete the following information):

Additional context

kennethjiang commented 7 months ago

Sorry I have been really busy post-Thanksgiving.

This sounds like a pretty bad problem. However I haven't been able to reproduce it.

I notice you are using RTSP stream provided by OctoPrint's new webcam streaming stack. Can you disable it (do the opposite of https://community.octoprint.org/t/got-rtsp-on-octopi-working/54091), restart the Pi, and see if ffmpeg will still have memory leak?

Also do you remember if this problem has always been there, or it started happening at some point? If the latter, do you remember if there was any change to your setup right before this started to happen?

kennethjiang commented 7 months ago

Oh I'm 99% sure the datachannel false alarms in the log file doesn't have to do with this issue. That was an annoying false alarm in Janus, another 3rd-party software that we don't have much control over... :(

calexandre commented 7 months ago

Sorry I have been really busy post-Thanksgiving.

This sounds like a pretty bad problem. However I haven't been able to reproduce it.

I notice you are using RTSP stream provided by OctoPrint's new webcam streaming stack. Can you disable it (do the opposite of https://community.octoprint.org/t/got-rtsp-on-octopi-working/54091), restart the Pi, and see if ffmpeg will still have memory leak?

Also do you remember if this problem has always been there, or it started happening at some point? If the latter, do you remember if there was any change to your setup right before this started to happen?

Sorry for the delay... I will your suggestion to check if we can narrow it down...

Regarding your other question I cant recall exactly any particular change...this is something i've been tracking for a while (lets say, 6 months max?), because my OctoPrint instance was getting very unstable and I finally started the debug process to find the culprit.

Ever since I created a cronjob watchdog that kills the ffmpeg process every time it reaches 30% memory, the system has been rock solid stable.

kennethjiang commented 7 months ago

It sounds like the issue has been going on for a while now.

I guess you were using the octopi 1.0 SD card image? If so, have you done system upgrade? I'm wondering if it's possible that a system lib was upgraded to a version that causes the memory leak.