Docker: Very high memory usage while remuxing (eventually leads to OOM)

[GGG-KILLER]

Describe the bug When remuxing a relatively long stream (10+ hours), the docker container appears to hit a 6 GB RAM usage which then leads to an OOM on my home server as it doesn't have any more available memory.

To Reproduce Steps to reproduce the behavior:

1. Start capturing a long stream from the start.
2. Stream ends.
3. Remux starts (along with VOD and chat download).
4. Container hits 6 GB memory usage and OOMs.

Expected behavior Memory usage doesn't explode when remuxing.

Screenshots

Logs OAuth token, public URL, youtube callback IDs and other possibly PII info has been removed, but I don't know what else could also classify as private info so please let me know if I goofed up.

capture_michimochievee_41650896199_stderr.log capture_michimochievee_41650896199_stdout.log chatdump_michimochievee_41650896199_stderr.log chatdump_michimochievee_41650896199_stdout.log docker-live-stream-dvr.service.log download_vod_2272194386_stderr.log download_vod_2272194386_stdout.log LiveStreamDVR-2024-10-09.log LiveStreamDVR-2024-10-10.log remux_2024-10-10 01-13-39 MichiMochievee - 🔴 ADVENTURING WITH MY SHINY POKEMON! - POKEMON WHITE pt.3 !merch !tts !game !discord !collab [41650896199].ts_stderr.log remux_2024-10-10 01-13-39 MichiMochievee - 🔴 ADVENTURING WITH MY SHINY POKEMON! - POKEMON WHITE pt.3 !merch !tts !game !discord !collab [41650896199].ts_stdout.log

Server:

• OS or Docker: NixOS 24.11.20241004.bc947f5 (Vicuna) / Docker version 27.3.0, build v27.3.0

Additional context Unsure what else to add, but will reply with more info as necessary.

Unfortunately my current monitoring tools cannot capture the processes inside the container nor their memory usage to figure out what exactly was happening inside it.

[GGG-KILLER]

Issue seems to be related to either downloading vods or chatlogs, when I triggered both on the same capture that failed (after re-running the ffmpeg command that got interrupted and fixing the .jsonfor it), memory usage spiked hard:

[GGG-KILLER]

Update: the issue happens with any remuxing, be it the capture or the VOD's. This is it remuxing a 10 hour long capture:

[GGG-KILLER]

Assuming the issue stems from storing entire stdout and stderr in memory unnecessarily (because it gets written to a file anyways): https://github.com/MrBrax/LiveStreamDVR/blob/34a1d961de184a6ac6fc5f00f4bfc90926c21c21/server/src/Helpers/Execute.ts#L367-L378

[MrBrax]

that would explain it, i was thinking of looking into this. a bit weird since i'm pretty sure i use a flag to disable file output on stdout

[GGG-KILLER]

ffmpeg does output quite a bit of stuff on stderr though, might be worth making a flag that disables storing the stdout/stderr for ffmpeg or something similar since important things are already processed line-per-line

[MrBrax]

it shouldn't be enough for an oom though, unless there's something extremely memory inefficient with string arrays.

besides, the remuxing should be using another method that doesn't store the logging in ram

[GGG-KILLER]

it shouldn't be enough for an oom though, unless there's something extremely memory inefficient with string arrays.

besides, the remuxing should be using another method that doesn't store the logging in ram

It is using the Job logging as well (which directly outputs to a file), but exec also stores stdout+stderr in memory.

I'm creating a build that disables this in-memory storage of stdout/stderr and will report back with the results (if any). Ended up getting distracted reducing the size of the docker image though 😅

[GGG-KILLER]

Did a bit of testing on my fork and removing the in memory arrays didn't change anything, the issue seems to be elsewhere.

I'll see if I can get a dump of the process and open it up on chrome to see where all this usage is coming from.

[MrBrax]

i have never gotten the issue myself, i run it outside docker on a shared vps so ram usage can't be high or else it gets killed. must be something specific to docker

[GGG-KILLER]

I've enabled a few node flags to create a memory dump once it runs out of memory, so maybe I'll be able to get some more info on this.

Will report back if I find anything.

[GGG-KILLER]

Well, it definitely seems to be something related to remuxing but I have no idea what. I've removed all .on("data"s from the remuxing path and it still leaks a lot of memory. I even went as far as disabling stdin/stdout/stderr by adding this to the spawn options:

            stdio: ["ignore", "ignore", "ignore"],

but it still results in a really high memory usage.

I've made a few heapdumps and they all seem to indicate it's native memory that's using this, not the Js side:

Unless if I get a full process dump and even try to figure out where the native memory is at I don't have enough knowledge for it so I'm just giving up for now. I'll either try to run this outside of docker or give up, because node.js seems to be cursed somehow.

[GGG-KILLER]

Yeah, I've set it up to run outside of docker and still running into the exact same issue with latest commit on master:

Time to give up I guess.