search

jmbannon / ytdl-sub

Lightweight tool to automate downloading and metadata generation with yt-dlp
https://ytdl-sub.readthedocs.io
GNU General Public License v3.0
1.7k stars 65 forks source link

weird: ram usage when using ytdl-sub w/ proxmox #1023

Open LECOQQ opened 1 month ago

LECOQQ commented 1 month ago

Hey,

I've been playing w/ ytdl-sub for some days now, on my homelab, w/ a proxmox hypervisor on it, running ubuntu lts 24.04 as a vm containerizing ytdl-sub w/ docker compose.

I've noticed that when downloading videos, the RAM usage of my docker container (and thus my node, and thus my proxmox) will go up for each video downloaded. The thing is, it will not stop going up till it overflow from what is physically present in my homelab (32 G). In doing so, it will crash, killing my node and needing me to restart it manually, then restart the docker and the download. I've found a way to circumvent the issue by fixing a limit on my container (16 G for instance). When downloading, it will do the same behavior then at 16G it will not go up and the downloads will keep going. After the downloads, even after killing the docker container, my proxmox node will stick show that i'm currently using the total amount of RAM that I limited earlier (16G+).

It doesn't seems right to me, and this might be a RAM management issue inside the docker. I think that this should be natively taken into account, or atleast documented somewhere, though I don't know if it's a Proxmox issue or not.

Quentin

jmbannon commented 1 month ago

Can you post your config/subscription @LECOQQ ? I know yt-dlp can sometimes eat quite a bit of memory when scraping large channels for the first time

Yankees4life commented 3 weeks ago

I don't know why but it seems like the Linux version of ytdlp has this nasty memory leak. It certainly doesn't act this way when I'm using windows version of ytdlp. Might be an issue the developers may need to sort on their end...

LECOQQ commented 3 weeks ago

Sorry for the late reply. I've added the docker compose. For the subscription configuration, nothing fancy as I've had troubles dealing w/ it. I've taken the generic conf provided and changed the url to meet a channel I wanted to download. I've tried it on a ubuntu computer and on a ubuntu vm running through a proxmox hypervisor on my homelab. Both have this memory leak that increases after downloading a video.

ytdl-sub-docker-compose.txt