Docker CPU usage idling at 1-8% spikes at 20-40%

[vanhoopstallion]

Hello

I'm using the docker nightly or latest branch as a docker on synology NAS. I can see the CPU % on idle is 1-8% and spiking to 20-40% briefly on occasion. Please tell me the logs you may need

[Tzahi12345]

Is there a way you can check the processes running in the container? It might be zombies or rogue processes.

Also, I would recommend going on the nightly Docker tag and checking if the issue persists there (a fix for CPU usage was implemented there)

[vanhoopstallion]

Hello,

I'm currently running the latest nightly from installing on June 5th. I'm seeing spikes every so often. Attached is a graph from portainer

[Tzahi12345]

Hm still not obvious what's causing this. Just a few questions so we can narrow down the culprit:

How many subscriptions do you have? Also, what do you have set as your subscriptions check interval (this is in the settings)? Spreading these out might help reduce the spikes.

Also, maybe you can check the logs to see if anything shows up there (like any errors). Also, if you can peek into the container and check the processes that are running (and their CPU usage), that would also help.

[vanhoopstallion]

I have 3 subscriptions & 4 playlist subscriptions. The interval check is 300 seconds.

I have attached my logs from a restart this morning and I can only see 1 process running with attached screen shot. I've managed to grab screen shots of when a process kicks in, roughly every minute or so.

LOG 2020-06-07 07:39:52 | stdout | 2020-06-07T07:39:52.287Z INFO: YoutubeDL-Material v4.0 started on PORT 17442 2020-06-07 07:39:48 | stdout | sh: app.js: unknown operand

[Tzahi12345]

Thanks for the screenshots! So it looks like the problem is subscriptions.

Some background: the checks for each subscription is spread out over the course of the subscription interval (300 seconds). You have 7 subscriptions in total, so every 300/7 seconds (~43s) another subscription is checked.

Here's what I suspect is happening: those CPU spikes above 25% (e.g. 100%+) can be multiple subscriptions being checked at the same time. As in, maybe your first subscription takes 1 minute to fully check, so 43 seconds in, you're now checking the first and the second subscription at the same time.

This stacking effect is limited to 7 concurrent subscriptions in your case because of a fix I implemented (briefly mentioned before in this thread). So my suggestion here is to increase the subscriptions check interval, perhaps to 600 or 900 seconds. In the future, I want to change this setting to be the check interval for each individual subscription, not the interval for all the subscriptions (so you don't have to keep increasing it as you add subscriptions).

Anyways, hope this helps!

[vanhoopstallion]

thank you

I've made the updates as you suggested.

So is it fine that the spikes are happening like that? That is expected?

I ask as I've been watching the processes (boring!!) and can only see one sub being checked each time. On some occasions two, but typically one with 20-30% CPU usage. This is based on the default 300 seconds interval.

Thanks for a great docker!

[Tzahi12345]

So is it fine that the spikes are happening like that? That is expected?

Hm, not sure, and that's why I'm keeping this issue open. In the past, I've seen CPU spikes from subscriptions with the setting "download videos in the last xxx {units}" turned on. For some reason, this setting causes extremely high CPU usage (it's a problem on youtube-dl's end).

I ask as I've been watching the processes (boring!!) and can only see one sub being checked each time. On some occasions two, but typically one with 20-30% CPU usage. This is based on the default 300 seconds interval.

Haha sorry you had to go through that! The data is helpful though, it seems like each individual check is simply eating up a lot of usage.

If this becomes a problem for your whole system, it is possible to limit the container's overall CPU usage, so that might be helpful as well.

[web-connect]

I have this problem too. I don't have hard metrics now, but I've had to kill my instance due to it spiking resources.

[Sputgnocchi]

Hey, sorry for digging this topic out again but I believe I found the source of CPU usage spikes. I've been using the latest nightly docker version and often my CPU was reaching 25% for several minutes, even thought nothing was happening in the background.

Also, I only have 2 subscriptions and my interval to update them was set to every 3h (10800s). Anyway, after doing some tests I finally brought my cpu to <1% usage in idle after I disabled the player; that is, activate the "download mode only".

Good luck!

[Tzahi12345]

@Sputgnocchi I think you're exactly right, great digging!

See the screenshot below. The time 1 and 2, a video was initially loaded. Time 3 and subsequent mini-spikes corresponded to when additional buffering was done.

I'll keep digging and see if I can find a performant way to stream video from the server, because it shouldn't be eating up so much CPU power.

[Sputgnocchi]

@Tzahi12345 I eyeballed it for a while and the behaviour was very consistent. Every 5-10min I would get a 25-28% spike that lasted for a while (~1-2min) with nothing going on in the background, not even a recent download that would justify some kind of caching. Since the player was deactivated, the CPU is constantly at <1% in idle.

Great app btw, it's been very helpful. And thanks for the active support.

[Sputgnocchi]

Oh well.. forget it.. I still have the issue. After a while the it started to use the cpu 15% even on idle.

Not sure what to do besides stopping the docker when I'm not using it.

[Tzahi12345]

@Sputgnocchi Sorry for the delay in getting back to you, I wasn't able to find a solution that reduced CPU usage. The main issue is that Node.js is serving the videos and it's not really built for that. For large videos in particular this can be very intensive.

One way around this is by utilizing apache2 or nginx to take care of it instead, but then I'm requiring users to install a web server which I moved away from for other reasons as well.

Limiting CPU usage seems possible with docker-compose, I think adding the cpus option can do it though I'm not sure exactly how this works (to me it looks like it would limit it to 10% usage)

services:
    ytdl_material:
        cpus: 0.1

[tehniemer]

I'm also experiencing this with the latest nightly. Memory usage is a similar graph.

[tehniemer]

Any updates on this?

[GlassedSilver]

@Sputgnocchi Sorry for the delay in getting back to you, I wasn't able to find a solution that reduced CPU usage. The main issue is that Node.js is serving the videos and it's not really built for that. For large videos in particular this can be very intensive.

One way around this is by utilizing apache2 or nginx to take care of it instead, but then I'm requiring users to install a web server which I moved away from for other reasons as well.

Limiting CPU usage seems possible with docker-compose, I think adding the cpus option can do it though I'm not sure exactly how this works (to me it looks like it would limit it to 10% usage)

services:
    ytdl_material:
        cpus: 0.1

What's the reason you shied away from this?

I did some research into ways to optimize node.js apps something that I keep stumbling over is that serving static assets is repeatedly recommended to be handled by a dedicated web server, e.g. in a separate docker.

Whilst it increases the amount of containers/apps this project utilizes it can actually help run more economically and maybe in some cases at all on lower-end servers, e.g. Raspberry Pis when there's a lot of other stuff going on.

Just wondering, and I'd be curious to hear what made you reject the idea.

[GlassedSilver]

I'm also experiencing this with the latest nightly. Memory usage is a similar graph.

[...]

Do you have subscriptions set up and if so how many? And with which check interval?

And which ytdl binary are you using? yt-dlp?

All in all, I can tell you that those subscriptions are a very intensive task, the approachability of running those checks so easily can make you underestimate how much processing is going on in the end. Although I do agree that this should probably be more efficient, if it's subscription checks causing it rather than watching your downloads in YTDL-M's player I see little immediate potential to greatly reduce this from our end, since for subscriptions we basically act as middleware for youtube-dl or its forks yt-dlc and yt-dlp.

[yllekz]

Does anyone have a good working docker-compose configuration that can set CPU/Memory limits for these containers? The issue I'm running into is that youtubedl-material and its DB container are burning up so much CPU/RAM that it actually takes down the entire server it's running on. I want to give both containers a max CPU/RAM threshold to work with so that doesn't happen.

I found this page on StackOverflow on limiting the CPU/RAM in docker-compose, but wasn't sure if this would work for youtubedl-material:

version: "3.9"
services:
  database:
    image: mariadb:10.10.2-jammy
    container_name: mydb
    environment:
      MYSQL_ROOT_PASSWORD: root_secret
      MYSQL_DATABASE: mydb
      MYSQL_USER: myuser
      MYSQL_PASSWORD: secret
      TZ: "Europe/Zurich"
      MARIADB_AUTO_UPGRADE: "true"
    tmpfs:
      - /var/lib/mysql:rw
    ports:
      - "127.0.0.1:3306:3306"
    deploy:
      resources:
        limits:
          cpus: "4.0"
          memory: 200M
    networks:
      - mynetwork