High CPU usage when running Jellystat

[harsh0596]

I'm running jellystat using docker and I'm getting high CPU usage at all times. CPU goes down to normal when i stop the container

CPU usage after stopping the container:

OS: OpenMediaVault (debian linux) CPU: i3-10105 Ram: 16gb Media Library size:

[CyferShepard]

Hi @harsh0596 , do you mind also including the specs of your system. EG OS, CPU,Ram size etc. Aswell as a general size of your media library EG 100+ videos

[harsh0596]

Hi @harsh0596 , do you mind also including the specs of your system. EG OS, CPU,Ram size etc. Aswell as a general size of your media library EG 100+ videos

added

[CyferShepard]

Thanks. Il look into this and see how I can optimise it better.

[cleming]

Hi ! we're seeing intensive CPU and IOPS consumption here too. Feel free to reach out to me if you need metrics or any other information.

[CyferShepard]

Hi @cleming , iv managed to identify a few regions causing these spikes and have optimized them, im still working on the code but a fix for this will be out with the next version.

[linedpaper]

Looking forward to this fix. It keeps pounding my i7, i've had to stop the container until the resource issue is resolved, but otherwise it looks good so far!

[dinosmm]

I am seeing this too. Please correct me if I'm wrong, but my culprit is 'node server.js' process which the container is starting and it ends up taking 30-40% CPU and increases the load on my system. This happens constantly.

[CyferShepard]

Hi @dinosmm , yes this is correct, its due to how imports and activity monitor processes are being run, im currently optimizing this to reduce the cpu and memory load

[CyferShepard]

Improved Performance

[EVOTk]

Same issue for me. Keep my cpu at about 20% ( i3-10100T )

[CyferShepard]

Hi Everyone, iv implemented some optimizations to address both high CPU and Memory usage, if anyone has the time, can you pull the latest unstable image and compare to let me know if the improvements work

[EVOTk]

Hello, Thank you very much! That sounds much better. After startup, and for about 2h, I had some activity coming from Jellystat ( surely indexing, ... ) now the container is quiet :)

[CyferShepard]

Awesome, im glad it worked. The first 2 hours where probably the Full sync running (I assume you have a really large library if took that long). The Recently Added Scan is alot less resource intensive but you should see a short spike in cpu usage every hour for a second or 2, this would be that scan. Hopefully i can further improve the efficiency but i feel now it has a more acceptable resource usage

[dinosmm]

I got unstable going, but on the old database, and it kept load and CPU high again; I assumed it was scanning but the movie etc. counts weren't changing (and were completely wrong), so I deleted all the old data and recreated the container. I'll leave it to scan now, and see how it behaves after a few hours.

[CyferShepard]

Hey @dinosmm , counts dont update until the scan is completed, you can check if the scan is still running under Settings>Logs. The full sync is still abit resource heavy, i only managed to marginally improve that, but by creating a more light weight version of it that does a scan on recently added items only, i was able to reduce the frequency of the full scan so the overall resource usage should now be alot lower

[dinosmm]

Is it normal that the 'Full Jellyfin Sync' is still running? It's been 6 hours since it started. This is on a 34TB library.

[EVOTk]

What is you CPU ?

With i3-10100T ~2h for ~70To

[dinosmm]

3 vCores on a VPS.

[EVOTk]

I have no experience in this sector, but VPS are generally not very powerful. I think it can take a long time on this type of machine.

[CyferShepard]

Hmmm, so im not really sure how well this scales to such a large library, How many items do you have?,also at what stage is the sync at? you should see it on the little toast notifier on the bottom right of your screen (reload the page if it doesn't show).

[dinosmm]

900 in one library, 300 in another. Jellyfin scan takes less than an hour if done from scratch though.

little toast notifier

That showed briefly at the start of the scan, but hasn't shown again for hours. It doesn't show if I refresh at all.

[CyferShepard]

and do you see any activity in your resource monitors? IE CPU/ RAM usage? im trying to see if its actually running or perhaps its so slow that updates a few and far apart. If you see little to no CPU and RAM usage maybe try restarting the container and rerunning the sync, it could be frozen

[dinosmm]

Previously (a few hours ago) I could see 'node server.js' taking up 30% CPU and load up because of it, now I see it running still but much lower CPU usage, and load is down below 1 again. I just stopped and restarted the container; as you said, the full sync job had failed, and the library totals were showing as 10 movies and a few episodes. I started a new full sync again, let's see how it goes. If there are any logs anywhere that you think might be useful, please do let me know where to get them and I can post them.

[CyferShepard]

There should be logs under the settings, if thats blank it means the sync failed on start/fetch. Can you also check the container logs for any error messages

[dinosmm]

Oh yea the logs box in settings was completely blank before and it is blank again now under the 'running' full sync. Docker logs (using 'docker logs' command) doesn't show much:

> jfstat@1.0.8 start
> cd backend && node server.js

[JELLYSTAT] Database exists. Skipping creation
FS-related option specified for migration configuration. This resets migrationSource to default FsMigrations
FS-related option specified for migration configuration. This resets migrationSource to default FsMigrations
[JELLYSTAT] Server listening on http://127.0.0.1:3000
Activity Interval: 1000
Running Recently Added Scheduled Sync
Running Scheduled Backup
Scheduled Backup Complete
Scheduled Recently Added Sync Complete

EDIT: although now I get a 'syncing episode info' counter on bottom right, going slowly but steadily.

[CyferShepard]

sweet, the log box doesn't populate until the task is complete, wasn't done on purpose, just didn't get around to fixing that minor detail

[dinosmm]

Awesome, sync done and it seems it has picked up everything, except for a library I have for live concerts, which shows all zero. I know the Roku app also doesn't support those, is Jellystat also meant to not support them? Not a big issue anyway.

Weird that the initial auto full sync didn't work, but there could be timeout/throttling issues throwing off your scanner as I have a very rclone-heavy setup. Thank you for this update, I'll watch out for this going to stable.

EDIT: Now the scan is finished, Jellystat is not using any more CPU and the load is right down to what it should be 👌

[CyferShepard]

Hmm, it depends, what type of library is the Live concert? As long its not of type Boxset or type Playlist it should get pulled

[dinosmm]

It's type 'Music Videos' (from the types that appear in the dropdown box when creating a new library).

[CyferShepard]

Okay, il do some research on this and how its reflected in jellyfins API. But i wont post about it here as its of topic for this issue.