Open dmccombs opened 2 days ago
Thanks for this report! I will prioritize fixing it.
I have an ongoing PR fixing some serious performance issues around source.drop
, which is the operator used to compute data on the files to get replaygain
information.
I was wondering:
2024/10/22 16:44:04 [source.drop:4] Start dropping source (ratio: 50.00x)
I'll make sure to ping y'all again by them!
Thanks @toots!
Grepping the ~700MB of logs I have from this last run with this issue there's no instances of source.drop
or dropping
.
When I was first dealing with this problem I thought it might be due to replaygain being calculated at playtime for each track, so I ran the loudness tag
command on all my mp3 files to calculate and tag them with the replaygain once, and I've no longer seen any messages about it being calculated in my logs, but that didn't help the problem unfortunately.
Hi @dmccombs ok thanks and, damn!
I just merged the PR in question and pushed a new build with it to the rolling-release-v2.3.x
branch.
I'm gonna try and see if I can reproduce your issue. How long does it take for you to start seeing the CPU increase?
@toots it's varied between days, weeks, or even a couple of months before I started seeing the log errors since I've been dealing with it, but that's across different 2.x versions and I hadn't been looking at CPU metrics directly to see when they started to climb before errors started showing up.
Last night was the first time I've restarted the 2.3.0 build I'm on since having the problem. I'll keep a closer eye on CPU use and let you know when I start seeing it climb again.
Description
Over time releases of Liquidsoap 2.x steadily use more and more CPU until "Too much latency" and catchup warnings start filling the logs, leading to resetting active sources repeatedly. This happens after a couple of weeks of instances running, whereas it didn't happen on Liquidsoap 1.x in largely the same configuration running for years.
Logs are hundreds of MB in size after a month, but some sample log lines are:
Restarting the instances clears the issue for another couple of weeks, and resets CPU usage down to ~10%, before it climbs back to 100% over time.
These are metrics showing the state the system was in after a month of running, then the reset when liquidsoap was restarted:
This has happened both when running 1 instance of Liquidsoap with 3 playlists in it, and also when running 3 separate instances of Liquidsoap with 1 playlist in each.
Here's one of the 3 scripts, the problem happens with or without the
buffer()
around the playlist in the output:Steps to reproduce
Run the above script for weeks and observe CPU usage increasing over time. The playlist used has 372 tracks on it, all with replaygain already computed and tagged in the files.
Expected behavior
Liquidsoap should use a steady amount of CPU over time, and not need restarts to continue to run normally.
Liquidsoap version
Liquidsoap build config
Installation method
From official packages in the release artifacts
Additional Info
Happy to provide any additional info that might help track this down, as it's prevented me from upgrading my production instances to Liquidsoap 2.x for over a year.