search

warren-bank / HLS-Proxy

Node.js server to proxy HLS video streams
http://webcast-reloaded.surge.sh/proxy.html
GNU General Public License v2.0
256 stars 80 forks source link

Cpu usage 100% issue #57

Open sampatsharma143 opened 3 weeks ago

sampatsharma143 commented 3 weeks ago

image Current i am running HLS-Proxy on a production app , there are 300-200 concurrent users and i have a vps with 90gb ram and 32 threads. it runs okay for some time and than after few hours( 3-4 ) cpu usages reaches 100% and it makes the video player unresponsive. can you tell me how can i identify the core issue and fix it. it will be very helpful. thank you

warren-bank commented 3 weeks ago

Off-hand, I can't imagine what would cause this. I've never run into any CPU or RAM issues, but I've also never used the proxy in a production environment. I'm going to cross-reference this issue with another, which also has similar observations.

Either myself, or someone else, will need to run some kind of software to profile usage.. that can identify exactly which block of code within hlsd.js is responsible for the excessive usage.

My first question would be: what command-line options do you use when you start the proxy?

update: I've added some scripts to the repo that simplify the task of running the Node.js built-in profiler, applying some load on the proxy server, and then processing the raw log file to produce a log file that provided good insight. Unfortunately, the trivial load applied by this script doesn't produce any performance issues; nearly all of the CPU cycles are spent performing regex operations.. as would be expected. But.. it's a starting point…

sampatsharma143 commented 3 weeks ago

This is the pm2 command that i use to start the proxy - pm2 start hls-proxy/bin/hlsd.js -i max --name hls-proxy -- --host "domain:443" --req-insecure --port "5001" -v 3

sampatsharma143 commented 3 weeks ago

I still not able to find the real cause of this so i made a workaround for some time. i made a bash script to check the cpu usages every minute and if it's >=90% it will restart the pm2 process

Bash Script `#!/bin/bash

Load NVM directly

export NVM_DIR="/root/.nvm" [ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh" # Load nvm [ -s "$NVM_DIR/bash_completion" ] && . "$NVM_DIR/bash_completion" # Load nvm bash_completion (optional)

Use the specific Node.js version

nvm use 20.18.0

IDLE=$(top -bn1 | grep "Cpu(s)" | sed "s/., ([0-9.])% id.*/\1/" | awk '{print $1}')

Check if idle is 10% or less

if (( $(echo "$IDLE <= 10" | bc -l) )); then echo "$(date -u) Overall CPU idle is at or below 10%. Restarting PM2 process hls-proxy" pm2 restart hls-proxy else echo "$(date -u) Overall CPU usage is normal." fi`