in03
commented
2 years ago More research is necessary. We must really try to account for memory. On second thought to the initial post, disk / network speed shouldn't be too much of a problem, since the CPU shouldn't block while waiting for IO? I think? Better find out "Does an IO bottleneck affect CPU usage?".
Problem
Unfortunately we had our main NAS go down recently, so all transcoding needed to take place on a slow USB3 network drive. I noticed that despite efficiently utilising available CPU threads, the proxies were encoding very slowly. Of course encoding to/from the same drive puts increased load on that single drive's bandwidth. So I checked task manager and found memory usage was quite high and disk speed was indeed quite slow.
Potential Solutions
Test resource availability
We could run a quick disk speed check on all available mapped drives both to warn user and better suggest how many workers to run. Hopefully we can do this all in parallel. We could even keep a per-system log of mapped drives we've tested before so we don't test if we know they're fast.
If we can find out how much memory each worker process requires, we can limit it so we keep plenty of RAM margin.
Resource thresholds
We could also add resource usage thresholds in settings for user to change and adjust
rprox workcommand so it always starts workers directly unless called with a specific number.Supervisor scaling
Taking resource thresholds into account, we could kill workers when we go over thresholds. If the workers exit safely, the tasks will quickly be requeued. This would require a supervisor process to monitor resources and act accordingly. This supervisor process could also be used to better monitor encoding metrics and control other aspects of workers.