SimplyCorbett
commented
3 years ago This is true. On AV1 I've seen upwards of 50GB RAM usage when encoding a two minute 4K trailer.
I'm not sure how RAM is impacted with the default settings but this would still be a good option to have.
vid-bin
Describe the problem to be solved
The bottleneck for transcoding concurrency is typically RAM usage, and RAM can vary quite a bit depending on the resolution of the video to be transcoded. My server can really only handle one 4K video at a time, for example, but I may want to allow several lower-resolution videos to transcode concurrently.
In the current system I need to set the concurrency to 1 in order to avoid out-of-memory conditions when multiple 4K videos are in the queue, and all users on that server must wait for the 4K video to finish before any more transcoding work can be done, even if there's room for multiple lower-resolution transcodes to run.
Describe the solution you would like:
The simplest way to deal with this seems like splitting the transcode queue into a "large transcodes" and "small transcodes", using thresholds that can be configured, and giving those different queues their own concurrency levels. I'm not sure it's helpful to split the queues with more granularity.
Describe alternatives you have considered
The goal of concurrency is typically to allow users to experience faster transcoding of shorter videos even when very long videos are being worked on, or to use resources more effectively. There may be a more comprehensive solution for this problem where a server admin could define the amount of RAM and CPU to set aside for transcoding, and then jobs can be packed from a fair queue into the available resources.