Chocobozzz / PeerTube

ActivityPub-federated video streaming platform using P2P directly in your web browser
https://joinpeertube.org/
GNU Affero General Public License v3.0
13.1k stars 1.51k forks source link

Requesting separate transcoding concurrency settings for high resolution videos #4425

Open ghost opened 3 years ago

ghost commented 3 years ago

Describe the problem to be solved

The bottleneck for transcoding concurrency is typically RAM usage, and RAM can vary quite a bit depending on the resolution of the video to be transcoded. My server can really only handle one 4K video at a time, for example, but I may want to allow several lower-resolution videos to transcode concurrently.

In the current system I need to set the concurrency to 1 in order to avoid out-of-memory conditions when multiple 4K videos are in the queue, and all users on that server must wait for the 4K video to finish before any more transcoding work can be done, even if there's room for multiple lower-resolution transcodes to run.

Describe the solution you would like:

The simplest way to deal with this seems like splitting the transcode queue into a "large transcodes" and "small transcodes", using thresholds that can be configured, and giving those different queues their own concurrency levels. I'm not sure it's helpful to split the queues with more granularity.

Describe alternatives you have considered

The goal of concurrency is typically to allow users to experience faster transcoding of shorter videos even when very long videos are being worked on, or to use resources more effectively. There may be a more comprehensive solution for this problem where a server admin could define the amount of RAM and CPU to set aside for transcoding, and then jobs can be packed from a fair queue into the available resources.

SimplyCorbett commented 3 years ago

This is true. On AV1 I've seen upwards of 50GB RAM usage when encoding a two minute 4K trailer.

I'm not sure how RAM is impacted with the default settings but this would still be a good option to have.

vid-bin commented 9 months ago

Any updates?

4k transcoding with hardware encoding removes a lot of the bottlenecks but it would still be nice to have an option to set concurrency by resolution.