search

mediacms-io / mediacms

MediaCMS is a modern, fully featured open source video and media CMS, written in Python/Django and React, featuring a REST API.
https://mediacms.io
GNU Affero General Public License v3.0
2.52k stars 459 forks source link

Prioritization of encodings #946

Open KyleMaas opened 6 months ago

KyleMaas commented 6 months ago

Describe the feature you'd like When uploading a lot of massive files at once, they seem to be processed in order. So once the first video uploads and it is chunked, Celery is then saturated with tasks. When the second video uploads, it now has to wait until all of the encodings of the first one finish processing before the preview and low-res encodings of the second video are processed. It would be nice if you could set encoding priorities so you could have all new videos processed for preview and for 240P MP4 before proceeding on to the higher resolution encodings.

Describe alternatives you've considered N/A

Additional context N/A

KyleMaas commented 5 months ago

https://github.com/celery/celery/issues/4028

KyleMaas commented 5 months ago

https://github.com/mediacms-io/mediacms/blob/c5047d8df8686d75100e5099489be4fd1bf5f733/files/tasks.py#L125-L128

KyleMaas commented 5 months ago

https://github.com/mediacms-io/mediacms/blob/c5047d8df8686d75100e5099489be4fd1bf5f733/files/models.py#L596

https://github.com/mediacms-io/mediacms/blob/c5047d8df8686d75100e5099489be4fd1bf5f733/files/models.py#L609-L612

KyleMaas commented 5 months ago

So it looks like if the encoding is created by a chunkize_media task, priority on anything in settings.MINIMUM_RESOLUTIONS_TO_ENCODE is made to run with the same highest priority.

Otherwise if it's created by encode and if it needs to be chunkized and is encoding a GIF, then the priority is set to the highest priority (0). Or for anything other than GIFs, the priority is set backwards from in tasks.py - the more important resolutions are set to the lowest priority (9).

So it's correct or backward depending on which code path it takes. But it is at least partially implemented. It just needs to be cleaned up so stuff happens in the right order.

KyleMaas commented 5 months ago

Oh, and the 0==highest and 9==lowest priority thing seems to be due to using the Redis backend for Celery. Otherwise it's the other way around, and the documentation flip flops a bit on this. But that seems to be the way it is currently.