I have a 64-core server available; ffmpeg has multi-threading but it never seems to use more than a few at once, so to use all the processing power we would need multiple ffmpeg instances. I did a hacky thing of setting encode order to random, commenting out the lock file, and then running multiple processmedia2 docker instances, but that seemed messy as they would occasionally overwrite each other's work.
It would be nice if processmedia2 itself could encode multiple tracks in parallel. I tried updating encode_media.pydef encode_media() to use a ThreadPool with map(), but somehow that broke in a lot of different ways and I don't even understand what happened...
shish
commented
1 year ago
I have a 64-core server available; ffmpeg has multi-threading but it never seems to use more than a few at once, so to use all the processing power we would need multiple ffmpeg instances. I did a hacky thing of setting encode order to random, commenting out the lock file, and then running multiple processmedia2 docker instances, but that seemed messy as they would occasionally overwrite each other's work.
It would be nice if processmedia2 itself could encode multiple tracks in parallel. I tried updating
encode_media.pydef encode_media()to use aThreadPoolwithmap(), but somehow that broke in a lot of different ways and I don't even understand what happened...