Stitcher improvement ideas - Add H/W acceleration support, Support 4:4:4 subsampling

[mooyoul]

Hello there, I really love this project, Because i tried to build similar personal project about 6 years ago - capture each frame then communicate to javascript runtime to render next frame using phantomjs, and stitched them using ffmpeg. but its performance wasn't optimized, hard to maintain, non-suitable for general purpose. So i'm so glad to see this project which is what i've been looking for - Thank you!

Based on that experience, I'd like to recommend few things:

• [ ] 1. H/W Accel - FFmpeg supports hardware accelerated encoding which is blazing fast. Most devices/platforms already satisfies requirements (e.g. Intel QuickSync, Apple MediaToolbox, NVIDIA NVenc). And we can fallback to x264 encoder if there's no hardware support. It also gives better HEVC or 4K encoding performance. See https://trac.ffmpeg.org/wiki/HWAccelIntro for details.
• [x] 2. 4:4:4 Chroma Subsampling - 4:2:0 chroma subsampling usually gives poor text quality due to its behavior. So it would be nice if there's 4:4:4 chroma subsampling option. See https://github.com/leandromoreira/digital_video_introduction#chroma-subsampling and http://web.archive.org/web/20190324110017/http://images.guide/#chroma-subsampling for details.
• [x] 3. More output codecs - HEVC gives better compression ratio (but requires more computing power)

Will submit a PR for sure, but i want to hear your thoughts!

[JonnyBurger]

Thanks a lot, these are some great tips!

For 1.: Can we just add the -hwaccel to benefit from the speed gains?

For 2. I have very little knowledge, is the 4:2:0 determined by the -yuv420p flag? Will definitely accept if you suggest an optimization!

For 3. Great idea, this would make a lot of sense to specify codec and container via flag! When I get time (might take a bit, but I'll start working on new features from today), I'll add a flag!

[mooyoul]

For 1.: Can we just add the -hwaccel to benefit from the speed gains? Unfortunately No. -hwaccel flag is only applies to decoding. Hardware accelerated encoding requires explicit encoder selection (e.g. -v:c h264_videotoolbox for Apple T2 Chip, -v:c h264_nvenc for NVIDIA CUDA). So application should guess applicable encoder.

For 2. I have very little knowledge, is the 4:2:0 determined by the -yuv420p flag? Will definitely accept if you suggest an optimization!

Yes. yuv420p means YUV 4:2:0 pixel format. so you can change YUV 4:4:4 by supplying yuv444p. But please note that YUV 4:4:4 pixel format is unsupported in some encoders (or devices. For example, Older NVIDIA Graphics Card does not support yuv444p h/w accel encoding)

For 3. Great idea, this would make a lot of sense to specify codec and container via flag! When I get time (might take a bit, but I'll start working on new features from today), I'll add a flag!

Also i will take a look this and submit a PR. Please add labels ;)

[timhaywood]

Personally I think the option to either:

• Specify a custom configuration to FFMPEG, or
• skip the encoding step and only output the PNG sequence

Would be better than adding another single purpose option.

I think 4:2:0 is a sensible trade-off in file size for most people, and those who need a higher quality output or custom settings could pass their own options to FFMPEG (including specifying a 444 pixel format), or encode the file themselves from the PNG sequence.

Otherwise it seems like you would end up adding more and more config to the CLI to allow for different encoding options in order to support everyone's preferences.

That way most people would go with the default sensible h264 settings, but others could output a 444 ProRes, or just the PNG's etc if it's more of an intermediate or they need a higher quality file.

Just my 2 cents, hope it's helpful rather than making things more complicated!

[JonnyBurger]

Thanks for weighing in @timhaywood!

+1 for allowing to customize the format instead of hard-coding it. Many more important issues came in but I will tackle it in the next days hopefully.

PNG output is supported btw, using the --png flag! 🥳

[JonnyBurger]

Now you can change the pixel format if you like using configuration file or CLI flag. Will ship in next version or you can try 1.3.0-alpha.88363356 on NPM.

https://www.remotion.dev/docs/config#setpixelformat

Would you mind giving me some guidance on the algorithm for guessing hardware acceleration support? Adding more output codecs is also planned!

[JonnyBurger]

Just merged support for HEVC (and VP8/VP9 as well)! Will make a release today for v1.4.

Regarding hardware acceleration, I have played a bit around with it and there's a wide variety of methods and a huge amount of differences between platforms. If enabled, other features such as CRF don't work and hardware encoding apparently gives worse performance than CPU encoding. Detection of when hardware acceleration is available is also very tricky. And even though H264 hardware acceleration says it's available, I can't even get it to work on my Mac. So it seems like a headache and after some consideration, I will not implement it directly in Remotion (maybe allow to customize FFMPEG flags in general), at least not for now.