Use WebCodecs for a faster encoding!

[dalecurtis]

The README says to create an issue for suggestions on how to make it faster :)

https://web.dev/webcodecs/ offers a way to directly use the hardware encoders for the various codecs that Chrome supports. It's in origin trial right now, so feedback is appreciated!

https://github.com/WICG/web-codecs https://bugs.chromium.org/p/chromium/issues/entry?components=Blink%3eMedia%3eWebCodecs

[mattdesl]

Thanks Dale – I'd be really interested in using WebCodecs (I have a webcodecs branch here testing vp9) but I felt blocked by two problems:

• I haven't found a simple way of encoding H264 frames, it seems to be mostly focused on codecs like vp9?
• WebCodecs doesn't help with building an MP4 container – I might be able to use my WASM build for this, but how would I get NAL data from WebCodecs API?

[dalecurtis]

• H264 encoding should be working on stable, though I recommend developing against canary/dev channel since the API is in flux.
• Here's a web test which shows how H264 encoding can be done. A key aspect with H264 is the second parameter on the output callback which delivers you the decoder configuration. For H264 the description field contains the AVCC payload data that's needed for an MP4.
• Since you're bundling ffmpeg, muxing shouldn't be much of an issue using their internal muxing API. The outputs are already in AVCC format, so you just need to provide the AVCC extradata when configuring the muxer.
• We do want to allow users to choose AnnexB or AVCC output. This bug tracks WebCodecs providing an option to choose AnnexB or AVCC. With AnnexB, ffmpeg's normal raw h264 demuxer will accept the bitstream as is. So simply ffmpeg -i .h264 -vcodec copy output.mp4 will mux without any custom code.

[mattdesl]

Thanks! I'm actually not using ffmpeg here, but small open source C libraries, minimp4 (for turning h264 frames into mp4 container) and minih264 (for encoding YUV to h264).

I don't know much about AnnexB vs AVCC, but to use the minimp4 library I have to send it chunks of NAL data to mux into a mp4. I guess that NAL data is just the same as the AnnexB data?

[dalecurtis]

The second parameter of the output callback is a VideoDecoderConfig object. The description field on that object should have the NAL chunks you need to hand to minimp4. The payload is already in AVC format so it should be ready to insert into an MP4.

[mattdesl]

Ok, got it all working after a bit of a struggle. Had to convert the frames from AVC to AnnexB on the fly, for minimp4 to accept them. Presumably this will be easier once AnnexB lands in WebCodecs.

Now with WebCodecs instead of minih264 encoder, it's really fast (encoding ~300 frames in 5 seconds, instead of 300 frames in ~14 seconds).

• Using different baseline/level codecs doesn't seem to have much difference in filesize or quality, I'm trying the highest quality with avc1.640034 (High Level 5.2)
• The bitrate configuration dosen't seem to have any effect either, I'm trying really high values (24_000_000) and the filesize and quality remains the same
• I can't increase framerate to 60, the video becomes half-resolution for some reason
• Basically no matter what it's always producing lower quality videos (noticeably fuzzy) compared to my raw Emscripten encoder+muxer approach

So other than the quality issues, it seems to be working pretty well! You can see my code here:

https://github.com/mattdesl/mp4-h264/blob/cfd8e1ca99c88f2af9d6ca07d0531324587618cb/test/webcodecs.html

[dalecurtis]

Awesome, glad it's working! Sorry it's a bit complicated to get going.

• We're working on making it easier to get AnnexB right now: https://chromium-review.googlesource.com/c/chromium/src/+/2587596 hopefully that'll be available on canary later this week.
• The encoders are tuned for RTC so it may not always yield better quality at the moment. Did you try setting the "acceleration" prefer to require or deny to see if there were any differences? https://source.chromium.org/chromium/chromium/src/+/master:third_party/blink/renderer/modules/webcodecs/video_encoder_config.idl;l=16

What platform are you testing on? We'll want to track your issues in the Chromium tracker so they don't get forgotten. @chcunningham

[mattdesl]

Alright thanks! I think this might have actually just been my own code not setting the size correctly, so it was sending lower quality frames, although I'm still noticing a few issues here and there and overall the image appears more faded/muted in color. I'll try to log them in the tracker as I see them.

[dalecurtis]

Probably color space information isn't correctly passed to the encoder somewhere (could be on our side). If you have a repro case with a comparison shot we can dig further.