Help with avc1 (h264) codec support

[jimmyn]

Hello, I know that this library does not support h264 out of the box but maybe you can point me in the right direction.

I'm trying to implement this example using polyfill, so it works in all the browsers.

I'm passing the decoder configuration like this:

{
      codec: {
        libavjs: {
          codec: 'h264',
        },
      },
      codedHeight: track.video.height,
      codedWidth: track.video.width,
      description: this.description(track),
}

But when I try to decode a sample, I get the following error

[h264 @ 0x1602b0] No start code is found.
[h264 @ 0x1602b0] Error splitting the input into NAL units.

I suspect that the cause of the issue is described here: https://stackoverflow.com/a/55515809/5444431

mkv containers are storing SPS/PPS data aside from frame so the default decoder context construction will always cause NAL search error

I've noticed that the description field from the VideoDecoderConfig is not used anywhere in the video-decoder.ts

I have tried setting it like this:

await libav.AVCodecContext_extradata_s(self._c, config.description);

But it does not seem to be working.

I've spent a lot of time trying to make it work. I don't have much experience working with video decoding, I would really appreciate if you can point me in the right direction.

[Yahweasel]

The only reason why the description isn't used is that none of the codecs that the polyfill supports require it :)

Generally speaking, this sort of description would be a codec parameter passed to libav. The libavjs object does support ctx for AVCodecContextProps, and I believe that the extradata for h264 should correspond to the description here. To do that would be a bit grotesque, but I think just about possible. There's no convenient interface for this because, as said, I haven't needed one for anything the polyfill supports.

Generally speaking, you would want something like this:

...
// configure as usual
...
const ptr = decoder._libav.malloc(description.length);
decoder._libav.HEAPU8.set(description, ptr);
decoder._libav.AVCodecContext_extradata_s(decoder._c, ptr);
decoder._libav.AVCodecContext_extradata_size_s(decoder._c, description.length);
...
// do decoding
...
decoder._libav.free(ptr);

It's unfortunate that this process is this gross, but the reason it's this gross is because of some unfortunate ordering of steps. It should be possible to pass this as part of the libavjs metacodec (in the ctx element), but because this element is an opaque byte array, it needs to be allocated in the libav memory space, and there's no way to get a pointer to that before configuring.

Still, I think this should set you in the right direction. I'll leave this ticket open to see if this progresses.

N.B.: Software h264 decoding in libav.js is going to be very slow. Presumably you already know this :)

[jimmyn]

Thanks a lot for the advice.

I didn't manage to make it work so far. HEAPU8 is not defined in libav. I'm using libav.copyin_u8, I assume it is essentially the same thing.

const ptr = await libav.malloc(config.description.length);
await libav.copyin_u8(ptr, config.description);
await libav.AVCodecContext_extradata_s(self._c, ptr);
await libav.AVCodecContext_extradata_size_s(self._c, config.description.length);
await libav.AVCodecContext_width_s(self._c, config.codedWidth);
await libav.AVCodecContext_height_s(self._c, config.codedHeight);
await libav.AVCodecContext_time_base_s(self._c, 1, 1000);

Here is the code, I've copied your package directly to simplify things. If I can make it work, I'll gladly do a PR to add h264 support.

I've created an example, you can find it here - https://github.com/jimmyn/libavjs-webcodecs/

To run it:

- npm install
- npm start

I'm still getting

libav-3.9.5.1-mediarecorder-transcoder.simd.js:745 [h264 @ 0x160280] No start code is found.
libav-3.9.5.1-mediarecorder-transcoder.simd.js:745 [h264 @ 0x160280] Error splitting the input into NAL units.

Maybe description should be modified somehow? My example does work natively in Chrome if you pass avc1.4d001f as codec

[jimmyn]

@Yahweasel let me know if you have any ideas, thanks

[Yahweasel]

It is rude to "ping" in this way. I am aware of this issue and will address it in time. I have other responsibilities and can't always respond to libavjs-webcodecs-polyfill requests within a couple days. This is not my job.

[Yahweasel]

The tricky thing here is that libav is really designed to take this information in an AVCodecParameters, and AVCodecParameters are really supposed to come from other parts of libav (i.e., libav's own demuxer), so figuring out how to mash the data in is a bit complicated. I'm currently looking into constructing a AVCodecParameters "from scratch", and if that's feasible, then that will probably be the right way to make this work. Unfortunately, ffmpeg's documentation is... lackluster :)

[Yahweasel]

It turns out the problem is that the extradata must be in an AVCodecParameters when the codec is initially opened. Adding it the the context before decoding the first packet isn't sufficient. This (rather hideous) fix got it working:

          const ptr = await libav.malloc(config.description.length);
          await libav.copyin_u8(ptr, config.description);
          const parm = await libav.calloc(1, 1024);
          await libav.AVCodecParameters_extradata_s(parm, ptr);
          await libav.AVCodecParameters_extradata_size_s(parm, config.description.length);
          [self._codec, self._c, self._pkt, self._frame] = await libav.ff_init_decoder(
            supported.codec, parm
          );
          await libav.AVCodecContext_time_base_s(self._c, 1, 1000);

At the very least in libav.js I should add a cleaner way to allocate an AVCodecParameters. In libav, you're never really meant to allocate your own, because you're supposed to be getting that information from libav's own demuxer or have it as a local stack variable, but obviously this is a different situation. Otherwise, this is the correct way to solve this problem. The codec parameters are copied (including reallocation to copy the extradata), so it's safe to libav.free ptr and parm at this point and avoid the memory leak.

extradata is the only part of AVCodecParameters I've ever known to matter for any codec, so I think it would be reasonable to conditionally do this code (or an improved version with less silly allocation of the AVCodecParameters) when description is set.

[jimmyn]

Hi, thank you so much for your help, and sorry for the persistence. The frame is decoded successfully, but now I get the following error:

[swscaler @ 0x160280] No accelerated colorspace conversion found from yuv420p to rgba.

And the createImageBitmap generates a corrupted image.

Looks like I need to scale the frame somehow

I have updated my example - https://github.com/jimmyn/libavjs-webcodecs/

[Yahweasel]

"No accelerated conversion" isn't a real error, and is expected for all conversions on WebCodecs (it has no accelerator). I integrated some changes into libav.js itself to allow allocating the AVCodecParameters more sensibly, so I'll have descriptions in correctly soon. As per what's up with your example, I'll take a look now.

[Yahweasel]

The problem has to do with how the data is (or rather isn't) packed. libavjs-webcodecs-polyfill expected all frame data to be packed; this was a simplification I made early on that's now coming back to bite me (it's not even always correct for the codecs I have). Correcting it should be relatively straightforward. I just need to communicate the width properly while copying out the data.

[Yahweasel]

I can confirm that with the change in 4e419da75, your frames show. I'll have a better (less inefficient) fix shortly.

[jimmyn]

Hi, I'm sorry for the delayed response, was busy with other things. I'm still trying to make it work in all the browsers. I've noticed another issue, VideoDecoder close method does not seem to be working, Even after close it keeps decoding frames and calling output callback.

I've updated my example - https://github.com/jimmyn/libavjs-webcodecs/ Maybe you know what could be the problem? Should I create another issue for this?

[Yahweasel]

Please create another issue, yes.