No output or error decoding streamed h264 video

[Metalit]

I have an Android application continuously sending h264 encoded frames (from the MediaCodec API) over a websocket. When I attempt to use WebCodecs to play the stream in the browser, I never get any output or any errors from the decoder.

Details:

• Every packet begins with 0 0 0 1, which is correct according to my understanding
• If I write the stream (including the SPS) to a file, it can be viewed
• JMuxer is also able to display the stream, but with horrible performance
• The stream sends the SPS as the first packet and never again. If I pass it to the decoder, I get the error Failed to execute 'decode' on 'VideoDecoder': A key frame is required after configure() or flush(). If you're using AVC formatted H.264 you must fill out the description field in the VideoDecoderConfig., however nothing other than that error changes either way
• decodeQueueSize never increases
• ondequeue appears to be triggered after every decode call
• If it matters, the code is running in the system webview with Tauri and SolidJS, on Windows 11

Simplified, my implementation is:

const config: VideoDecoderConfig = {
  codec: "avc1.42000a",
  codedWidth: 1080,
  codedHeight: 720,
  optimizeForLatency: true,
};

const init: VideoDecoderInit = {
  // never called
  output: (frame) => {
    canvasContext.drawImage(frame, 0, 0);
    frame.close();
  },
  error: (error) => {
    console.error("decoder error", error);
  },
};

// VideoDecoder.isConfigSupported(config) = true
const decoder = new VideoDecoder(init);
decoder.configure(config);

onWebsocketMessage = (data: number[]) => {
  const chunk = new EncodedVideoChunk({
    data: new Uint8Array(data),
    timestamp: 0,
    type: data[4] === 101 ? "key" : "delta",
  });
  const isSps = data[4] === 103;
  if (!isSps) decoder.decode(chunk);
}

Is this the right place to get help with this? If these issues are specifically for the specification and nothing else, I still think that it could be improved with some way of getting a more detailed current state, validity of input, etc.

[Djuffin]

It looks like an implementation bug, not a spec issue.

Please open a bug for a browser. Having a repro page with actual data would really help. Chromium: https://crbug.com/new Firefox: https://support.mozilla.org/en-US/kb/file-bug-report-or-feature-request-mozilla

[Djuffin]

That said.

Since you don't populate VideoDecoderConfig.description you need to make sure that you chunks have valid annexb NALs (start codes, parameter sets and everything)

https://www.w3.org/TR/webcodecs-avc-codec-registration/#videodecoderconfig-description

Here, as far as I case see, you just skip parameter sets and the decoder never has a chance to process them.

  const isSps = data[4] === 103;

[Metalit]

Thanks. I managed to get it working when configured to use software with a small change (appending the sps data to the start of the first frame packet instead of passing it separately, as I had tried) but the same stream doesn't work if it prefers hardware, so I'll see if I can make a simple example and report that to the implementation.