Can mp4box feed containerless h264 to the WebCodecs API?

[FrostKiwi]

Question

The official WebCodecs API sample uses mp4box.js eg. here: https://github.com/w3c/webcodecs/blob/261401a02ff2fd7e1d3351e3257fe0ef96848fde/samples/video-decode-display/demuxer_mp4.js#L44 to feed EncodedVideoChunks of a MP4 file into the VideoDecoder.decode() function in this line: https://github.com/w3c/webcodecs/blob/261401a02ff2fd7e1d3351e3257fe0ef96848fde/samples/video-decode-display/worker.js#L87

Can mp4box.js provide these EnvodedVideoChunks for an h264, which has not been muxed into a .mp4 container as well?

Context:

I want to play back a raw .h264 stream using the Webcodecs API and extract specific frames. It is saved as a .h264 file, essentially just the video data before being muxed into a .mp4 container format.

It should be possible to play this back using the new WebCodecs API. I fail at providing the avc1.64001e VideoDecoder with the properly formatted EncodedVideoChunk. How can I do so? I know the resolution, framerate and frame count, but how do I take this information and create an EncodedVideoChunk from that?

I have a potentially gigabytes big local file, which I receive via:

        const readableStream = this.data.file.stream();
        const reader = readableStream.getReader();

How to feed this into VideoDecoder: decode()? Can mp4box.js be instructed to provide chunks to do so? So how do I overcome this step, besides muxing and feeding it through mp4box.js, only to have it demuxed again?

Same question as in https://stackoverflow.com/questions/77013398/how-to-playback-containerless-raw-h264-streams-via-webcodecs-api.

[jozefchutka]

Following is a raw copy-paste of code (mix of js/ts) around my app which should help you feed VideoDecoder:

// trackId to desscription
// https://github.com/josephrocca/getVideoFrames.js/blob/main/mod.js#L101
// https://github.com/gpac/mp4box.js/issues/243
// https://github.com/vjeux/mp4-h264-re-encode/blob/main/mp4box.html#L50
// https://w3c.github.io/webcodecs/samples/video-decode-display/demuxer_mp4.js
function getDescription(id:number):Uint8Array | undefined {
        const trak = this.mp4.getTrackById(id);
        const entries = trak?.mdia?.minf?.stbl?.stsd?.entries;
        if(entries)
            for(const {avcC, hvcC} of entries) {
                const box = avcC || hvcC;
                if(box) {
                    const stream = new DataStream(undefined, 0, DataStream.BIG_ENDIAN);
                    box.write(stream);
                    return new Uint8Array(stream.buffer, 8);
                }
            }
        return;
    }

const decoder = new VideoDecoder({
    output: (a,b,c) => ...,
    error: error => ...
});

const codec = "avc1.64002a";
const description = getDescription(someTrackId);
// example description:
// const description = new Uint8Array([1, 100, 0, 42, 255, 225, 0, 27, 103, 100, 0, 42, 172, 217, 64, 120, 2, 39, 229, 192, 68, 0, 0, 3, 0, 4, 0, 0, 3, 1, 224, 60, 96, 198, 88, 1, 0, 6, 104, 235, 227, 203, 34, 192, 253, 248, 248, 0]);
const codedWidth = 1920;
const codedHeight = 1080;
decoder.configure({codec, description, codedWidth, codedHeight,
    avc:{format:"annexb"}});

decoder.decode(new EncodedVideoChunk({data:frame1, type:"key", timestamp:4200000, duration:16666.666666666668}));

At some point I managed to make it work https://github.com/gpac/mp4box.js/issues/243#issuecomment-1067617184 but I do not have single file POC