Decoder doesn't do anything when fed NAL units manually

[ryangriggs]

I am attempting to feed in the NAL units from a h.264 file encoded with ffmpeg (baseline profile). As a test, I have manually split the h.264 file into NAL units based on the '00 00 00 01' header, into individual elements of an array. Then I feed these to player.decode() one at a time. The onPictureDecoded() callback never fires and the player canvas never changes from black. I have tried two different h.264 files, same result. No error messages or console logs for either one. I have used data which includes the '00 00 00 01' NAL header, and also have removed this header from the data before sending to decode().

I would be very appreciative of any advice to understand why it's not decoding the data, or what I'm doing wrong.

Here's the code (in the document body):

// A manually-generated list of NAL units in original order, base64 encoded:
var nals = ["AAAAAA<snip>", "AAAAA<snip>", <snip about 10MB of NAL units>];

// Set up a player:
var player = new Player({
        useWorker: false,
        webgl: false
    });
document.body.append(player.canvas);
// At this point, a black canvas appears in the browser window.

// Set up onPictureDecoded callback:
player.decoder.onPictureDecoded = () => {
    console.log('onPictureDecoded!');
};

// Dump the NALs into the decode function
nals.forEach((data) => {
    player.decode(atob(data));  // Convert from base64 back to binary, then pass to decode()
});

// Nothing happens...

[yahsaves]

I'm not an expert at nal parsing, but if you look at mp4.js file around line 851 you'll see the start of the "play" function.

This function just feeds nals into a decoder it made earlier called "avc.decode". You'll notice it feeds 2 nals at lines 867 and 868, called sps and pps respectively. Then the function loops to feed in the remaining nals.

You can take the video you want to decode and write some custom code within this play function to save these nals to a file. Then compare the nals generated by mp4.js with the nals you are getting out of ffmepg and see if you can find the error.

My guess is that ffmpeg isn't saving the sps and pps nals the mp4.js decoder creates. I hope this helps get you in the right direction. Let us know what you find!

[ryangriggs]

Found the issue: I had to convert the NAL units to Uint8Array before passing to the decoder. Works now. Thx!

[njw-bupt]

@ryangriggs Hello. Could you please show me what's the NAL units that you put into the function decode() look like? I input the Uint8Array [0, 0, 1, 98, 65, 154, 224, ...] to Player.decode, but nothing happens.

[soliton4]

i think broadway expects the nal seperator 0, 0, 0, 1 it usually works when you fill in the missing 0 unless your nals are slices which will probably not work

[njw-bupt]

@soliton4 It seems I didn't fully understand the H264. I used the nal which is a slice, so I got nothing. Thank you very much!

[soliton4]

yeah slices should be fine as long as you keep them all in one bigger nal its usually 0,0,0,1 for the main nal and then 0,0,1 as seperators within the nal for the different slices. just keep the slices together and it will work in broadway

[njw-bupt]

Hello @soliton4! I have another question. Could you please give me an advice? I noticed that this project does not support weighted prediction for P-frames. If I have a H264 bit stream that contains I-frame and P-frame, it means I cannot use the Broadway to decode the video?

[soliton4]

I frames and p frames are supported