search

mbebenita / Broadway

A JavaScript H.264 decoder.
Other
2.73k stars 426 forks source link

Why isn't nalu slice code redundant? #227

Open SoylentGraham opened 3 years ago

SoylentGraham commented 3 years ago

Broadway does NALU splitting in the C code itself for cases when the data pushed in isn't split by NALU, (hence the data-read feedback)

Is there some benefit I don't see (lower memory use by filling only the small static scratch memory in C?) to having NALU splitting (slices) in decoder.js?

It inhibits correct meta being passed around. I have fixed this in my fork so input meta is synchronised with output meta, although it's a little more tricky in slice code, but it seems to me that the slice code is redundant anyway :)

soliton4 commented 3 years ago

without testing it out myself, but i believe the c code assumes its a complete nalu unless there is a 00 00 00 01 separator sequence. so if you feed it an incomplete nalu it will decode incomplete and as a follow up error the next feed will break

if you feed it all nalus of a video file at once it will overload. so you are probably partially right in, that you could feed more than one nalu at a time, but for practical reasons it was better to split it in the demo code, as well as in my streaming demo

SoylentGraham commented 3 years ago

Oh, maybe I overlooked that this code buffers up and waits for a FULL nalu (or rather, it would have to wait for the next one)

The broadway code splits 001 as well as 0001 (and you can feed it more than one, that's why it returns "I read X bytes" :)

Broadway is quite resillient in some respects though, if you send half an intra frame, it'll decode half a intra frame :) (I've been using broadway for a few years on various platforms, so found a few quirks & work arounds :)

soliton4 commented 3 years ago

00 00 01 is most likely separator between slices. it makes sense to decode them all at once

there is no way i know of to wait for a full nalu aside waiting for the next nal. i dont think the c decoder does that. so if you feed incomplete nalus you will have a degraded stream

SoylentGraham commented 3 years ago

Ah so you are just duplicating the work C/Wasm side, you're not buffering up waiting for "complete" slices (as it's not really possible unless you feed a fake EOF nalu - I do this in my h264 implementations)

I just wanted to confirm I wasn't missing anything. :)

I've removed the redundancy in my fork, but I am splitting NALU's before feeding to broadway, as I have meta synchronised to "frames" (Nalu's, but I know Nalu isn't exclusively as single frame) correctly

https://github.com/SoylentGraham/Broadway

soliton4 commented 3 years ago

it depends on your application. in one of my demos / apps i get the stdout from ffmpeg, which doesnt tell me the length of each nalu, in this case i have to wait for the next nalu before feeding one to broadway.

if you have an encoder or a stream where you know each nalu, you can just feed it complete nalus. no need to add extra separators there. the nalu splitter code in broadway is optional. if you know your nalu you can just feed it as long as its complete

SoylentGraham commented 3 years ago

no need to add extra separators there.

I'm not adding seperators, I'm pushing one EOS NALU (there's a nalu content type for end of stream) to my stream, when I know I have EOF, so I don't end up not-sending the final nalu.

soliton4 commented 2 years ago

is this issue resolved?