H264 Video Sources with several NALU in one packet

ard-be commented 1 year ago

Bug After some tests with a TCP video camera, webrtc-streamer was unable to decode the h264 stream while VLC succeeded in. Investigating on this show a problem with ffmpeg decoder : it was unable to parse the input stream. Other tests lead us to cut the input stream and send each NALU individualy to ffmpeg.

To Reproduce

have a camera streaming in TCP (no tests done with UDP)
look at buffer data and try to dump each h264 NALU present
our tests have been made with both the camera and webrtc-streamer behind NAT
unfortunately this is not a public camera

Where file : inc/livevideosource.h function : 'onData()' in the 'if (codec == "H264")' part

How to fix it

parse the whole buffer
extract each NALU one by one (using 'H26x_marker', ie "\0\0\0\1")
send them one by one to ffmpeg ('m_decoder()')

Notes After sending NALU one by one, you can send out of order frames to ffmpeg : it does not complain anymore and the stream is fully decoded (and out of order frames are part of the h264 protocol). We made a working patch for this bug but written in legacy C, not C++. It does not feat the webrtc-streamer standards.

mpromonet commented 1 year ago

Hi @ard-be

You may use -o that will not uncompress the video, forwarding compressed frames.

Best Regards, Michel.

ard-be commented 1 year ago

Hello,

The video stream is resized with a lower bitrate :) It was to share ou findings.

Thank you

mpromonet commented 1 year ago

Hi,

Sure, to resize, uncompressing is needed. Your problem should be solved by commit 1ba1f647fd61d5b7d6515dd84b949ec78434182b

Best Regards, Michel.

ard-be commented 1 year ago

Thank you, We will have a look asap and get back to you.

Best regards

ard-be commented 1 year ago

Hello, Got time to test the patch, it raised an exception :

[021:532][3135945] (livevideosource.h:105): LiveVideoSource:onData id:liveMedia4 size:23197 ts:1671190462301
[021:532][3135945] (livevideosource.h:112): LiveVideoSource:onData nbNalu:3
[021:532][3135945] (livevideosource.h:117): LiveVideoSource:onData SPS
[021:532][3135945] (livevideosource.h:130): LiveVideoSource:onData SPS set format 352x4294967204 fps:25
[021:532][3135945] (livevideosource.h:137): LiveVideoSource:onData PPS
terminate called after throwing an instance of 'std::length_error'
what(): vector::_M_range_insert

Best regards

mpromonet commented 1 year ago

Hi,

SPS seems wrong 352x4294967204 ....

Best Regards, Michel.

mpromonet / webrtc-streamer

H264 Video Sources with several NALU in one packet #542