H264 SPS / PPS are not received correctly if pushed as separate packets

[jrsnen]

Description in #174:

Yeah, so for some reason I am not getting 2 successive RTP H264 type [1..23] (single) "frames" (if you count sps / pps as a frame) from uvgRTP as an rx. If I embed the SPS & PPS into an aggregation unit (e.g. using a gstreamer tx pipeline) I don't have this issue.

[jrsnen]

So the SPS is delivered, but the PPS is not.

The most interesting pieces of code are here and here. My first guess is that the PPS is somehow interpreted as the wrong type of NAL unit due to some bug in determining the NAL type.

[Consti10]

I guess I just gotta do these tests again with DCMAKE_BUILD_TYPE=Debug and see if I get something in the log ?

[Consti10]

not sure if the issue is the fact there is an SPS / PPS inside, could also be 2 successive non-aggregate packets containing anything. But I noticed the issue with the PPS only so far.

Btw, I don't think it'l help but this is the dirty rtp parser we wanna replace - both single and aggregate packets are quite easy to do.

[jrsnen]

@Consti10 Did you say you were using uvgRTP also for sending? I added support for NAL units with size 8 to start code lookup. Sizes smaller than 8 are not yet supported, but would probably be trivial to add.

I was unable to find any issues with specific NAL types in H264.

Does d7f0af7c4c3bd1b0d52063c7c02fee02b736e14a fix your issue?

[jrsnen]

Now also sizes smaller than 8 should work in 5ca3c1399165620b2ab90bf64a874f5972dfc522.

[Consti10]

@Consti10 Did you say you were using uvgRTP also for sending? I added support for NAL units with size 8 to start code lookup. Sizes smaller than 8 are not yet supported, but would probably be trivial to add.

I was unable to find any issues with specific NAL types in H264.

Does d7f0af7 fix your issue?

That sounds like it might be the solution - the PPS NALUs were of size 9 (I think that was without a long NALU start code, so could be called size 8 as well).

I'l check it out. Yes, I was using uvgRTP for sending, too (at first) but then switched to gstreamer as tx to eliminate tx as possible error cause. I'l try again, will take some time to setup though.

[Consti10]

Hm, this seems to have made it worse, I am now only getting packets of size 6. I've changed 2 lines in examples/receiving_hoock.cc:

//std::cout << "Received RTP frame" << std::endl;
std::cout<<"Received RTP frame payload:"<<frame->payload_len<<" padding:"<<frame->padding_len<<"\n";

and changed udp input port to 5600

with the following gstreamer pipeline as tx:

gst-launch-1.0 videotestsrc ! video/x-raw, format=I420,width=640,height=480,framerate=30/1 ! x264enc name=swencoder bitrate=5000 speed-preset=ultrafast tune=zerolatency key-int-max=15 sliced-threads=0 ! queue ! h264parse config-interval=-1 ! rtph264pay mtu=1024 ! udpsink host=127.0.0.1 port=5600

I am exclusively getting: Received RTP frame payload:6 padding:0 and a lot of messages in the form of: [uvgRTP][DEBUG][::packet_handler] Received an RTP packet belonging to a completed frame! Timestamp: 553220523, seq: 40818

which cannot be correct (e.g. the above pipeline definitely creates NALUs bigger than 6 bytes)

[jrsnen]

The previous fixes were only for the sending side.

Based on that debug print, I think I know what is going on in the receiving side: There is a discarding of duplicated packets in uvgRTP based on timestamps, but I would guess GStreamer puts all those different NAL units in non-aggregate packets with same timestamps, leading uvgRTP to discard the later NAL units as it thinks they have been duplicated.

Does c07a8c2b577269d908501c980b8bfbaafa10e67b fix receiving video from GStreamer?

I also created a new issue for duplicate discarding, but this is not a high priority: #179

[Consti10]

Yeah, that makes sense. Just tested gstreamer sending and uvgRTP receive (h264), and the issue(s) seem to be gone. I think we can close this issue, thanks for the help !

[jrsnen]

@Consti10 Great! Thanks for bringing up this issue! I've been trying to fix all issues related to basic RTP/SRTP streaming for some time now.

BR, Joni