gt-flexr / FleXR

FleXR: A System Enabling Flexibly Distributed Extended Reality
MIT License
5 stars 1 forks source link

Fix RTP kernels for missing packets #26

Closed jheo4 closed 3 years ago

jheo4 commented 3 years ago

During the streaming experiment, I found some packets are missing with this message max delay reached. need to consume packet. After some research about RTP in libav, it can be related to the rtp settings with sdp. I need to solve this issue for the practical use case.

jheo4 commented 3 years ago

FFmpeg RTP modules have an issue with their functionalities with zero-delay streaming for the real-time use cases. I have tested it with the library-level streaming functions of libavcodec and ffplay which is delivered with ffmpeg distribution. With both cases, FFmpeg showed packet drops and high delay even with all the zero-latency options.

I found that the high latency came from the color space conversion of swscaler of libavcodec. For example YUV2RGB or RGB2YUV. It took ~ 30ms only for the color conversion of 1080p images. I have tested it with OpenCV and it took less than 5 ms.

Even after removing the inefficient implementation of color conversion, there is still packet drop and delay from streaming modules. I tested them with different bitrates (lossy compression) and source frequencies (5~30 fps), but all the cases showed the same issue. I checked the network usage, but it was not very busy.

I found that the issue came from the inefficient protocol implementation of ffmpeg and switched the streaming components to another library. I have tested other available implementations such as ccrtp and uvgrtp, but uvgrtp is the proper replacement based on their experimental results. The mixre rtp components are rewritten and MIXRE streaming micro-benchmark results are quite good.