scottlamb / retina

High-level RTSP multimedia streaming library, in Rust
https://crates.io/crates/retina
Apache License 2.0
244 stars 48 forks source link

Add packets from UDP to jittrbuffer in order to handle unordered packets #107

Open nemosupremo opened 3 weeks ago

nemosupremo commented 3 weeks ago

This is a prototype implementation of using hemisphere-studio/jittr in order to reorder UDP packets.

The simple idea is we hold packets for a configurable time, whenever we detect a packet drop, before they become available to the rest of the application. This is similar to gstreamer's rtpjitterbuffer.

The actual implementation depends on a fork (there is a PR implementation), and inserts a chunk of code into Session's Stream impl. The idea is to provide a high level prototype on how this could work, figure out some testing, and incorporate any preferred style or structure changes.

nemosupremo commented 3 weeks ago

Looking at rtpjitterbuffer a bit more, the 200ms latency is imposed on all packets - so that means having something like this by default may be a breaking change.

This implementation currently doesn't work (or, rather jittr seems to make assumptions that don't work for video)

nemosupremo commented 3 weeks ago

In my testing I'm finding that the jitter buffer is kind of useless without a retransmission mechanism. Ultimately, packets tend to get dropped and rarely come in out of order; and without a retransmission mechanism, there is an introduction of a lot of delay without much benefit.

xnorpx commented 3 weeks ago

In my testing I'm finding that the jitter buffer is kind of useless without a retransmission mechanism. Ultimately, packets tend to get dropped and rarely come in out of order; and without a retransmission mechanism, there is an introduction of a lot of delay without much benefit.

This was my first thought, I have looked at a lot of data for larger UDP RTC systems, reorder rarely happens. It's so rare that it's hardly worth coding for at least for audio. For video and RTX is another story but then you would anyway need a depacketize buffer to handle this.