Closed ghost closed 2 weeks ago
Yea this is an issue :/
when I am at my desk I will type up some ideas we can do to fix! Thanks for reporting @slavazhil
Currently Pion doesn't support RTX with a distinct SSRC. When we interact with a remote client that uses a distinct SSRC for RTX we drop those packets. These are the goals/things that are important when implementing.
RTX should be returned to the user via Read
and ReadRTP
Most users don't care if a packet is a RTX, they just want to process packets as they arrive
RTX should be distinguishable by advanced users Users who do care should be able to distinguish a packet that was an RTX
We need to support RTX for Simulcast and non-Simulcast
srtp.ReadStreamSRTP
to pull multiple SSRCesCurrently a stream is created for each incoming SSRC. These streams are what Read
and ReadRTP
is called upon. I propose we add an additional method to allow multiple SSRCes to a stream.
AddSSRC
to ReadStreamSRTP
GetSSRC
-> GetSSRCes
on srtp.ReadStreamSRTP
BufferFactory
needs to instead take events instead of constructor arguments. We will fire two AddSSRC
one for the original, one for the SSRC.Update streamsForSSRC
to optionally support opening the RTX Stream as well.
Add a RTCRtpRtxParameters
to RTPCodingParameters
TrackRemote
Read/RTP to detect SSRCTrackRemote.readRtp
will inspect the incoming SSRC. If it is the RTX SSRC it will remove the head off of the buffer to give you the original.
This method will also add isRTX
to the Attributes as needed.
Simulcast SSRC probing needs to also look for repaired-stream-id. We need to support adding the RTX
stream async to the RTP Receiver.
TrackRemote.readRTP
Most users don't care if a packet is a RTX, they just want to process packets as they arrive
In my understanding, the whole point of having a separate RTX track is to distinguish between original and resent packets. If there is no need to make the distinction, then there is no need for the overhead of a separate RTX track — resent packets can be sent over the main media track. In the words of RFC 4588 Section 3:
One approach may be to retransmit the RTP packet with its original
sequence number and send original and retransmission packets in the
same RTP stream. The retransmission packet would then be identical
to the original RTP packet, i.e., the same header (and thus same
sequence number) and the same payload. However, such an approach is
not acceptable because it would corrupt the RTCP statistics. As a
consequence, requirement 1 would not be met. Correct RTCP statistics
require that for every RTP packet within the RTP stream, the sequence
number be increased by one.
Note that retransmitting the packets in the main RTP stream is well supported by current browsers. Note further that it is not too difficult to compensate for the retransmissions and get accurate RTP statistics, although I don't think that browsers do that correctly (Galène does).
@jech @Sean-Der So I was trying to find if there was an advantage for getting RTX stream besides better RTC stats, but I could not find any and as Julius said that is something it can be done without RTX stream. So not sure now if It worth the overhead of getting another stream.
The same mechanism is used for RED. Unilke RTX, which is nice to have but not difficult to work around, RED is what allows you to do FEC with audio, which is the secret sauce behind Zoom's audio quality over lossy WiFi.
So while it might not be that critical to implement ancillary tracks for RTX, the work will be reused when somebody implements RED.
Let me explain some more why a separate track is essential for FEC but not strictly required for retransmissions.
Suppose you are doing retransmissions on the same track. You receive
P1 P2 P4 P5
You NACK P3, so you later receive
P1 P2 P4 P5 P6 P3 P7
In order to maintain proper statistics, you need to determine that P3 is a reemission, which is not too difficult: P3 was NACKed and it arrived out of order, that's a pretty good indication that it was a retransmission. Having a separate RTX track makes this simpler and more reliable, but it's not essential.
Suppose now you're doing FEC on a single track. The sender sends
P1 FEC2 P3
and you receive
P1 P3
I don't see a way for the receiver to determine whether the missed packet was FEC or data. If the receiver is an SFU, for example, if the missed packet was FEC it must renumber P3 before sending it out, while it must not do so if the missed packet was data.
In conclusion, support for ancillary tracks is required:
Sean, I'm therefore enthusiastic about you willing to do the work, just keep in mind that any work you do for RTX must be easy to extend to RED.
@jech Do you find any advantage on using FEC with RED? On my statics gathered, when network gets congested packets gets dropped in big batches (5+ packets), also the complexity of RED since not all browser supports it makes the SFU to terminate it for some users.
I think that for audio inband-fec offers good performance and you can set a fixed packet loss rate to control the % of FEC.
Do you find any advantage on using FEC with RED?
I haven't done any experiments yet.
On my statics gathered, when network gets congested packets gets dropped in big batches (5+ packets)
That happens if your bottleneck router does tail drop. Modern routers use AQM and should not drop packets in bursts. (Or at least a bunch of people much smarter than me are working very hard on getting modern AQMs widely deployed.)
However, the main problem I'm trying to solve is people lecturing behind a poor quality WiFi connection. This happens a lot in my world, since a lot of lecturers have small children, and they need to move to an isolated room in order to give their lectures.
makes the SFU to terminate it
Exactly. Opus FEC is great for end-to-end FEC, while ulpfec is the right thing if you want to terminate FEC at the server. There's no point to negotiate out-of-band FEC if you're not going to reconstruct it at the server, Opus FEC is fine in that case.
Terminating FEC at the server has a number of advantages. Since the signal is reconstructed at the server, it reduces the amount of loss in the case where loss occurs on both legs of the flow. It allows you to use different amounts of FEC for different clients, which reduces both overhead and latency. And, finally, it allows you to use higher FEC rates than what the browsers do, which means that audio can survive on extremely lossy links.
(Opus FEC is limited to 100% FEC. Given that an Opus flow is on the order of 30kbps, it is quite reasonable to use 200% FEC, i.e. to multiply the amount of traffic by 3. With suitable interleaving, this should allow acceptable audio quality on even the worst links.)
wait, I need to clarify pc ERROR: 2021/02/13 07:35:09 Incoming unhandled RTP ssrc(3537335204), OnTrack will not be fired. mid RTP Extensions required for Simulcast
OnTrack will not be fired
here, means OnTrack will not be fired at all, or only not be fired for FEC track, but can be fired for regular video/audio track, I am confused.
I believe that's correct.
Hi all,
maybe we need to remove "video/rtx" tracks from func (m *MediaEngine) RegisterDefaultCodecs()
since rtx doesn't work in anyway ?
Now when we use webrtc.NewPeerConnection to create "default" peerConnection packet re-transmission doesn't work with chrome as chrome offer RTX and pion confirms it supports RTX but it doesn't.
I also don't understand why video/ulpfec declared in func (m *MediaEngine) RegisterDefaultCodecs()
{
RTPCodecCapability: RTPCodecCapability{"video/ulpfec", 90000, 0, "", nil},
PayloadType: 116,
},
Your environment.
What did you do?
A PeerConnection created with
NewPeerConnection
(not api) logs an errorpc ERROR: 2021/02/13 07:35:09 Incoming unhandled RTP ssrc(3537335204), OnTrack will not be fired. mid RTP Extensions required for Simulcast
after the browser sends first retransmitted packet. Looks like pion/webrtc confuses the retransmitted packet for another stream/track.After recreating the code from
NewPeerConnection
manually, but registering only some codecs showed that adding the codec below triggers the error.The rest of the code:
What did you expect?
Expected to receive retransmitted packets from the browser
What happened?
Adding the codec above triggers the error and prevents any retransmitted packet from being received.