search

sipsorcery-org / sipsorcery

A WebRTC, SIP and VoIP library for C# and .NET. Designed for real-time communications apps.
https://sipsorcery-org.github.io/sipsorcery
Other
1.44k stars 438 forks source link

Cannot detect when a track is added by the remote peer #1135

Open evan-hilscher opened 3 months ago

evan-hilscher commented 3 months ago

Similar to #983.

I am attempting to add a media track some time after the peer connection has been established (this code is executed by the local peer):

var localVideoTrack = new MediaStreamTrack(AudioVideoWellKnown.WellKnownVideoFormats[SDPWellKnownMediaFormatsEnum.JPEG], MediaStreamStatusEnum.SendOnly);
_peerConnection.AddTrack(localVideoTrack);

Renegotiation occurs as expected, but at no point is the remote peer informed that the local peer has added a track. I have subscribed to and am logging every RTCPeerConnection event (including OnRtpPacketReceivedByIndex as mentioned in #983), but none of them are raised on the remote peer.

Attempting to send any data over the track results in this logged warning, despite creating the track as SendOnly:

20:08:40 warn: sipsorcery[0] SendRtpRaw was called for an video packet on an RTP session with a Stream Status set to Inactive
ChristopheI commented 1 month ago

Both peers are using SIPSorcery ? Are you sure SDP are well exchanged between peers ? Did you check SDP offers and answers content ?

evan-hilscher commented 1 month ago

Both peers are using SIPSorcery. I have verified that the offer SDP contains a new video track.

The issue appears to be here: https://github.com/sipsorcery-org/sipsorcery/blob/06f8e3c922ae0a5ad82cf939dd8170aa7f0d3691/src/net/RTP/RTPSession.cs#L1015-L1021

When the remote peer sends a media track that the local peer does not know about, it creates a new MediaStream and sets the LocalTrack of that stream to Inactive. I am not a WebRTC expert, but I think that perhaps an ontrack event should be raised somewhere near the referenced code, and that the local track should not be created as inactive.

ChristopheI commented 1 month ago

Could you provide all SDP exchanged from both sides ? (and from the beginning including the creation of the PeerConnection)

PeerA is the one adding the video track: So the SDP (as offer) sent must include two media (the one used to create the PeerConnection) and the new video with SendRecv or SendOnly

Once this SDP is received on PeerB, it must add a Video MediaTrack with RecvOnly or SendRecv according your needs and create an SDP (as anwser) and send it to PeerA

evan-hilscher commented 3 weeks ago

Here is the sequence of SDP exchanges:

Local peer receiving offer:

v=0
o=- 43754 0 IN IP4 127.0.0.1
s=sipsorcery
t=0 0
a=group:BUNDLE 0
m=application 9 UDP/DTLS/SCTP webrtc-datachannel
c=IN IP4 0.0.0.0
a=ice-ufrag:XSOI
a=ice-pwd:ETVICNGSOOBXHCHIXVYUUGDH
a=fingerprint:sha-256 A2:2C:21:02:5C:9E:A7:C7:47:E1:C9:ED:13:C5:6F:6E:96:71:21:C1:69:DB:9C:69:60:68:20:4A:4E:92:F2:BA
a=setup:actpass
a=candidate:3913191201 1 udp 8447 157.245.114.91 58553 typ relay raddr 0.0.0.0 rport 0 generation 0
a=candidate:2564284589 1 udp 1677730047 75.100.125.142 45834 typ srflx raddr 0.0.0.0 rport 0 generation 0
a=candidate:2422434839 1 udp 2113937663 10.0.3.122 63054 typ host generation 0
a=ice-options:ice2,trickle
a=mid:0
a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=end-of-candidates
a=sctp-port:5000
a=max-message-size:262144

Remote peer receiving answer:

v=0
o=- 32058 0 IN IP4 127.0.0.1
s=sipsorcery
t=0 0
a=group:BUNDLE 0
m=application 9 UDP/DTLS/SCTP webrtc-datachannel
c=IN IP4 0.0.0.0
a=ice-ufrag:JRJJ
a=ice-pwd:DNVTQRUDPANEWTNZBVOJFZOC
a=fingerprint:sha-256 5D:1D:39:7C:FE:29:91:4E:C8:6C:CF:5B:55:D1:D2:BC:38:5E:3C:AF:79:41:4D:82:32:F0:6E:A4:E7:A1:DD:C6
a=setup:active
a=candidate:3913191201 1 udp 8447 157.245.114.91 59754 typ relay raddr 0.0.0.0 rport 0 generation 0
a=candidate:2564284589 1 udp 1677730047 75.100.125.142 52771 typ srflx raddr 0.0.0.0 rport 0 generation 0
a=candidate:2422434839 1 udp 2113937663 10.0.3.122 61986 typ host generation 0
a=ice-options:ice2,trickle
a=mid:0
a=end-of-candidates
a=sctp-port:5000
a=max-message-size:262144

Local peer receiving offer, with new video track:

v=0
o=- 43754 0 IN IP4 127.0.0.1
s=sipsorcery
t=0 0
a=group:BUNDLE 0 1
m=video 9 UDP/TLS/RTP/SAVP 26
c=IN IP4 0.0.0.0
a=ice-ufrag:XSOI
a=ice-pwd:ETVICNGSOOBXHCHIXVYUUGDH
a=fingerprint:sha-256 A2:2C:21:02:5C:9E:A7:C7:47:E1:C9:ED:13:C5:6F:6E:96:71:21:C1:69:DB:9C:69:60:68:20:4A:4E:92:F2:BA
a=setup:passive
a=candidate:3913191201 1 udp 8447 157.245.114.91 58553 typ relay raddr 0.0.0.0 rport 0 generation 0
a=candidate:2564284589 1 udp 1677730047 75.100.125.142 45834 typ srflx raddr 0.0.0.0 rport 0 generation 0
a=candidate:2422434839 1 udp 2113937663 10.0.3.122 63054 typ host generation 0
a=ice-options:ice2,trickle
a=mid:0
a=rtpmap:26 JPEG/90000
a=rtcp-fb:26 goog-remb
a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=rtcp-mux
a=rtcp:9 IN IP4 0.0.0.0
a=end-of-candidates
a=sendonly
a=ssrc:1297181212 cname:d0860e88-a474-405d-a44f-e54a39390539
m=application 9 UDP/DTLS/SCTP webrtc-datachannel
c=IN IP4 0.0.0.0
a=ice-ufrag:XSOI
a=ice-pwd:ETVICNGSOOBXHCHIXVYUUGDH
a=fingerprint:sha-256 A2:2C:21:02:5C:9E:A7:C7:47:E1:C9:ED:13:C5:6F:6E:96:71:21:C1:69:DB:9C:69:60:68:20:4A:4E:92:F2:BA
a=setup:passive
a=ice-options:ice2,trickle
a=mid:1
a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=sctp-port:5000
a=max-message-size:262144

Remote peer receiving answer:

v=0
o=- 32058 0 IN IP4 127.0.0.1
s=sipsorcery
t=0 0
a=group:BUNDLE 0 1
m=video 9 UDP/TLS/RTP/SAVP 26
c=IN IP4 0.0.0.0
a=ice-ufrag:JRJJ
a=ice-pwd:DNVTQRUDPANEWTNZBVOJFZOC
a=fingerprint:sha-256 5D:1D:39:7C:FE:29:91:4E:C8:6C:CF:5B:55:D1:D2:BC:38:5E:3C:AF:79:41:4D:82:32:F0:6E:A4:E7:A1:DD:C6
a=setup:active
a=candidate:3913191201 1 udp 8447 157.245.114.91 59754 typ relay raddr 0.0.0.0 rport 0 generation 0
a=candidate:2564284589 1 udp 1677730047 75.100.125.142 52771 typ srflx raddr 0.0.0.0 rport 0 generation 0
a=candidate:2422434839 1 udp 2113937663 10.0.3.122 61986 typ host generation 0
a=ice-options:ice2,trickle
a=mid:0
a=rtpmap:26 JPEG/90000
a=rtcp-fb:26 goog-remb
a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=rtcp-mux
a=rtcp:9 IN IP4 0.0.0.0
a=end-of-candidates
a=inactive
m=application 9 UDP/DTLS/SCTP webrtc-datachannel
c=IN IP4 0.0.0.0
a=ice-ufrag:JRJJ
a=ice-pwd:DNVTQRUDPANEWTNZBVOJFZOC
a=fingerprint:sha-256 5D:1D:39:7C:FE:29:91:4E:C8:6C:CF:5B:55:D1:D2:BC:38:5E:3C:AF:79:41:4D:82:32:F0:6E:A4:E7:A1:DD:C6
a=setup:active
a=ice-options:ice2,trickle
a=mid:1
a=sctp-port:5000
a=max-message-size:262144
ChristopheI commented 3 weeks ago

Local peer receiving offer:

You mean instead "Local peer creates offer", isn't it ? Then send it to Remote Peer. And you have shared here the SDP sent to Remote peer.

Remote peer receiving answer:

You mean instead "Remote peer creates answer using offer", isn't it ? Then send it to Local Peer. And you have shared here the SDP sent to the Local peer.

Local peer receiving offer, with new video track:

Same as before: You mean instead "Local peer creates offer", isn't it ? Then send it to Remote Peer. And you have shared here the SDP sent to Remote peer.

Remote peer receiving answer:

Again: You mean instead "Remote peer creates answer using offer", isn't it ? Then send it to Local Peer. And you have shared here the SDP sent to the Local peer.

If all I assume before is correct, 1) it's the Local Peer which add Video and don't want to receive any because in the Offer SDP we have this;:

m=video 9 UDP/TLS/RTP/SAVP 26
...
a=sendonly

2) Remote Peer dont' want also to have Video because in its answer SDP we have this:

m=video 9 UDP/TLS/RTP/SAVP 26
...
a=inactive
evan-hilscher commented 3 weeks ago

No, I meant it the way I wrote it. I'm debugging both ends of the connection, and I was only logging the OfferReceived (called by the "local" peer) and AnswerReceived (called by the "remote" peer) methods.

The remote peer is adding the video, and the local peer is answering with inactive.

ChristopheI commented 3 weeks ago

Ok my bad. So on Local Peer, once you have received the SDP offer with a new video track,. If you want to receive it, you have to add a Video track with RecvOnly mode

evan-hilscher commented 3 weeks ago

That makes sense, but what is the best way to detect that the SDP offer contains a new video track? I would expect that the RTPSession would fire some kind of event to let my application know that the SDP offer contains a new video track.

ChristopheI commented 3 weeks ago

Perhaps there is a better way, but I'm doing this :

Extract

var temporaryPeerConnection = new RTCPeerConnection(rtcConfiguration, videoAsPrimary: videoAsPrimary);
temporaryPeerConnection.setRemoteDescription(rtcSessionDescriptionInit);
int nbAudio = temporaryPeerConnection.AudioStreamList.Count;
int nbVideo = temporaryPeerConnection.VideoStreamList.Count;

// Check status of Audio Remote Tracks
...

// Check status of Video Remote Tracks
...

var result = rtcPeerConnection.setRemoteDescription(rtcSessionDescriptionInit);
ChristopheI commented 3 weeks ago

Notice also I follow this rules:

So I can manage this kind of scenario:

evan-hilscher commented 3 weeks ago

I see. Interesting workaround, basically using a dummy RTCPeerConnection as a parser. My workaround was searching for inactive tracks after setting the remote description on the live connection, but I think it's fragile.

The WebRTC specification contains a "track" event (https://w3c.github.io/webrtc-pc/#event-track), which sipsorcery does not appear to implement. I am not certain, but I think the "track" event exists for just this situation.

evan-hilscher commented 3 weeks ago

Actually, I see it commented out here: https://github.com/sipsorcery-org/sipsorcery/blob/06f8e3c922ae0a5ad82cf939dd8170aa7f0d3691/src/net/WebRTC/IRTCPeerConnection.cs#L412-L421

ChristopheI commented 3 weeks ago

Yes. Don't hesitate to create a PR ;) There is so much that can be done to further improve features of this library...

evan-hilscher commented 3 weeks ago

Dang, I was afraid you'd say that 😜 Guess that's my week sorted.