Better WebRTC API - Githubissues

ibc commented 7 years ago

Implement RTCRtpTransceiver object.
Better handling of Unified-Plan (Firefox) and Plan-B (Chrome).
Don't require sending audio+video initial SDP offer from the client. Instead:
- Allow receiving initial SDP offer from client with just audio or video, and just in receive mode.
- Allow mediasoup to generate the SDP offer for new clients.

NOTE: This may change, and probably will do. More in comments below.

ibc commented 7 years ago

SDP offer generated by Chrome 55 and Firefox Nightly 54 for this scenario:

Create a RTCPeerConnection.
Add a local MediaStream with audio and video.
Add a local MediaStream with just video.

Chrome 55

v=0
o=- 21249255991560522 3 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE audio video
a=msid-semantic: WMS A0PJkyE11YU7HMTGf4XqDT0PqaH5pkmpRgri wVZHHQERYnUSk7Ddehop6mF8frhxRqocB1nc

m=audio 9 UDP/TLS/RTP/SAVPF 111 103 104 9 0 8 106 105 13 126
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:dImP
a=ice-pwd:rFInqADrmnT4afhtZUyvnJgk
a=fingerprint:sha-256 2A:33:9F:4C:70:16:F2:EA:17:AD:51:86:D8:E1:2E:82:63:94:EE:89:2F:AF:89:F3:EA:36:9B:8A:9C:69:50:8F
a=setup:actpass
a=mid:audio
a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level
a=extmap:3 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=sendrecv
a=rtcp-mux
a=rtpmap:111 opus/48000/2
a=rtcp-fb:111 transport-cc
a=fmtp:111 minptime=10;useinbandfec=1
a=rtpmap:103 ISAC/16000
a=rtpmap:104 ISAC/32000
a=rtpmap:9 G722/8000
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:106 CN/32000
a=rtpmap:105 CN/16000
a=rtpmap:13 CN/8000
a=rtpmap:126 telephone-event/8000

a=ssrc:2842620998 cname:vCdee5R7uS5u4dVJ
a=ssrc:2842620998 msid:A0PJkyE11YU7HMTGf4XqDT0PqaH5pkmpRgri 9de9961b-20a2-4b78-a3a6-e8526b2f19cf
a=ssrc:2842620998 mslabel:A0PJkyE11YU7HMTGf4XqDT0PqaH5pkmpRgri
a=ssrc:2842620998 label:9de9961b-20a2-4b78-a3a6-e8526b2f19cf

m=video 9 UDP/TLS/RTP/SAVPF 100 101 107 116 117 96 97 99 98
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:dImP
a=ice-pwd:rFInqADrmnT4afhtZUyvnJgk
a=fingerprint:sha-256 2A:33:9F:4C:70:16:F2:EA:17:AD:51:86:D8:E1:2E:82:63:94:EE:89:2F:AF:89:F3:EA:36:9B:8A:9C:69:50:8F
a=setup:actpass
a=mid:video
a=extmap:2 urn:ietf:params:rtp-hdrext:toffset
a=extmap:3 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:4 urn:3gpp:video-orientation
a=extmap:5 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay
a=sendrecv
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:100 VP8/90000
a=rtcp-fb:100 ccm fir
a=rtcp-fb:100 nack
a=rtcp-fb:100 nack pli
a=rtcp-fb:100 goog-remb
a=rtcp-fb:100 transport-cc
a=rtpmap:101 VP9/90000
a=rtcp-fb:101 ccm fir
a=rtcp-fb:101 nack
a=rtcp-fb:101 nack pli
a=rtcp-fb:101 goog-remb
a=rtcp-fb:101 transport-cc
a=rtpmap:107 H264/90000
a=rtcp-fb:107 ccm fir
a=rtcp-fb:107 nack
a=rtcp-fb:107 nack pli
a=rtcp-fb:107 goog-remb
a=rtcp-fb:107 transport-cc
a=fmtp:107 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f
a=rtpmap:116 red/90000
a=rtpmap:117 ulpfec/90000
a=rtpmap:96 rtx/90000
a=fmtp:96 apt=100
a=rtpmap:97 rtx/90000
a=fmtp:97 apt=101
a=rtpmap:99 rtx/90000
a=fmtp:99 apt=107
a=rtpmap:98 rtx/90000
a=fmtp:98 apt=116

a=ssrc-group:FID 2001461993 1533564893
a=ssrc:2001461993 cname:vCdee5R7uS5u4dVJ
a=ssrc:2001461993 msid:A0PJkyE11YU7HMTGf4XqDT0PqaH5pkmpRgri 25b3cf76-b171-43c1-bd0d-d94ba17f5d40
a=ssrc:2001461993 mslabel:A0PJkyE11YU7HMTGf4XqDT0PqaH5pkmpRgri
a=ssrc:2001461993 label:25b3cf76-b171-43c1-bd0d-d94ba17f5d40
a=ssrc:1533564893 cname:vCdee5R7uS5u4dVJ
a=ssrc:1533564893 msid:A0PJkyE11YU7HMTGf4XqDT0PqaH5pkmpRgri 25b3cf76-b171-43c1-bd0d-d94ba17f5d40
a=ssrc:1533564893 mslabel:A0PJkyE11YU7HMTGf4XqDT0PqaH5pkmpRgri
a=ssrc:1533564893 label:25b3cf76-b171-43c1-bd0d-d94ba17f5d40

a=ssrc-group:FID 1311538594 372516962
a=ssrc:1311538594 cname:vCdee5R7uS5u4dVJ
a=ssrc:1311538594 msid:wVZHHQERYnUSk7Ddehop6mF8frhxRqocB1nc e59c9cd6-0597-4863-a947-9be9b9347044
a=ssrc:1311538594 mslabel:wVZHHQERYnUSk7Ddehop6mF8frhxRqocB1nc
a=ssrc:1311538594 label:e59c9cd6-0597-4863-a947-9be9b9347044
a=ssrc:372516962 cname:vCdee5R7uS5u4dVJ
a=ssrc:372516962 msid:wVZHHQERYnUSk7Ddehop6mF8frhxRqocB1nc e59c9cd6-0597-4863-a947-9be9b9347044
a=ssrc:372516962 mslabel:wVZHHQERYnUSk7Ddehop6mF8frhxRqocB1nc
a=ssrc:372516962 label:e59c9cd6-0597-4863-a947-9be9b9347044

Firefox Nightly 54

o=mozilla...THIS_IS_SDPARTA-53.0a1 8086752193817072400 1 IN IP4 0.0.0.0
s=-
t=0 0
a=fingerprint:sha-256 05:89:DB:9F:49:A8:38:76:3D:6B:BA:45:E8:ED:A6:7E:BD:0E:02:66:C1:44:BD:DA:B0:4F:14:BA:9F:08:44:A6
a=group:BUNDLE sdparta_0 sdparta_1 sdparta_2
a=ice-options:trickle
a=msid-semantic:WMS *

m=audio 9 UDP/TLS/RTP/SAVPF 109 9 0 8 101
c=IN IP4 0.0.0.0
a=sendrecv
a=extmap:1/sendonly urn:ietf:params:rtp-hdrext:ssrc-audio-level
a=fmtp:109 maxplaybackrate=48000;stereo=1;useinbandfec=1
a=fmtp:101 0-15
a=ice-pwd:b2c9137388e5ee91d59411ff91d87278
a=ice-ufrag:487ff5ee
a=mid:sdparta_0
a=msid:{2ec8f41a-8038-b046-b1d4-d09949be3e37} {1d462a22-cff1-3b42-a35c-63f1066979fb}
a=rtcp-mux
a=rtpmap:109 opus/48000/2
a=rtpmap:9 G722/8000/1
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:101 telephone-event/8000/1
a=setup:actpass

a=ssrc:1083752730 cname:{944cb849-7420-a542-8d1f-3c1424e9651f}

m=video 9 UDP/TLS/RTP/SAVPF 120 121 126 97
c=IN IP4 0.0.0.0
a=sendrecv
a=fmtp:126 profile-level-id=42e01f;level-asymmetry-allowed=1;packetization-mode=1
a=fmtp:97 profile-level-id=42e01f;level-asymmetry-allowed=1
a=fmtp:120 max-fs=12288;max-fr=60
a=fmtp:121 max-fs=12288;max-fr=60
a=ice-pwd:b2c9137388e5ee91d59411ff91d87278
a=ice-ufrag:487ff5ee
a=mid:sdparta_1
a=msid:{2ec8f41a-8038-b046-b1d4-d09949be3e37} {ff580dfc-38a8-2b46-8382-f1d8f1af843a}
a=rtcp-fb:120 nack
a=rtcp-fb:120 nack pli
a=rtcp-fb:120 ccm fir
a=rtcp-fb:120 goog-remb
a=rtcp-fb:121 nack
a=rtcp-fb:121 nack pli
a=rtcp-fb:121 ccm fir
a=rtcp-fb:121 goog-remb
a=rtcp-fb:126 nack
a=rtcp-fb:126 nack pli
a=rtcp-fb:126 ccm fir
a=rtcp-fb:126 goog-remb
a=rtcp-fb:97 nack
a=rtcp-fb:97 nack pli
a=rtcp-fb:97 ccm fir
a=rtcp-fb:97 goog-remb
a=rtcp-mux
a=rtpmap:120 VP8/90000
a=rtpmap:121 VP9/90000
a=rtpmap:126 H264/90000
a=rtpmap:97 H264/90000
a=setup:actpass

a=ssrc:625109825 cname:{944cb849-7420-a542-8d1f-3c1424e9651f}

m=video 9 UDP/TLS/RTP/SAVPF 120 121 126 97
c=IN IP4 0.0.0.0
a=bundle-only
a=sendrecv
a=fmtp:126 profile-level-id=42e01f;level-asymmetry-allowed=1;packetization-mode=1
a=fmtp:97 profile-level-id=42e01f;level-asymmetry-allowed=1
a=fmtp:120 max-fs=12288;max-fr=60
a=fmtp:121 max-fs=12288;max-fr=60
a=ice-pwd:b2c9137388e5ee91d59411ff91d87278
a=ice-ufrag:487ff5ee
a=mid:sdparta_2
a=msid:{be2be358-719f-cc49-a89d-7ffb8e6f4848} {1d97904f-4006-1048-87e0-4989a7b4dba2}
a=rtcp-fb:120 nack
a=rtcp-fb:120 nack pli
a=rtcp-fb:120 ccm fir
a=rtcp-fb:120 goog-remb
a=rtcp-fb:121 nack
a=rtcp-fb:121 nack pli
a=rtcp-fb:121 ccm fir
a=rtcp-fb:121 goog-remb
a=rtcp-fb:126 nack
a=rtcp-fb:126 nack pli
a=rtcp-fb:126 ccm fir
a=rtcp-fb:126 goog-remb
a=rtcp-fb:97 nack
a=rtcp-fb:97 nack pli
a=rtcp-fb:97 ccm fir
a=rtcp-fb:97 goog-remb
a=rtcp-mux
a=rtpmap:120 VP8/90000
a=rtpmap:121 VP9/90000
a=rtpmap:126 H264/90000
a=rtpmap:97 H264/90000
a=setup:actpass

a=ssrc:2351831813 cname:{944cb849-7420-a542-8d1f-3c1424e9651f}

ibc commented 7 years ago

After considering all the options, I'll go for the following "WebRTC 1.0 signaling procedures" for mediasoup:

Join a room

Let's assume that a browser wants to join a room.
The browser gets its media capabilities as follows:
- It creates a pc.
- It may (or not) attach its local stream to it.
- It calls pc.createOffer({ offerToReceiveAudio:1, offerToReceiveVideo:1 }) and gets an offer that represents its media "capabilities".
Browser sends its capabilities to the room (by using any kind of "join" API provided by the Node app).
The app creates a Peer and a RTCPeerConnection associated to it.
The app then calls pc.setCapabilities() with those capabilities. Such a method does:
- Parses the given SDP and calls peer.setCapabilities().
  - IMPORTANT: By design, before peer.setCapabilities() resolves, it's it guaranteed that all the associated RtpSenders are already created.
- When resolved, it creates a Transport for the browser (may be two if no bundle, etc).
- When resolved, it does not fire negotiationneeded but, instead resolves the promise.
When pc.setCapabilities() resolves, the app should call pc.createOffer() + pc.setLocalDescription() and send the offer to the browser.
pc.createOffer() does:
- It iterates over all the RtpSenders and provides them with the previously created transport.
- Then, it a creates a SDP initial offer for the browser, that contains:
  - The room media capabilities filtered and adapted for the browser according to its capabilities.
  - The transport local parameters.
  - All the RtpSenders information so we don't need yet another O/A.
The browser receives the offer and calls pc.setRemoteDescription(offer).
If no local stream was previously attached to pc, the browser could do it now.
The browser calls pc.createAnswer() + pc.setLocalDescription() and signals the answer to mediasoup pc.
The Node app calls pc.setRemoteDescription(answer) which:
- It sets the DTLS parameters to its local Transport (according to values in the answer) by calling transport.setRemoteDtlsParameters().
- When resolved, it creates the corresponding RtpReceivers with such a transport and calls receive() on them with the appropriate params.
- It sets a newrtpsender listener on the Peer for future changes.

New peer joins

When a new peer joins, newrtpsender will eventually fire for all the other peers, who will be re-offered (nothing new here).

Browser adds/removes audio/video

The browser may add/remove video at any time by re-offering a new SDP with the updates. When mediasoup receives a re-offer from the browser:

mediasoup inspects remote stream tracks addition or removal and creates destroys RtpReceivers according to the updated info.
- This must be inspected different in both Unified Plan and Plan B.

NOTE: As @lminiero warned me, mediasoup must prevent glare (SDP O/A race conditions) in this scenario.

ibc commented 7 years ago

Mostly done and working (for Chrome and Firefox) in the refactor branch.

ibc commented 7 years ago

About merging refector branch into master.

The missing stuff in the new WebRTC API is:

[x] Allow re-offer from the client and inspect its track addition/removal.
- [x] When Firefox adds an audio track and re-offers, if there was an a=inactive second m=audio section, Firefox will fill it but will set a new a=mid on it. Must handle that and match m= sections via numeric index rather than via a=mid.
[ ] In PlanB we must check a=ssrc-simulcast and not just a=ssrc-groups to determine sending tracks.
[ ] API to change the MID of a RtpSender (and mangle the corresponding RTP header extension in RTP packets).
[x] Be ready for client's capabilities without audio or video m= sections.

ibc commented 7 years ago

refactor branch merged into master. Let's keep this issue open until all the pending stuff indicated in the previous comment is done.

ibc commented 7 years ago

Issues in Firefox that affect us:

Finally I'm gonna mandate offers and re-offers always from mediasoup, so first issue is not a problem.

The second one is a bug in Firefox.

ibc commented 7 years ago

Let's close this issue as it's done. There are two pending tasks, but don't belong here:

In PlanB we must check a=ssrc-simulcast and not just a=ssrc-groups to determine sending tracks.

This will be done once simulcast is implemented.

API to change the MID of a RtpSender (and mangle the corresponding RTP header extension in RTP packets).

Finally there is no need for an API like this. RtpSender::SendRtpPacket() must internally do it and update its associated RtpParameters.mid.

versatica / mediasoup

Better WebRTC API #54

Chrome 55

Firefox Nightly 54

Join a room

New peer joins

Browser adds/removes audio/video