search

microsoft / MixedReality-WebRTC

MixedReality-WebRTC is a collection of components to help mixed reality app developers integrate audio and video real-time communication into their application and improve their collaborative experience
https://microsoft.github.io/MixedReality-WebRTC/
MIT License
908 stars 282 forks source link

Question: Legacy offer options. #318

Open HyperLethalVector opened 4 years ago

HyperLethalVector commented 4 years ago

Hi there, quick question!

I've managed to succeed in connecting the MixedReality WebRTC package to another WebRTC library, and along the way I've found that my secondary webrtc library isn't playing back the video feeds on the secondary library once the handshake is successful (but the incoming video and audio feeds do play on the MixedReality WebRTC Package).

The SDP messages are correct, one of my colleagues seems to think that this might be the result of a legacy option not being set.

Is there a way to set offer options when creating the SDP offer within the unity library? or are these automatically set. "offerToReceiveAudio": true, "offerToReceiveVideo": true

Thanks~

HyperLethalVector commented 4 years ago

Just in case it'll help diagnose, I'll leave some SDP message examples, My current setup is as follows:

Client 1) Unity Editor, my third party library Client 2) Unity Editor, MixedReality WebRTC

Generated by alternating between who is sending the initial offer (Client 1 or Client 2)

Under the situation where client 1 is the offer sender, the handshake is successful, but the remote video feed from client 2 doesn't show on client 1 (Client 1's video feed can be seen on client 2 however).

Under the situation where client 2 sends the initial offer, the handshake succeeds, but a couple of asserts fail, and neither remote video feed displays: image image

Situation 1( Offer From the secondary Library):

v=0
o=- 1467090617971097578 2 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE 0 1 2
a=msid-semantic: WMS stream_label
m=audio 9 UDP/TLS/RTP/SAVPF 111 103 104 9 102 0 8 106 105 13 110 112 113 126
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:kSBt
a=ice-pwd:FFGxkqpnXPN7ZpvGWzlxsjra
a=ice-options:trickle
a=fingerprint:sha-256 85:72:2C:D9:3A:AE:B4:F3:DC:7B:C7:75:63:6E:DE:22:B2:74:69:7F:59:6B:58:D7:41:0F:5F:18:AA:66:17:64
a=setup:actpass
a=mid:0
a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level
a=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid
a=sendrecv
a=msid:stream_label audio_label
a=rtcp-mux
a=rtpmap:111 opus/48000/2
a=rtcp-fb:111 transport-cc
a=fmtp:111 minptime=10;useinbandfec=1
a=rtpmap:103 ISAC/16000
a=rtpmap:104 ISAC/32000
a=rtpmap:9 G722/8000
a=rtpmap:102 ILBC/8000
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:106 CN/32000
a=rtpmap:105 CN/16000
a=rtpmap:13 CN/8000
a=rtpmap:110 telephone-event/48000
a=rtpmap:112 telephone-event/32000
a=rtpmap:113 telephone-event/16000
a=rtpmap:126 telephone-event/8000
a=ssrc:923755116 cname:3TxxJJzQwSL8leIG
a=ssrc:923755116 msid:stream_label audio_label
a=ssrc:923755116 mslabel:stream_label
a=ssrc:923755116 label:audio_label
m=video 9 UDP/TLS/RTP/SAVPF 96 97 98 99 100 101 127
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:kSBt
a=ice-pwd:FFGxkqpnXPN7ZpvGWzlxsjra
a=ice-options:trickle
a=fingerprint:sha-256 85:72:2C:D9:3A:AE:B4:F3:DC:7B:C7:75:63:6E:DE:22:B2:74:69:7F:59:6B:58:D7:41:0F:5F:18:AA:66:17:64
a=setup:actpass
a=mid:1
a=extmap:2 urn:ietf:params:rtp-hdrext:toffset
a=extmap:3 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:4 urn:3gpp:video-orientation
a=extmap:5 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay
a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type
a=extmap:8 http://www.webrtc.org/experiments/rtp-hdrext/video-timing
a=extmap:10 http://tools.ietf.org/html/draft-ietf-avtext-framemarking-07
a=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid
a=sendrecv
a=msid:stream_label video_label
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:96 VP8/90000
a=rtcp-fb:96 goog-remb
a=rtcp-fb:96 transport-cc
a=rtcp-fb:96 ccm fir
a=rtcp-fb:96 nack
a=rtcp-fb:96 nack pli
a=rtpmap:97 rtx/90000
a=fmtp:97 apt=96
a=rtpmap:98 VP9/90000
a=rtcp-fb:98 goog-remb
a=rtcp-fb:98 transport-cc
a=rtcp-fb:98 ccm fir
a=rtcp-fb:98 nack
a=rtcp-fb:98 nack pli
a=fmtp:98 profile-id=0
a=rtpmap:99 rtx/90000
a=fmtp:99 apt=98
a=rtpmap:100 red/90000
a=rtpmap:101 rtx/90000
a=fmtp:101 apt=100
a=rtpmap:127 ulpfec/90000
a=ssrc-group:FID 2344523808 2160916162
a=ssrc:2344523808 cname:3TxxJJzQwSL8leIG
a=ssrc:2344523808 msid:stream_label video_label
a=ssrc:2344523808 mslabel:stream_label
a=ssrc:2344523808 label:video_label
a=ssrc:2160916162 cname:3TxxJJzQwSL8leIG
a=ssrc:2160916162 msid:stream_label video_label
a=ssrc:2160916162 mslabel:stream_label
a=ssrc:2160916162 label:video_label
m=application 9 DTLS/SCTP 5000
c=IN IP4 0.0.0.0
a=ice-ufrag:kSBt
a=ice-pwd:FFGxkqpnXPN7ZpvGWzlxsjra
a=ice-options:trickle
a=fingerprint:sha-256 85:72:2C:D9:3A:AE:B4:F3:DC:7B:C7:75:63:6E:DE:22:B2:74:69:7F:59:6B:58:D7:41:0F:5F:18:AA:66:17:64
a=setup:actpass
a=mid:2
a=sctpmap:5000 webrtc-datachannel 1024

The answer generated by MicrosoftWebRTC:

v=0
o=- 7262325101022551452 2 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE 0 1 2
a=msid-semantic: WMS
m=audio 9 UDP/TLS/RTP/SAVPF 111 103 104 9 102 0 8 106 105 13 110 112 113 126
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:pTT6
a=ice-pwd:a+pMebAPXz1xucUNgp/F8N1r
a=ice-options:trickle
a=fingerprint:sha-256 8D:E8:34:CA:09:8E:C2:46:BB:0C:FD:19:F9:9D:10:C3:C0:6F:93:11:BD:9A:CD:77:B7:63:6D:D0:29:8C:1F:7F
a=setup:active
a=mid:0
a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level
a=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid
a=sendrecv
a=msid:- bde35c2c-692a-4655-99a7-ce1d71f6fe8f
a=rtcp-mux
a=rtpmap:111 opus/48000/2
a=rtcp-fb:111 transport-cc
a=fmtp:111 minptime=10;useinbandfec=1
a=rtpmap:103 ISAC/16000
a=rtpmap:104 ISAC/32000
a=rtpmap:9 G722/8000
a=rtpmap:102 ILBC/8000
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:106 CN/32000
a=rtpmap:105 CN/16000
a=rtpmap:13 CN/8000
a=rtpmap:110 telephone-event/48000
a=rtpmap:112 telephone-event/32000
a=rtpmap:113 telephone-event/16000
a=rtpmap:126 telephone-event/8000
a=ssrc:3572538980 cname:3x7XfqbnfBPBory2
m=video 9 UDP/TLS/RTP/SAVPF 96
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:pTT6
a=ice-pwd:a+pMebAPXz1xucUNgp/F8N1r
a=ice-options:trickle
a=fingerprint:sha-256 8D:E8:34:CA:09:8E:C2:46:BB:0C:FD:19:F9:9D:10:C3:C0:6F:93:11:BD:9A:CD:77:B7:63:6D:D0:29:8C:1F:7F
a=setup:active
a=mid:1
a=extmap:2 urn:ietf:params:rtp-hdrext:toffset
a=extmap:3 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:4 urn:3gpp:video-orientation
a=extmap:5 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay
a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type
a=extmap:8 http://www.webrtc.org/experiments/rtp-hdrext/video-timing
a=extmap:10 http://tools.ietf.org/html/draft-ietf-avtext-framemarking-07
a=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid
a=sendrecv
a=msid:- bfdc4c9e-180c-48f1-9aac-277aeb2a27c7
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:96 VP8/90000
a=rtcp-fb:96 goog-remb
a=rtcp-fb:96 transport-cc
a=rtcp-fb:96 ccm fir
a=rtcp-fb:96 nack
a=rtcp-fb:96 nack pli
a=ssrc-group:FID 1080336142 3249997939
a=ssrc:1080336142 cname:3x7XfqbnfBPBory2
a=ssrc:3249997939 cname:3x7XfqbnfBPBory2
m=application 9 DTLS/SCTP 5000
c=IN IP4 0.0.0.0
b=AS:30
a=ice-ufrag:pTT6
a=ice-pwd:a+pMebAPXz1xucUNgp/F8N1r
a=ice-options:trickle
a=fingerprint:sha-256 8D:E8:34:CA:09:8E:C2:46:BB:0C:FD:19:F9:9D:10:C3:C0:6F:93:11:BD:9A:CD:77:B7:63:6D:D0:29:8C:1F:7F
a=setup:active
a=mid:2
a=sctpmap:5000 webrtc-datachannel 1024

Scenario 2: The offer generated by MicrosoftWebRTC:

 v=0
o=- 3841947773570173375 2 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE 0 1 2
a=msid-semantic: WMS
m=audio 9 UDP/TLS/RTP/SAVPF 111 103 104 9 102 0 8 106 105 13 110 112 113 126
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:VnfP
a=ice-pwd:hEBHNSjPmjMu/vuorz+XjLPw
a=ice-options:trickle
a=fingerprint:sha-256 24:63:51:17:6D:7B:49:DB:58:84:46:E3:41:BF:D7:29:E5:C5:A2:53:20:9B:63:92:91:44:E7:94:8D:86:A1:AD
a=setup:actpass
a=mid:0
a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level
a=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid
a=recvonly
a=rtcp-mux
a=rtpmap:111 opus/48000/2
a=rtcp-fb:111 transport-cc
a=fmtp:111 minptime=10;useinbandfec=1
a=rtpmap:103 ISAC/16000
a=rtpmap:104 ISAC/32000
a=rtpmap:9 G722/8000
a=rtpmap:102 ILBC/8000
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:106 CN/32000
a=rtpmap:105 CN/16000
a=rtpmap:13 CN/8000
a=rtpmap:110 telephone-event/48000
a=rtpmap:112 telephone-event/32000
a=rtpmap:113 telephone-event/16000
a=rtpmap:126 telephone-event/8000
m=video 9 UDP/TLS/RTP/SAVPF 96 97 98 99 100 101 127 124 125
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:VnfP
a=ice-pwd:hEBHNSjPmjMu/vuorz+XjLPw
a=ice-options:trickle
a=fingerprint:sha-256 24:63:51:17:6D:7B:49:DB:58:84:46:E3:41:BF:D7:29:E5:C5:A2:53:20:9B:63:92:91:44:E7:94:8D:86:A1:AD
a=setup:actpass
a=mid:1
a=extmap:2 urn:ietf:params:rtp-hdrext:toffset
a=extmap:3 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:4 urn:3gpp:video-orientation
a=extmap:5 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay
a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type
a=extmap:8 http://www.webrtc.org/experiments/rtp-hdrext/video-timing
a=extmap:10 http://tools.ietf.org/html/draft-ietf-avtext-framemarking-07
a=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid
a=recvonly
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:96 VP8/90000
a=rtcp-fb:96 goog-remb
a=rtcp-fb:96 transport-cc
a=rtcp-fb:96 ccm fir
a=rtcp-fb:96 nack
a=rtcp-fb:96 nack pli
a=rtpmap:97 rtx/90000
a=fmtp:97 apt=96
a=rtpmap:98 VP9/90000
a=rtcp-fb:98 goog-remb
a=rtcp-fb:98 transport-cc
a=rtcp-fb:98 ccm fir
a=rtcp-fb:98 nack
a=rtcp-fb:98 nack pli
a=fmtp:98 x-google-profile-id=0
a=rtpmap:99 rtx/90000
a=fmtp:99 apt=98
a=rtpmap:100 multiplex/90000
a=rtcp-fb:100 goog-remb
a=rtcp-fb:100 transport-cc
a=rtcp-fb:100 ccm fir
a=rtcp-fb:100 nack
a=rtcp-fb:100 nack pli
a=fmtp:100 acn=VP9;x-google-profile-id=0
a=rtpmap:101 rtx/90000
a=fmtp:101 apt=100
a=rtpmap:127 red/90000
a=rtpmap:124 rtx/90000
a=fmtp:124 apt=127
a=rtpmap:125 ulpfec/90000
m=application 9 DTLS/SCTP 5000
c=IN IP4 0.0.0.0
a=ice-ufrag:VnfP
a=ice-pwd:hEBHNSjPmjMu/vuorz+XjLPw
a=ice-options:trickle
a=fingerprint:sha-256 24:63:51:17:6D:7B:49:DB:58:84:46:E3:41:BF:D7:29:E5:C5:A2:53:20:9B:63:92:91:44:E7:94:8D:86:A1:AD
a=setup:actpass
a=mid:2
a=sctpmap:5000 webrtc-datachannel 1024

Answer (From the secondary Library):

v=0
o=- 717269892407893289 2 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE 0 1 2
a=msid-semantic: WMS stream_label
m=audio 9 UDP/TLS/RTP/SAVPF 111 103 104 9 102 0 8 106 105 13 110 112 113 126
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:5T9a
a=ice-pwd:1rWQn7I1BaJeOz4UcIP+8w4i
a=ice-options:trickle
a=fingerprint:sha-256 B5:2E:C0:3D:D3:0E:1F:FB:8A:CC:50:18:F0:22:71:AD:F1:49:97:60:89:D0:67:76:A3:E4:E5:FA:27:38:99:D6
a=setup:active
a=mid:0
a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level
a=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid
a=sendonly
a=msid:stream_label audio_label
a=rtcp-mux
a=rtpmap:111 opus/48000/2
a=rtcp-fb:111 transport-cc
a=fmtp:111 minptime=10;useinbandfec=1
a=rtpmap:103 ISAC/16000
a=rtpmap:104 ISAC/32000
a=rtpmap:9 G722/8000
a=rtpmap:102 ILBC/8000
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:106 CN/32000
a=rtpmap:105 CN/16000
a=rtpmap:13 CN/8000
a=rtpmap:110 telephone-event/48000
a=rtpmap:112 telephone-event/32000
a=rtpmap:113 telephone-event/16000
a=rtpmap:126 telephone-event/8000
a=ssrc:1788781833 cname:P5+V/Ld66GaWlnko
a=ssrc:1788781833 msid:stream_label audio_label
a=ssrc:1788781833 mslabel:stream_label
a=ssrc:1788781833 label:audio_label
m=video 9 UDP/TLS/RTP/SAVPF 96 97 98 99 127 124 125
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:5T9a
a=ice-pwd:1rWQn7I1BaJeOz4UcIP+8w4i
a=ice-options:trickle
a=fingerprint:sha-256 B5:2E:C0:3D:D3:0E:1F:FB:8A:CC:50:18:F0:22:71:AD:F1:49:97:60:89:D0:67:76:A3:E4:E5:FA:27:38:99:D6
a=setup:active
a=mid:1
a=extmap:2 urn:ietf:params:rtp-hdrext:toffset
a=extmap:3 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:4 urn:3gpp:video-orientation
a=extmap:5 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay
a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type
a=extmap:8 http://www.webrtc.org/experiments/rtp-hdrext/video-timing
a=extmap:10 http://tools.ietf.org/html/draft-ietf-avtext-framemarking-07
a=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid
a=sendonly
a=msid:stream_label video_label
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:96 VP8/90000
a=rtcp-fb:96 goog-remb
a=rtcp-fb:96 transport-cc
a=rtcp-fb:96 ccm fir
a=rtcp-fb:96 nack
a=rtcp-fb:96 nack pli
a=rtpmap:97 rtx/90000
a=fmtp:97 apt=96
a=rtpmap:98 VP9/90000
a=rtcp-fb:98 goog-remb
a=rtcp-fb:98 transport-cc
a=rtcp-fb:98 ccm fir
a=rtcp-fb:98 nack
a=rtcp-fb:98 nack pli
a=fmtp:98 profile-id=0
a=rtpmap:99 rtx/90000
a=fmtp:99 apt=98
a=rtpmap:127 red/90000
a=rtpmap:124 rtx/90000
a=fmtp:124 apt=127
a=rtpmap:125 ulpfec/90000
a=ssrc-group:FID 3539970632 1095039150
a=ssrc:3539970632 cname:P5+V/Ld66GaWlnko
a=ssrc:3539970632 msid:stream_label video_label
a=ssrc:3539970632 mslabel:stream_label
a=ssrc:3539970632 label:video_label
a=ssrc:1095039150 cname:P5+V/Ld66GaWlnko
a=ssrc:1095039150 msid:stream_label video_label
a=ssrc:1095039150 mslabel:stream_label
a=ssrc:1095039150 label:video_label
m=application 9 DTLS/SCTP 5000
c=IN IP4 0.0.0.0
b=AS:30
a=ice-ufrag:5T9a
a=ice-pwd:1rWQn7I1BaJeOz4UcIP+8w4i
a=ice-options:trickle
a=fingerprint:sha-256 B5:2E:C0:3D:D3:0E:1F:FB:8A:CC:50:18:F0:22:71:AD:F1:49:97:60:89:D0:67:76:A3:E4:E5:FA:27:38:99:D6
a=setup:active
a=mid:2
a=sctpmap:5000 webrtc-datachannel 1024
djee-ms commented 4 years ago

How are the transceivers set up on the PeerConnection?

The SDP messages are correct, one of my colleagues seems to think that this might be the result of a legacy option not being set.

That is not possible. The legacy option is just changing the SDP messages. Either the SDP messages are correct, and the legacy options won't do anything, or the SDP messages are wrong, and the legacy options are just the legacy way to modify them, and there is another better way.

In your Scenario 2, MixedReality-WebRTC is offering to receive only, and the third-party client is accepting to send only. So as expected the video is not transmitted. Hence my question, how did you configure the transceiver list on the PeerConnection Unity component?

HyperLethalVector commented 4 years ago

Hi there, Thanks for helping with the insight, it makes alot of sense.
I'll presume the SDP messages are at least valid since the handshake succeeds. As I said in scenario #1, the remote client's video shows on the mixed reality webrtc project. but the video transmitted by the mixed reality webrtc project doesn't show on the remote client.

I've attached a screenshot to show you how the transceivers are set up. image

My understanding is that the setup I've created creates a input/output video channel, and an input/output audio channel. Would this be wrong?

djee-ms commented 4 years ago

Yes that's what it should do, the setup looks correct to me. And the offer doesn't match that, so there is something wrong here.

Can I confirm you are calling PeerConnection.StartConnection() and not PeerConnection.Peer.CreateOffer()? Because the former does some extra work, and so the latter should not be used with the Unity integration.

HyperLethalVector commented 4 years ago

Thanks for the tip, you were correct, I was calling CreateOffer() instead of StartConnection();

After changing, there is no longer any assert errors and the SDP message is far better, but sadly I still don't receive the video on the remote library. (this is in both scenarios)

a=group:BUNDLE 0 1 2
a=msid-semantic: WMS
m=audio 2079 UDP/TLS/RTP/SAVPF 111 103 104 9 102 0 8 106 105 13 110 112 113 126
c=IN IP4 180.150.115.121
a=rtcp:9 IN IP4 0.0.0.0
a=candidate:3257970329 1 udp 2122260223 172.24.192.1 56861 typ host generation 0 network-id 1
a=candidate:2999745851 1 udp 2122194687 192.168.56.1 56862 typ host generation 0 network-id 2
a=candidate:2530088836 1 udp 2122129151 192.168.1.106 56863 typ host generation 0 network-id 3 network-cost 10
a=candidate:2360562281 1 tcp 1518280447 172.24.192.1 60516 typ host tcptype passive generation 0 network-id 1
a=candidate:4233069003 1 tcp 1518214911 192.168.56.1 60517 typ host tcptype passive generation 0 network-id 2
a=candidate:3628985204 1 tcp 1518149375 192.168.1.106 60518 typ host tcptype passive generation 0 network-id 3 network-cost 10
a=candidate:394662192 1 udp 1685921535 180.150.115.121 2079 typ srflx raddr 192.168.1.106 rport 56863 generation 0 network-id 3 network-cost 10
a=ice-ufrag:pZMn
a=ice-pwd:EaIYoAOp6HDYPLBKp9es/6F5
a=ice-options:trickle
a=fingerprint:sha-256 B8:05:CE:22:27:E2:9D:52:51:97:F2:0A:C2:D3:B5:27:68:D4:3A:A8:1A:27:4D:40:B3:1C:F2:28:E8:13:1C:F3
a=setup:actpass
a=mid:0
a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level
a=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid
a=sendrecv
a=msid:- 5b52a87f-d46f-4b40-b31a-a596a8013796
a=rtcp-mux
a=rtpmap:111 opus/48000/2
a=rtcp-fb:111 transport-cc
a=fmtp:111 minptime=10;useinbandfec=1
a=rtpmap:103 ISAC/16000
a=rtpmap:104 ISAC/32000
a=rtpmap:9 G722/8000
a=rtpmap:102 ILBC/8000
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:106 CN/32000
a=rtpmap:105 CN/16000
a=rtpmap:13 CN/8000
a=rtpmap:110 telephone-event/48000
a=rtpmap:112 telephone-event/32000
a=rtpmap:113 telephone-event/16000
a=rtpmap:126 telephone-event/8000
a=ssrc:2880911330 cname:qtW35SOB9WUW//fB
a=ssrc:2880911330 msid: 5b52a87f-d46f-4b40-b31a-a596a8013796
a=ssrc:2880911330 mslabel:
a=ssrc:2880911330 label:5b52a87f-d46f-4b40-b31a-a596a8013796
m=video 2082 UDP/TLS/RTP/SAVPF 96
c=IN IP4 180.150.115.121
a=rtcp:9 IN IP4 0.0.0.0
a=candidate:3257970329 1 udp 2122260223 172.24.192.1 56864 typ host generation 0 network-id 1
a=candidate:2999745851 1 udp 2122194687 192.168.56.1 56865 typ host generation 0 network-id 2
a=candidate:2530088836 1 udp 2122129151 192.168.1.106 56866 typ host generation 0 network-id 3 network-cost 10
a=candidate:2360562281 1 tcp 1518280447 172.24.192.1 60519 typ host tcptype passive generation 0 network-id 1
a=candidate:4233069003 1 tcp 1518214911 192.168.56.1 60520 typ host tcptype passive generation 0 network-id 2
a=candidate:3628985204 1 tcp 1518149375 192.168.1.106 60521 typ host tcptype passive generation 0 network-id 3 network-cost 10
a=candidate:394662192 1 udp 1685921535 180.150.115.121 2082 typ srflx raddr 192.168.1.106 rport 56866 generation 0 network-id 3 network-cost 10
a=ice-ufrag:pZMn
a=ice-pwd:EaIYoAOp6HDYPLBKp9es/6F5
a=ice-options:trickle
a=fingerprint:sha-256 B8:05:CE:22:27:E2:9D:52:51:97:F2:0A:C2:D3:B5:27:68:D4:3A:A8:1A:27:4D:40:B3:1C:F2:28:E8:13:1C:F3
a=setup:actpass
a=mid:1
a=extmap:2 urn:ietf:params:rtp-hdrext:toffset
a=extmap:3 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:4 urn:3gpp:video-orientation
a=extmap:5 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay
a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type
a=extmap:8 http://www.webrtc.org/experiments/rtp-hdrext/video-timing
a=extmap:10 http://tools.ietf.org/html/draft-ietf-avtext-framemarking-07
a=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid
a=sendrecv
a=msid:- a7fb9cea-5ec0-4f14-9a59-c8f6108dca37
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:96 VP8/90000
a=rtcp-fb:96 goog-remb
a=rtcp-fb:96 transport-cc
a=rtcp-fb:96 ccm fir
a=rtcp-fb:96 nack
a=rtcp-fb:96 nack pli
a=ssrc-group:FID 909536087 795786900
a=ssrc:909536087 cname:qtW35SOB9WUW//fB
a=ssrc:909536087 msid: a7fb9cea-5ec0-4f14-9a59-c8f6108dca37
a=ssrc:909536087 mslabel:
a=ssrc:909536087 label:a7fb9cea-5ec0-4f14-9a59-c8f6108dca37
a=ssrc:795786900 cname:qtW35SOB9WUW//fB
a=ssrc:795786900 msid: a7fb9cea-5ec0-4f14-9a59-c8f6108dca37
a=ssrc:795786900 mslabel:
a=ssrc:795786900 label:a7fb9cea-5ec0-4f14-9a59-c8f6108dca37
m=application 2085 DTLS/SCTP 5000
c=IN IP4 180.150.115.121
a=candidate:3257970329 1 udp 2122260223 172.24.192.1 56867 typ host generation 0 network-id 1
a=candidate:2999745851 1 udp 2122194687 192.168.56.1 56868 typ host generation 0 network-id 2
a=candidate:2530088836 1 udp 2122129151 192.168.1.106 56869 typ host generation 0 network-id 3 network-cost 10
a=candidate:2360562281 1 tcp 1518280447 172.24.192.1 60522 typ host tcptype passive generation 0 network-id 1
a=candidate:4233069003 1 tcp 1518214911 192.168.56.1 60523 typ host tcptype passive generation 0 network-id 2
a=candidate:3628985204 1 tcp 1518149375 192.168.1.106 60524 typ host tcptype passive generation 0 network-id 3 network-cost 10
a=candidate:394662192 1 udp 1685921535 180.150.115.121 2085 typ srflx raddr 192.168.1.106 rport 56869 generation 0 network-id 3 network-cost 10
a=ice-ufrag:pZMn
a=ice-pwd:EaIYoAOp6HDYPLBKp9es/6F5
a=ice-options:trickle
a=fingerprint:sha-256 B8:05:CE:22:27:E2:9D:52:51:97:F2:0A:C2:D3:B5:27:68:D4:3A:A8:1A:27:4D:40:B3:1C:F2:28:E8:13:1C:F3
a=setup:actpass
a=mid:2
a=sctpmap:5000 webrtc-datachannel 1024
djee-ms commented 4 years ago

Ok now just for good measure, are you using PeerConnection.HandleConnectionMessageAsync() instead of PeerConnection.Peer.SetRemoteDescriptionAsync()? For the same reasons :)

HyperLethalVector commented 4 years ago

HandleConnectionMessageAsync is what I'm using, see the following snippet: image

djee-ms commented 4 years ago

Alright, so next thing to validate is whether the video track is added. The easiest way is to add an event handler on PeerConnection.Peer.VideoTrackAdded. That will tell us if the track is actually created and reaches C#, or if there's a problem while negotiating.

If that works, then next is making sure the VideoReceiver component gets paired with that track. If you can put a breakpoint or debug log in VideoReceiver.OnPaired(MediaTrack track) you should see it called from HandleConnectionMessageAsync().

After that it's a matter of correctly registering a callback with the VideoReceiver, which is only possible once the track is created; see VideoReceiver.RegisterCallback(I420AVideoFrameDelegate callback). One possible error is is to call that before the track exists, in which case this skips the registration. Admittedly we should probably not silently do nothing, and at least log a warning here.

HyperLethalVector commented 4 years ago

Hi there, I've decided to continue testing using the MixedRealityWebRTC package as both the client and host to eliminate the third party package as the source of the issue. The host doesn't receive the client's video! (Which matches my earlier scenario with the third party library)

I checked with PeerConnection.Peer.VideoTrackAdded, Seems only the remote video track is added, at least this is the only time I get a debug log

This is the offer/answer from the host perspective (which doesn't play the client video)

Sending Offer: v=0
o=- 7708104285549180821 2 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE 0 1 2
a=msid-semantic: WMS
m=audio 9 UDP/TLS/RTP/SAVPF 111 103 104 9 102 0 8 106 105 13 110 112 113 126
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:cjjA
a=ice-pwd:NOy3t9Ou46xJsk+Cbf5R0y2x
a=ice-options:trickle
a=fingerprint:sha-256 89:8A:12:9D:A7:9E:D0:65:57:C8:99:B5:2C:58:71:4F:40:20:0B:72:6A:86:4A:0F:4C:92:E0:85:AA:5B:34:3A
a=setup:actpass
a=mid:0
a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level
a=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid
a=sendonly
a=msid:- ed13fa46-cdbc-4c71-8754-206c80bffe56
a=rtcp-mux
a=rtpmap:111 opus/48000/2
a=rtcp-fb:111 transport-cc
a=fmtp:111 minptime=10;useinbandfec=1
a=rtpmap:103 ISAC/16000
a=rtpmap:104 ISAC/32000
a=rtpmap:9 G722/8000
a=rtpmap:102 ILBC/8000
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:106 CN/32000
a=rtpmap:105 CN/16000
a=rtpmap:13 CN/8000
a=rtpmap:110 telephone-event/48000
a=rtpmap:112 telephone-event/32000
a=rtpmap:113 telephone-event/16000
a=rtpmap:126 telephone-event/8000
a=ssrc:3231927114 cname:3VQJdTvGebwMcuX8
a=ssrc:3231927114 msid: ed13fa46-cdbc-4c71-8754-206c80bffe56
a=ssrc:3231927114 mslabel:
a=ssrc:3231927114 label:ed13fa46-cdbc-4c71-8754-206c80bffe56
m=video 9 UDP/TLS/RTP/SAVPF 96
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:cjjA
a=ice-pwd:NOy3t9Ou46xJsk+Cbf5R0y2x
a=ice-options:trickle
a=fingerprint:sha-256 89:8A:12:9D:A7:9E:D0:65:57:C8:99:B5:2C:58:71:4F:40:20:0B:72:6A:86:4A:0F:4C:92:E0:85:AA:5B:34:3A
a=setup:actpass
a=mid:1
a=extmap:2 urn:ietf:params:rtp-hdrext:toffset
a=extmap:3 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:4 urn:3gpp:video-orientation
a=extmap:5 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay
a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type
a=extmap:8 http://www.webrtc.org/experiments/rtp-hdrext/video-timing
a=extmap:10 http://tools.ietf.org/html/draft-ietf-avtext-framemarking-07
a=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid
a=sendonly
a=msid:- 0b053ecc-f457-4f51-a725-fc79bc196c89
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:96 VP8/90000
a=rtcp-fb:96 goog-remb
a=rtcp-fb:96 transport-cc
a=rtcp-fb:96 ccm fir
a=rtcp-fb:96 nack
a=rtcp-fb:96 nack pli
a=ssrc-group:FID 1048513215 421907194
a=ssrc:1048513215 cname:3VQJdTvGebwMcuX8
a=ssrc:1048513215 msid: 0b053ecc-f457-4f51-a725-fc79bc196c89
a=ssrc:1048513215 mslabel:
a=ssrc:1048513215 label:0b053ecc-f457-4f51-a725-fc79bc196c89
a=ssrc:421907194 cname:3VQJdTvGebwMcuX8
a=ssrc:421907194 msid: 0b053ecc-f457-4f51-a725-fc79bc196c89
a=ssrc:421907194 mslabel:
a=ssrc:421907194 label:0b053ecc-f457-4f51-a725-fc79bc196c89
m=application 9 DTLS/SCTP 5000
c=IN IP4 0.0.0.0
a=ice-ufrag:cjjA
a=ice-pwd:NOy3t9Ou46xJsk+Cbf5R0y2x
a=ice-options:trickle
a=fingerprint:sha-256 89:8A:12:9D:A7:9E:D0:65:57:C8:99:B5:2C:58:71:4F:40:20:0B:72:6A:86:4A:0F:4C:92:E0:85:AA:5B:34:3A
a=setup:actpass
a=mid:2
a=sctpmap:5000 webrtc-datachannel 1024

UnityEngine.Debug:Log(Object)
DenkiKitchen.WebRTC.WebSocketSignaler:OnSdpOfferReadyToSend(String) (at Assets/Scripts/WebSocketSignaler.cs:225)
Microsoft.MixedReality.WebRTC.Unity.<>c__DisplayClass10_0:<OnLocalSdpReadyToSend_Listener>b__0() (at Assets/Microsoft.MixedReality.WebRTC.Unity/Scripts/Signaling/Signaler.cs:185)
Microsoft.MixedReality.WebRTC.Unity.Signaler:Update() (at Assets/Microsoft.MixedReality.WebRTC.Unity/Scripts/Signaling/Signaler.cs:210)
DenkiKitchen.WebRTC.WebSocketSignaler:Update() (at Assets/Scripts/WebSocketSignaler.cs:175)
Received Answer: v=0
o=- 8571794146507220566 2 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE 0 1 2
a=msid-semantic: WMS
m=audio 9 UDP/TLS/RTP/SAVPF 111 103 104 9 102 0 8 106 105 13 110 112 113 126
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:AtK7
a=ice-pwd:ewUPcKWk00tAhqF3ruRAW7Me
a=ice-options:trickle
a=fingerprint:sha-256 4D:9C:8D:36:0E:30:31:E6:A9:3B:A3:C5:E0:5F:E7:9C:71:F1:09:8A:88:27:8D:FD:73:D3:A8:07:D0:11:BC:64
a=setup:active
a=mid:0
a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level
a=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid
a=recvonly
a=msid:- 78ee86f1-ad86-488a-8503-e35ec08e1fa5
a=rtcp-mux
a=rtpmap:111 opus/48000/2
a=rtcp-fb:111 transport-cc
a=fmtp:111 minptime=10;useinbandfec=1
a=rtpmap:103 ISAC/16000
a=rtpmap:104 ISAC/32000
a=rtpmap:9 G722/8000
a=rtpmap:102 ILBC/8000
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:106 CN/32000
a=rtpmap:105 CN/16000
a=rtpmap:13 CN/8000
a=rtpmap:110 telephone-event/48000
a=rtpmap:112 telephone-event/32000
a=rtpmap:113 telephone-event/16000
a=rtpmap:126 telephone-event/8000
a=ssrc:1342451711 cname:N9xxcMCjETs64O/B
m=video 9 UDP/TLS/RTP/SAVPF 96
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:AtK7
a=ice-pwd:ewUPcKWk00tAhqF3ruRAW7Me
a=ice-options:trickle
a=fingerprint:sha-256 4D:9C:8D:36:0E:30:31:E6:A9:3B:A3:C5:E0:5F:E7:9C:71:F1:09:8A:88:27:8D:FD:73:D3:A8:07:D0:11:BC:64
a=setup:active
a=mid:1
a=extmap:2 urn:ietf:params:rtp-hdrext:toffset
a=extmap:3 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:4 urn:3gpp:video-orientation
a=extmap:5 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay
a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type
a=extmap:8 http://www.webrtc.org/experiments/rtp-hdrext/video-timing
a=extmap:10 http://tools.ietf.org/html/draft-ietf-avtext-framemarking-07
a=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid
a=recvonly
a=msid:- a6f5e8ae-1afd-4952-937f-87d4249452ed
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:96 VP8/90000
a=rtcp-fb:96 goog-remb
a=rtcp-fb:96 transport-cc
a=rtcp-fb:96 ccm fir
a=rtcp-fb:96 nack
a=rtcp-fb:96 nack pli
a=ssrc:4031464999 cname:N9xxcMCjETs64O/B
m=application 9 DTLS/SCTP 5000
c=IN IP4 0.0.0.0
b=AS:30
a=ice-ufrag:AtK7
a=ice-pwd:ewUPcKWk00tAhqF3ruRAW7Me
a=ice-options:trickle
a=fingerprint:sha-256 4D:9C:8D:36:0E:30:31:E6:A9:3B:A3:C5:E0:5F:E7:9C:71:F1:09:8A:88:27:8D:FD:73:D3:A8:07:D0:11:BC:64
a=setup:active
a=mid:2
a=sctpmap:5000 webrtc-datachannel 1024

UnityEngine.Debug:Log(Object)
DenkiKitchen.WebRTC.WebSocketSignaler:ProcessMessage(Byte[]) (at Assets/Scripts/WebSocketSignaler.cs:123)
DenkiKitchen.WebRTC.WebSocketSignaler:<AfterInit2>b__18_3(Byte[]) (at Assets/Scripts/WebSocketSignaler.cs:98)
NativeWebSocket.<Receive>d__26:MoveNext() (at Assets/Scripts/WebSocket/WebSocket.cs:466)
UnityEngine.UnitySynchronizationContext:ExecuteTasks()

I don't see any call back for OnPaired, nor do I get any debug callback for the remote video for PeerConnection.Peer.VideoTrackAdded. image

I'll try this again using node-dss and report back

HyperLethalVector commented 4 years ago

Hi,

After pulling the latest changes, and remodifying my socket signaller, I can create and succeed in a complete connection between MixedRealityWebRTC as both the host/client, with both node-dss and my custom websocket signalling.

Upon moving back to connecting MixedRealityWebRTC to my third party library, I still have the same issue (the video playback not showing on the third party library).

Now I'm unsure why this is the case, I've attached is the SDP offer/answer (with MixedRealityWebRTC being the one sending the answer)

EDIT: I do receive a warning when trying to force the VP8 codec, maybe it's related? PreferredVideoCodec is currently a per-PeerConnection setting; overriding the value for peer connection '' with track's value of 'VP8', I do think it's switching back to the VP9 codec which might not be supported by the third party library.

Edit2: The audio stream works well! It's only the video stream that's not being received properly. It's very strange!

Received Offer: v=0
o=- 5319249788270952167 2 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE 0 1 2
a=msid-semantic: WMS stream_label
m=audio 9 UDP/TLS/RTP/SAVPF 111 103 104 9 102 0 8 106 105 13 110 112 113 126
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:TmJl
a=ice-pwd:M8DjRoiaAVfU2vf50g+8xDCx
a=ice-options:trickle
a=fingerprint:sha-256 8B:35:19:BC:42:2B:D1:ED:06:43:6A:0F:20:FD:FD:0D:12:0C:A2:1F:11:1D:49:56:40:2C:A4:03:3E:40:2B:D3
a=setup:actpass
a=mid:0
a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level
a=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid
a=sendrecv
a=msid:stream_label audio_label
a=rtcp-mux
a=rtpmap:111 opus/48000/2
a=rtcp-fb:111 transport-cc
a=fmtp:111 minptime=10;useinbandfec=1
a=rtpmap:103 ISAC/16000
a=rtpmap:104 ISAC/32000
a=rtpmap:9 G722/8000
a=rtpmap:102 ILBC/8000
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:106 CN/32000
a=rtpmap:105 CN/16000
a=rtpmap:13 CN/8000
a=rtpmap:110 telephone-event/48000
a=rtpmap:112 telephone-event/32000
a=rtpmap:113 telephone-event/16000
a=rtpmap:126 telephone-event/8000
a=ssrc:133125592 cname:vYxeShNsucmTm4av
a=ssrc:133125592 msid:stream_label audio_label
a=ssrc:133125592 mslabel:stream_label
a=ssrc:133125592 label:audio_label
m=video 9 UDP/TLS/RTP/SAVPF 96 97 98 99 100 101 127
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:TmJl
a=ice-pwd:M8DjRoiaAVfU2vf50g+8xDCx
a=ice-options:trickle
a=fingerprint:sha-256 8B:35:19:BC:42:2B:D1:ED:06:43:6A:0F:20:FD:FD:0D:12:0C:A2:1F:11:1D:49:56:40:2C:A4:03:3E:40:2B:D3
a=setup:actpass
a=mid:1
a=extmap:2 urn:ietf:params:rtp-hdrext:toffset
a=extmap:3 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:4 urn:3gpp:video-orientation
a=extmap:5 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay
a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type
a=extmap:8 http://www.webrtc.org/experiments/rtp-hdrext/video-timing
a=extmap:10 http://tools.ietf.org/html/draft-ietf-avtext-framemarking-07
a=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid
a=sendrecv
a=msid:stream_label video_label
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:96 VP8/90000
a=rtcp-fb:96 goog-remb
a=rtcp-fb:96 transport-cc
a=rtcp-fb:96 ccm fir
a=rtcp-fb:96 nack
a=rtcp-fb:96 nack pli
a=rtpmap:97 rtx/90000
a=fmtp:97 apt=96
a=rtpmap:98 VP9/90000
a=rtcp-fb:98 goog-remb
a=rtcp-fb:98 transport-cc
a=rtcp-fb:98 ccm fir
a=rtcp-fb:98 nack
a=rtcp-fb:98 nack pli
a=fmtp:98 profile-id=0
a=rtpmap:99 rtx/90000
a=fmtp:99 apt=98
a=rtpmap:100 red/90000
a=rtpmap:101 rtx/90000
a=fmtp:101 apt=100
a=rtpmap:127 ulpfec/90000
a=ssrc-group:FID 751875419 3342962449
a=ssrc:751875419 cname:vYxeShNsucmTm4av
a=ssrc:751875419 msid:stream_label video_label
a=ssrc:751875419 mslabel:stream_label
a=ssrc:751875419 label:video_label
a=ssrc:3342962449 cname:vYxeShNsucmTm4av
a=ssrc:3342962449 msid:stream_label video_label
a=ssrc:3342962449 mslabel:stream_label
a=ssrc:3342962449 label:video_label
m=application 9 DTLS/SCTP 5000
c=IN IP4 0.0.0.0
a=ice-ufrag:TmJl
a=ice-pwd:M8DjRoiaAVfU2vf50g+8xDCx
a=ice-options:trickle
a=fingerprint:sha-256 8B:35:19:BC:42:2B:D1:ED:06:43:6A:0F:20:FD:FD:0D:12:0C:A2:1F:11:1D:49:56:40:2C:A4:03:3E:40:2B:D3
a=setup:actpass
a=mid:2
a=sctpmap:5000 webrtc-datachannel 1024

UnityEngine.Debug:Log(Object)
DenkiKitchen.WebRTC.WebSocketSignaler:ProcessMessage(Byte[]) (at Assets/Scripts/WebSocketSignaler.cs:122)
DenkiKitchen.WebRTC.WebSocketSignaler:<AfterInit2>b__19_3(Byte[]) (at Assets/Scripts/WebSocketSignaler.cs:103)
NativeWebSocket.<Receive>d__26:MoveNext() (at Assets/Scripts/WebSocket/WebSocket.cs:466)
UnityEngine.UnitySynchronizationContext:ExecuteTasks()
Sending Answer: v=0
o=- 3438589431541447947 2 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE 0 1 2
a=msid-semantic: WMS
m=audio 9 UDP/TLS/RTP/SAVPF 111 103 104 9 102 0 8 106 105 13 110 112 113 126
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:Np+C
a=ice-pwd:/RXuj94hNaXvVP1lkqUV84hY
a=ice-options:trickle
a=fingerprint:sha-256 E1:A7:D0:8C:61:A8:5A:AE:89:9E:53:09:79:30:25:F2:4E:0D:4B:0B:88:8D:2A:00:8F:8F:CB:2E:BA:B7:D2:2D
a=setup:active
a=mid:0
a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level
a=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid
a=sendrecv
a=msid:- ac794073-1cfb-4949-a0d6-13f543891346
a=rtcp-mux
a=rtpmap:111 opus/48000/2
a=rtcp-fb:111 transport-cc
a=fmtp:111 minptime=10;useinbandfec=1
a=rtpmap:103 ISAC/16000
a=rtpmap:104 ISAC/32000
a=rtpmap:9 G722/8000
a=rtpmap:102 ILBC/8000
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:106 CN/32000
a=rtpmap:105 CN/16000
a=rtpmap:13 CN/8000
a=rtpmap:110 telephone-event/48000
a=rtpmap:112 telephone-event/32000
a=rtpmap:113 telephone-event/16000
a=rtpmap:126 telephone-event/8000
a=ssrc:3608413869 cname:F5L7tCzL8Ul1F9Od
m=video 9 UDP/TLS/RTP/SAVPF 96
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:Np+C
a=ice-pwd:/RXuj94hNaXvVP1lkqUV84hY
a=ice-options:trickle
a=fingerprint:sha-256 E1:A7:D0:8C:61:A8:5A:AE:89:9E:53:09:79:30:25:F2:4E:0D:4B:0B:88:8D:2A:00:8F:8F:CB:2E:BA:B7:D2:2D
a=setup:active
a=mid:1
a=extmap:2 urn:ietf:params:rtp-hdrext:toffset
a=extmap:3 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:4 urn:3gpp:video-orientation
a=extmap:5 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay
a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type
a=extmap:8 http://www.webrtc.org/experiments/rtp-hdrext/video-timing
a=extmap:10 http://tools.ietf.org/html/draft-ietf-avtext-framemarking-07
a=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid
a=sendrecv
a=msid:- 65304cf3-6911-4796-93d5-a30a94928cd1
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:96 VP8/90000
a=rtcp-fb:96 goog-remb
a=rtcp-fb:96 transport-cc
a=rtcp-fb:96 ccm fir
a=rtcp-fb:96 nack
a=rtcp-fb:96 nack pli
a=ssrc-group:FID 2871221997 1488734963
a=ssrc:2871221997 cname:F5L7tCzL8Ul1F9Od
a=ssrc:1488734963 cname:F5L7tCzL8Ul1F9Od
m=application 9 DTLS/SCTP 5000
c=IN IP4 0.0.0.0
b=AS:30
a=ice-ufrag:Np+C
a=ice-pwd:/RXuj94hNaXvVP1lkqUV84hY
a=ice-options:trickle
a=fingerprint:sha-256 E1:A7:D0:8C:61:A8:5A:AE:89:9E:53:09:79:30:25:F2:4E:0D:4B:0B:88:8D:2A:00:8F:8F:CB:2E:BA:B7:D2:2D
a=setup:active
a=mid:2
a=sctpmap:5000 webrtc-datachannel 1024

UnityEngine.Debug:Log(Object)
DenkiKitchen.WebRTC.WebSocketSignaler:OnSdpAnswerReadyToSend(SdpMessage) (at Assets/Scripts/WebSocketSignaler.cs:262)
Microsoft.MixedReality.WebRTC.Unity.<>c__DisplayClass8_0:<OnLocalSdpReadyToSend_Listener>b__1() (at Assets/Microsoft.MixedReality.WebRTC.Unity/Scripts/Signaling/Signaler.cs:109)
Microsoft.MixedReality.WebRTC.Unity.Signaler:Update() (at Assets/Microsoft.MixedReality.WebRTC.Unity/Scripts/Signaling/Signaler.cs:130)
DenkiKitchen.WebRTC.WebSocketSignaler:Update() (at Assets/Scripts/WebSocketSignaler.cs:187)
djee-ms commented 4 years ago

First, the warning about PreferredVideoCodec just means that Unity exposes it as a per-track configuration, but in practice the implementation doesn't support that yet, so you cannot have one track forced to VP8 and another forced to H.264, because the implementation uses a global value. In your case with a single track you can safely ignore that warning.

Also I can confirm from those that it's using VP8 since the first codec (96) is VP8. And same in the answer.

m=video 9 UDP/TLS/RTP/SAVPF 96 97 98 99 100 101 127 [...] a=rtpmap:96 VP8/90000

The video is also correctly negotiated for send and receive:

a=sendrecv

So I don't see anything wrong with those SDP messages.

Unfortunately at this point as you confirmed everything works fine with MixedReality-WebRTC but only fails on the third-party library's side, I'm afraid I can't be of much help. I'd suggest to check that library logs if any, or get in touch with their dev team to figure out if there are limitations. Is that library supporting VP8? Another thing to confirm is whether that other library supports the Unified Plan semantic for SDP, which is the default on MixedReality-WebRTC. You can also try switching back to the old Plan B semantic in PeerConnectionConfiguration, see if that helps.