search

microsoft / MixedReality-WebRTC

MixedReality-WebRTC is a collection of components to help mixed reality app developers integrate audio and video real-time communication into their application and improve their collaborative experience
https://microsoft.github.io/MixedReality-WebRTC/
MIT License
903 stars 278 forks source link

No video stream: iOS (web) --> UWP #234

Open cy238 opened 4 years ago

cy238 commented 4 years ago

Hello,

I'm currently trying to get a video/audio connection working with a JS-WebRTC client and a UWP client based on this framework. Everthing works great with Desktop/Android (Chrome), but iOS (Safari) is causing issues. The iOS client doesn't transmit video to the UWP client. Audio and DataChannels work fine in any case. I don't think Safari is the culprit since iOS to Android/Desktop (Chrome) works fine aswell. I'm using iOS 13.3 which should support H264 and VP8, but none of the codecs allowed me to see the iPhone camera's feed on the UWP client.

I'm not sure what causes this. The SDPs seem fine to me.

iOS to UWP offer ``` v=0 o=- 2130917021007381875 2 IN IP4 127.0.0.1 s=- t=0 0 a=group:BUNDLE 0 1 a=msid-semantic: WMS 206daf99-775b-4dfb-84a9-d2313f92796b m=video 9 UDP/TLS/RTP/SAVPF 96 97 98 99 100 101 127 125 104 c=IN IP4 0.0.0.0 a=rtcp:9 IN IP4 0.0.0.0 a=ice-ufrag:a+08 a=ice-pwd:NBuN7VOYxARaW5i6T3/kkeER a=ice-options:trickle a=fingerprint:sha-256 FD:38:63:71:5F:B8:72:BB:AB:A9:F2:32:2A:18:AB:FA:53:6B:76:E3:40:D9:28:A0:81:BC:7F:78:BC:52:72:C6 a=setup:actpass a=mid:0 a=extmap:2 urn:ietf:params:rtp-hdrext:toffset a=extmap:3 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time a=extmap:4 urn:3gpp:video-orientation a=extmap:5 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01 a=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type a=extmap:8 http://www.webrtc.org/experiments/rtp-hdrext/video-timing a=extmap:10 http://tools.ietf.org/html/draft-ietf-avtext-framemarking-07 a=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid a=sendrecv a=msid:206daf99-775b-4dfb-84a9-d2313f92796b e56ed258-c8b6-45fa-86ba-cdd275682e2d a=rtcp-mux a=rtcp-rsize a=rtpmap:96 H264/90000 a=rtcp-fb:96 goog-remb a=rtcp-fb:96 transport-cc a=rtcp-fb:96 ccm fir a=rtcp-fb:96 nack a=rtcp-fb:96 nack pli a=fmtp:96 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=640c1f a=rtpmap:97 rtx/90000 a=fmtp:97 apt=96 a=rtpmap:98 H264/90000 a=rtcp-fb:98 goog-remb a=rtcp-fb:98 transport-cc a=rtcp-fb:98 ccm fir a=rtcp-fb:98 nack a=rtcp-fb:98 nack pli a=fmtp:98 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f a=rtpmap:99 rtx/90000 a=fmtp:99 apt=98 a=rtpmap:100 VP8/90000 a=rtcp-fb:100 goog-remb a=rtcp-fb:100 transport-cc a=rtcp-fb:100 ccm fir a=rtcp-fb:100 nack a=rtcp-fb:100 nack pli a=rtpmap:101 rtx/90000 a=fmtp:101 apt=100 a=rtpmap:127 red/90000 a=rtpmap:125 rtx/90000 a=fmtp:125 apt=127 a=rtpmap:104 ulpfec/90000 a=ssrc-group:FID 3622718431 685751338 a=ssrc:3622718431 cname:8GjrZ2FpV0JhXHHG a=ssrc:3622718431 msid:206daf99-775b-4dfb-84a9-d2313f92796b e56ed258-c8b6-45fa-86ba-cdd275682e2d a=ssrc:3622718431 mslabel:206daf99-775b-4dfb-84a9-d2313f92796b a=ssrc:3622718431 label:e56ed258-c8b6-45fa-86ba-cdd275682e2d a=ssrc:685751338 cname:8GjrZ2FpV0JhXHHG a=ssrc:685751338 msid:206daf99-775b-4dfb-84a9-d2313f92796b e56ed258-c8b6-45fa-86ba-cdd275682e2d a=ssrc:685751338 mslabel:206daf99-775b-4dfb-84a9-d2313f92796b a=ssrc:685751338 label:e56ed258-c8b6-45fa-86ba-cdd275682e2d m=audio 9 UDP/TLS/RTP/SAVPF 111 103 9 102 0 8 105 13 110 113 126 c=IN IP4 0.0.0.0 a=rtcp:9 IN IP4 0.0.0.0 a=ice-ufrag:a+08 a=ice-pwd:NBuN7VOYxARaW5i6T3/kkeER a=ice-options:trickle a=fingerprint:sha-256 FD:38:63:71:5F:B8:72:BB:AB:A9:F2:32:2A:18:AB:FA:53:6B:76:E3:40:D9:28:A0:81:BC:7F:78:BC:52:72:C6 a=setup:actpass a=mid:1 a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level a=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid a=sendrecv a=msid:206daf99-775b-4dfb-84a9-d2313f92796b 5cf1df1f-6d03-449a-b232-eaab495ee76d a=rtcp-mux a=rtpmap:111 opus/48000/2 a=rtcp-fb:111 transport-cc a=fmtp:111 minptime=10;useinbandfec=1 a=rtpmap:103 ISAC/16000 a=rtpmap:9 G722/8000 a=rtpmap:102 ILBC/8000 a=rtpmap:0 PCMU/8000 a=rtpmap:8 PCMA/8000 a=rtpmap:105 CN/16000 a=rtpmap:13 CN/8000 a=rtpmap:110 telephone-event/48000 a=rtpmap:113 telephone-event/16000 a=rtpmap:126 telephone-event/8000 a=ssrc:650121085 cname:8GjrZ2FpV0JhXHHG a=ssrc:650121085 msid:206daf99-775b-4dfb-84a9-d2313f92796b 5cf1df1f-6d03-449a-b232-eaab495ee76d a=ssrc:650121085 mslabel:206daf99-775b-4dfb-84a9-d2313f92796b a=ssrc:650121085 label:5cf1df1f-6d03-449a-b232-eaab495ee76d ```
UWP to iOS answer ``` v=0 o=- 8093639752169347905 2 IN IP4 127.0.0.1 s=- t=0 0 a=group:BUNDLE 0 1 a=msid-semantic: WMS local_av_stream m=video 9 UDP/TLS/RTP/SAVPF 98 99 100 101 127 125 104 c=IN IP4 0.0.0.0 a=rtcp:9 IN IP4 0.0.0.0 a=ice-ufrag:C0pe a=ice-pwd:Fe4P7LDJZobD53TM/epY2/ol a=ice-options:trickle a=fingerprint:sha-256 94:B7:0E:5C:25:20:82:89:E0:E7:59:FA:F0:EF:85:85:4F:F9:B7:AE:07:C5:26:DF:A4:10:95:04:57:19:0B:1D a=setup:active a=mid:0 a=extmap:2 urn:ietf:params:rtp-hdrext:toffset a=extmap:3 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time a=extmap:4 urn:3gpp:video-orientation a=extmap:5 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01 a=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type a=extmap:8 http://www.webrtc.org/experiments/rtp-hdrext/video-timing a=extmap:10 http://tools.ietf.org/html/draft-ietf-avtext-framemarking-07 a=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid a=sendrecv a=msid:local_av_stream c4579f0c-f3ef-47e9-8711-8ba66abd9a05 a=rtcp-mux a=rtcp-rsize a=rtpmap:98 H264/90000 a=rtcp-fb:98 goog-remb a=rtcp-fb:98 transport-cc a=rtcp-fb:98 ccm fir a=rtcp-fb:98 nack a=rtcp-fb:98 nack pli a=fmtp:98 packetization-mode=1;profile-level-id=42e01f a=rtpmap:99 rtx/90000 a=fmtp:99 apt=98 a=rtpmap:100 VP8/90000 a=rtcp-fb:100 goog-remb a=rtcp-fb:100 transport-cc a=rtcp-fb:100 ccm fir a=rtcp-fb:100 nack a=rtcp-fb:100 nack pli a=rtpmap:101 rtx/90000 a=fmtp:101 apt=100 a=rtpmap:127 red/90000 a=rtpmap:125 rtx/90000 a=fmtp:125 apt=127 a=rtpmap:104 ulpfec/90000 a=ssrc-group:FID 474118709 1724676474 a=ssrc:474118709 cname:qyqxjRQ+czmO3fOE a=ssrc:1724676474 cname:qyqxjRQ+czmO3fOE m=audio 9 UDP/TLS/RTP/SAVPF 111 103 9 102 0 8 105 13 110 113 126 c=IN IP4 0.0.0.0 a=rtcp:9 IN IP4 0.0.0.0 a=ice-ufrag:C0pe a=ice-pwd:Fe4P7LDJZobD53TM/epY2/ol a=ice-options:trickle a=fingerprint:sha-256 94:B7:0E:5C:25:20:82:89:E0:E7:59:FA:F0:EF:85:85:4F:F9:B7:AE:07:C5:26:DF:A4:10:95:04:57:19:0B:1D a=setup:active a=mid:1 a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level a=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid a=sendrecv a=msid:local_av_stream local_audio a=rtcp-mux a=rtpmap:111 opus/48000/2 a=rtcp-fb:111 transport-cc a=fmtp:111 minptime=10;useinbandfec=1 a=rtpmap:103 ISAC/16000 a=rtpmap:9 G722/8000 a=rtpmap:102 ILBC/8000 a=rtpmap:0 PCMU/8000 a=rtpmap:8 PCMA/8000 a=rtpmap:105 CN/16000 a=rtpmap:13 CN/8000 a=rtpmap:110 telephone-event/48000 a=rtpmap:113 telephone-event/16000 a=rtpmap:126 telephone-event/8000 a=ssrc:3234309454 cname:qyqxjRQ+czmO3fOE ```

For reference, the Android SDPs (this works):

Android to UWP offer ``` v=0 o=- 3012905752719936482 4 IN IP4 127.0.0.1 s=- t=0 0 a=group:BUNDLE 0 1 2 a=msid-semantic: WMS XJOLDN5aqODqAA82svjpBknGEtiCxsWIPuem m=audio 58163 UDP/TLS/RTP/SAVPF 111 103 9 0 8 105 13 110 113 126 c=IN IP4 159.89.29.128 a=rtcp:9 IN IP4 0.0.0.0 a=candidate:2641241375 1 udp 2122262783 2003:ee:bbf5:8b7:21b7:5cf6:b93a:4dda 37991 typ host generation 0 network-id 2 network-cost 10 a=candidate:2338487986 1 udp 2122194687 192.168.3.144 47843 typ host generation 0 network-id 1 network-cost 10 a=candidate:3555580399 1 tcp 1518283007 2003:ee:bbf5:8b7:21b7:5cf6:b93a:4dda 9 typ host tcptype active generation 0 network-id 2 network-cost 10 a=candidate:3319890498 1 tcp 1518214911 192.168.3.144 9 typ host tcptype active generation 0 network-id 1 network-cost 10 a=candidate:170555398 1 udp 1685987071 46.92.238.30 47843 typ srflx raddr 192.168.3.144 rport 47843 generation 0 network-id 1 network-cost 10 a=candidate:3096583213 1 udp 41820927 159.89.29.128 58163 typ relay raddr 46.92.238.30 rport 47843 generation 0 network-id 1 network-cost 10 a=candidate:4128366813 1 udp 25043199 159.89.29.128 56241 typ relay raddr 46.92.238.30 rport 45915 generation 0 network-id 1 network-cost 10 a=candidate:3900747176 1 udp 8265471 159.89.29.128 51013 typ relay raddr 46.92.238.30 rport 38994 generation 0 network-id 1 network-cost 10 a=candidate:4128366813 1 udp 25043455 159.89.29.128 51274 typ relay raddr 46.92.238.30 rport 39354 generation 0 network-id 1 network-cost 10 a=candidate:3096583213 1 udp 41821183 159.89.29.128 50938 typ relay raddr 46.92.238.30 rport 47843 generation 0 network-id 1 network-cost 10 a=candidate:3900747176 1 udp 8265727 159.89.29.128 61476 typ relay raddr 46.92.238.30 rport 46251 generation 0 network-id 1 network-cost 10 a=ice-ufrag:EnNU a=ice-pwd:EMYfYpePgj6Bs0YxsRanHF5f a=ice-options:trickle a=fingerprint:sha-256 92:79:E2:E7:C5:04:96:DD:E4:D3:78:A7:6C:BA:18:82:C4:77:A0:6B:10:38:B1:23:DD:28:A5:3F:3E:F6:31:57 a=setup:actpass a=mid:0 a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level a=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01 a=extmap:5 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id a=extmap:6 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id a=sendrecv a=msid:XJOLDN5aqODqAA82svjpBknGEtiCxsWIPuem a25c17a2-f87b-4da7-8645-b94da1206fb1 a=rtcp-mux a=rtpmap:111 opus/48000/2 a=rtcp-fb:111 transport-cc a=fmtp:111 minptime=10;useinbandfec=1 a=rtpmap:103 ISAC/16000 a=rtpmap:9 G722/8000 a=rtpmap:0 PCMU/8000 a=rtpmap:8 PCMA/8000 a=rtpmap:105 CN/16000 a=rtpmap:13 CN/8000 a=rtpmap:110 telephone-event/48000 a=rtpmap:113 telephone-event/16000 a=rtpmap:126 telephone-event/8000 a=ssrc:1895065870 cname:HyCP1rlrSvofPSjC a=ssrc:1895065870 msid:XJOLDN5aqODqAA82svjpBknGEtiCxsWIPuem a25c17a2-f87b-4da7-8645-b94da1206fb1 a=ssrc:1895065870 mslabel:XJOLDN5aqODqAA82svjpBknGEtiCxsWIPuem a=ssrc:1895065870 label:a25c17a2-f87b-4da7-8645-b94da1206fb1 m=video 51346 UDP/TLS/RTP/SAVPF 96 97 98 99 100 101 102 127 104 c=IN IP4 159.89.29.128 a=rtcp:9 IN IP4 0.0.0.0 a=candidate:2641241375 1 udp 2122262783 2003:ee:bbf5:8b7:21b7:5cf6:b93a:4dda 41720 typ host generation 0 network-id 2 network-cost 10 a=candidate:2338487986 1 udp 2122194687 192.168.3.144 45131 typ host generation 0 network-id 1 network-cost 10 a=candidate:3555580399 1 tcp 1518283007 2003:ee:bbf5:8b7:21b7:5cf6:b93a:4dda 9 typ host tcptype active generation 0 network-id 2 network-cost 10 a=candidate:3319890498 1 tcp 1518214911 192.168.3.144 9 typ host tcptype active generation 0 network-id 1 network-cost 10 a=candidate:170555398 1 udp 1685987071 46.92.238.30 45131 typ srflx raddr 192.168.3.144 rport 45131 generation 0 network-id 1 network-cost 10 a=candidate:4128366813 1 udp 25043455 159.89.29.128 51346 typ relay raddr 46.92.238.30 rport 39358 generation 0 network-id 1 network-cost 10 a=candidate:3096583213 1 udp 41820927 159.89.29.128 62252 typ relay raddr 46.92.238.30 rport 45131 generation 0 network-id 1 network-cost 10 a=candidate:3096583213 1 udp 41821183 159.89.29.128 59694 typ relay raddr 46.92.238.30 rport 45131 generation 0 network-id 1 network-cost 10 a=candidate:4128366813 1 udp 25043199 159.89.29.128 64855 typ relay raddr 46.92.238.30 rport 45919 generation 0 network-id 1 network-cost 10 a=candidate:3900747176 1 udp 8265727 159.89.29.128 55293 typ relay raddr 46.92.238.30 rport 46255 generation 0 network-id 1 network-cost 10 a=candidate:3900747176 1 udp 8265471 159.89.29.128 61153 typ relay raddr 46.92.238.30 rport 38998 generation 0 network-id 1 network-cost 10 a=ice-ufrag:EnNU a=ice-pwd:EMYfYpePgj6Bs0YxsRanHF5f a=ice-options:trickle a=fingerprint:sha-256 92:79:E2:E7:C5:04:96:DD:E4:D3:78:A7:6C:BA:18:82:C4:77:A0:6B:10:38:B1:23:DD:28:A5:3F:3E:F6:31:57 a=setup:actpass a=mid:1 a=extmap:14 urn:ietf:params:rtp-hdrext:toffset a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time a=extmap:13 urn:3gpp:video-orientation a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01 a=extmap:12 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay a=extmap:11 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-timing a=extmap:8 http://tools.ietf.org/html/draft-ietf-avtext-framemarking-07 a=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid a=extmap:9 http://www.webrtc.org/experiments/rtp-hdrext/color-space a=extmap:5 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id a=extmap:6 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id a=sendrecv a=msid:XJOLDN5aqODqAA82svjpBknGEtiCxsWIPuem 40c8833d-b376-4a45-b74b-8f3f975f9991 a=rtcp-mux a=rtcp-rsize a=rtpmap:96 VP8/90000 a=rtcp-fb:96 goog-remb a=rtcp-fb:96 transport-cc a=rtcp-fb:96 ccm fir a=rtcp-fb:96 nack a=rtcp-fb:96 nack pli a=rtpmap:97 rtx/90000 a=fmtp:97 apt=96 a=rtpmap:98 VP9/90000 a=rtcp-fb:98 goog-remb a=rtcp-fb:98 transport-cc a=rtcp-fb:98 ccm fir a=rtcp-fb:98 nack a=rtcp-fb:98 nack pli a=fmtp:98 profile-id=0 a=rtpmap:99 rtx/90000 a=fmtp:99 apt=98 a=rtpmap:100 H264/90000 a=rtcp-fb:100 goog-remb a=rtcp-fb:100 transport-cc a=rtcp-fb:100 ccm fir a=rtcp-fb:100 nack a=rtcp-fb:100 nack pli a=fmtp:100 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f a=rtpmap:101 rtx/90000 a=fmtp:101 apt=100 a=rtpmap:102 red/90000 a=rtpmap:127 rtx/90000 a=fmtp:127 apt=102 a=rtpmap:104 ulpfec/90000 a=ssrc-group:FID 1430957368 799624132 a=ssrc:1430957368 cname:HyCP1rlrSvofPSjC a=ssrc:1430957368 msid:XJOLDN5aqODqAA82svjpBknGEtiCxsWIPuem 40c8833d-b376-4a45-b74b-8f3f975f9991 a=ssrc:1430957368 mslabel:XJOLDN5aqODqAA82svjpBknGEtiCxsWIPuem a=ssrc:1430957368 label:40c8833d-b376-4a45-b74b-8f3f975f9991 a=ssrc:799624132 cname:HyCP1rlrSvofPSjC a=ssrc:799624132 msid:XJOLDN5aqODqAA82svjpBknGEtiCxsWIPuem 40c8833d-b376-4a45-b74b-8f3f975f9991 a=ssrc:799624132 mslabel:XJOLDN5aqODqAA82svjpBknGEtiCxsWIPuem a=ssrc:799624132 label:40c8833d-b376-4a45-b74b-8f3f975f9991 m=application 9 UDP/DTLS/SCTP webrtc-datachannel c=IN IP4 0.0.0.0 a=ice-ufrag:EnNU a=ice-pwd:EMYfYpePgj6Bs0YxsRanHF5f a=ice-options:trickle a=fingerprint:sha-256 92:79:E2:E7:C5:04:96:DD:E4:D3:78:A7:6C:BA:18:82:C4:77:A0:6B:10:38:B1:23:DD:28:A5:3F:3E:F6:31:57 a=setup:actpass a=mid:2 a=sctp-port:5000 a=max-message-size:262144 ```
UWP to Android answer ``` v=0 o=- 3381193782361828673 2 IN IP4 127.0.0.1 s=- t=0 0 a=group:BUNDLE 0 1 2 a=msid-semantic: WMS local_av_stream m=audio 9 UDP/TLS/RTP/SAVPF 111 103 9 0 8 105 13 110 113 126 c=IN IP4 0.0.0.0 a=rtcp:9 IN IP4 0.0.0.0 a=ice-ufrag:bb1y a=ice-pwd:ssmghrtMjjvsSTi2z0MivxfJ a=ice-options:trickle a=fingerprint:sha-256 25:F4:90:AF:B8:3E:2D:A0:08:0D:FF:74:D4:C7:71:3F:45:3C:C3:D5:19:A1:10:6E:8C:6C:8A:4B:85:53:66:DA a=setup:active a=mid:0 a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level a=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid a=sendrecv a=msid:local_av_stream local_audio a=rtcp-mux a=rtpmap:111 opus/48000/2 a=rtcp-fb:111 transport-cc a=fmtp:111 minptime=10;useinbandfec=1 a=rtpmap:103 ISAC/16000 a=rtpmap:9 G722/8000 a=rtpmap:0 PCMU/8000 a=rtpmap:8 PCMA/8000 a=rtpmap:105 CN/16000 a=rtpmap:13 CN/8000 a=rtpmap:110 telephone-event/48000 a=rtpmap:113 telephone-event/16000 a=rtpmap:126 telephone-event/8000 a=ssrc:3940255725 cname:Jsp5MGXG+8GDJZTO m=video 9 UDP/TLS/RTP/SAVPF 96 97 98 99 100 101 102 127 104 c=IN IP4 0.0.0.0 a=rtcp:9 IN IP4 0.0.0.0 a=ice-ufrag:bb1y a=ice-pwd:ssmghrtMjjvsSTi2z0MivxfJ a=ice-options:trickle a=fingerprint:sha-256 25:F4:90:AF:B8:3E:2D:A0:08:0D:FF:74:D4:C7:71:3F:45:3C:C3:D5:19:A1:10:6E:8C:6C:8A:4B:85:53:66:DA a=setup:active a=mid:1 a=extmap:14 urn:ietf:params:rtp-hdrext:toffset a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time a=extmap:13 urn:3gpp:video-orientation a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01 a=extmap:12 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay a=extmap:11 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-timing a=extmap:8 http://tools.ietf.org/html/draft-ietf-avtext-framemarking-07 a=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid a=sendrecv a=msid:local_av_stream 097a4733-ad2c-433a-868d-533911e22fa8 a=rtcp-mux a=rtcp-rsize a=rtpmap:96 VP8/90000 a=rtcp-fb:96 goog-remb a=rtcp-fb:96 transport-cc a=rtcp-fb:96 ccm fir a=rtcp-fb:96 nack a=rtcp-fb:96 nack pli a=rtpmap:97 rtx/90000 a=fmtp:97 apt=96 a=rtpmap:98 VP9/90000 a=rtcp-fb:98 goog-remb a=rtcp-fb:98 transport-cc a=rtcp-fb:98 ccm fir a=rtcp-fb:98 nack a=rtcp-fb:98 nack pli a=fmtp:98 x-google-profile-id=0 a=rtpmap:99 rtx/90000 a=fmtp:99 apt=98 a=rtpmap:100 H264/90000 a=rtcp-fb:100 goog-remb a=rtcp-fb:100 transport-cc a=rtcp-fb:100 ccm fir a=rtcp-fb:100 nack a=rtcp-fb:100 nack pli a=fmtp:100 packetization-mode=1;profile-level-id=42e01f a=rtpmap:101 rtx/90000 a=fmtp:101 apt=100 a=rtpmap:102 red/90000 a=rtpmap:127 rtx/90000 a=fmtp:127 apt=102 a=rtpmap:104 ulpfec/90000 a=ssrc-group:FID 1176804677 1504878379 a=ssrc:1176804677 cname:Jsp5MGXG+8GDJZTO a=ssrc:1504878379 cname:Jsp5MGXG+8GDJZTO m=application 9 UDP/DTLS/SCTP webrtc-datachannel c=IN IP4 0.0.0.0 b=AS:30 a=ice-ufrag:bb1y a=ice-pwd:ssmghrtMjjvsSTi2z0MivxfJ a=ice-options:trickle a=fingerprint:sha-256 25:F4:90:AF:B8:3E:2D:A0:08:0D:FF:74:D4:C7:71:3F:45:3C:C3:D5:19:A1:10:6E:8C:6C:8A:4B:85:53:66:DA a=setup:active a=mid:2 a=sctp-port:5000 ```

The only real difference is (afaik) the VP9 codec. But even if I force H264 or VP8 by setting _nativePeer.PreferredVideoCodec the behaviour stays the same. In a call with an iOS client, the VideoFrameReady events never fire. TrackAdded is called for both audio and video though.

I'm using a build based on master branch back from february, but I also tried the most recent master commit today, but that didn't seem to change anything.

Switching the caller/callee around so the UWP client sends the offers didn't change anything either.

There's an issue on the uwp-webrtc repo describing a similar problem (https://github.com/webrtc-uwp/webrtc-uwp-sdk/issues/190). It was solved by using 2 separate connections per client, but I'm not too keen on completely changing my working setup... unless I really have to.

Any ideas what might be wrong here? Anyone ever got a Safari <-> UWP WebRTC connection running?

djee-ms commented 4 years ago

Hi @cy238,

We don't officially support iOS (yet) so I have little experience with it.

Looking at the offer/answer messages, they all contain a=sendrecv so this looks all good. The only thing I see is that Android/UWP negotiates VP8, while iOS/UWP negotiates H.264. But using PreferredVideoCodec should change that as you mentioned. Can you just confirm that when using PreferredVideoCodec="VP8" then the offer/answer SDP messages only contain VP8 please? To be sure the setting is actually taken into account. But I see VP8 in the iOS offer so I'm pretty sure it's supported.

Beyond that, iOS/UWP is negotiating a particular H.264 profile (profile-level-id=42e01f; Constrained Baseline according to Table 5 of the RTP profile RFC), but that seems fine. The only difference seems to be that the iOS offer contains level-asymmetry-allowed=1 but the UWP answer doesn't; no idea if that has any effect though.

This sounds like a compatibility issue, and we never tested that with iOS, so I'm afraid the only way to figure out would be to look at the webrtc.lib logs in the Output window of Visual Studio. When you launch your app with the debugger attached, in Debug config (only) webrtc.lib should spew a ton of logs about what it's doing, and hopefully we can spot an error there.

cy238 commented 4 years ago

Hi @djee-ms,

thanks for your answer. It took me a while to get back to this topic, sorry.

Setting PreferredVideoCodec does indeed change the SDPs to include only the selected codec.

I made another set of tests in Debug config. Here are the logs: Log: UWP -> Android Log: UWP -> iOS (unfortunately thanks to multithreading you can't get a really clean diff between the two files, but that's the cleanest I could manage)

Testing was done on a local network, so all the STUN/TURN related stuff can probably be ignored. The connection was established in both cases. Everything but the iOS video worked perfectly.

I'm very unfamiliar with the webtc.lib debug output, so I'm not quite sure what to look for. At first glance I couldn't find anything that'd help me figure out what's going wrong with the iOS video...

djee-ms commented 4 years ago

Not sure what's happening. To summarize what we know:

It seems something goes wrong between adding the track and dispatching the video frame callback, either because there's no video data received (firewall?) or because there's a bug that prevent raising the event.

Can you please try to capture a trace with Windows Performance Recorder (download link is first link on that page) and the profile Microsoft.MixedReality.WebRTC.wprp found in tools/tracing/? This might capture something that we don't see in the logs. I have no other idea for now, sorry.

cy238 commented 4 years ago

Quick update on this: I just added some stat tracking to the JS-WebRTC side (using RTCPeerConnection.getStats()) and noticed that video frames get transmitted correctly. The UWP client also receives data (according to Windows' resmon.exe). Looks like it's just a matter of handling the frames on UWP's end.

I'll dig deeper and use the performance recorder tomorrow/on friday.

cy238 commented 4 years ago

As expected PeerConnection.Peer.I420ARemoteVideoFrameReady is never called eventhough the iOS side sends frames. Not quite sure where exactly the data is received in the lib.

Also I didn't have much success using Windows Performance Recorder. I used debug builds of the lib (not sure if that's even necessary) but there doesn't seem to be any WebRTC related logging.

Did I mess up somewhere in the WMR settings?

wpr_wpa

djee-ms commented 4 years ago

00AEE89E-B531-4F20-A2C5-D02F37CB6AA1 is the WebRTC profile; I have yet to understand why sometimes WPA doesn't find its name and instead use the GUID, sorry about that bug. What event can you observe from that series? I see some purple bar, but I can't see the details.

cy238 commented 4 years ago

Well... uhm...

wpa_events

I can upload the full log if you want, but there doesn't seem to be anything in there...

djee-ms commented 4 years ago

I think you're missing the PDBs so even with the trace profile it's not enough for WPA to display anything meaningful unfortunately.

I've re-read in details through the entire UWP -> iOS log, and some things look strange:

So it is strange but the implementation definitely receives and decodes some video frames. I have no idea why the callback is not called though, if those frames are indeed received.

Note that the UWP -> Android also has the same double session negotiation. It would be interesting I think to understand why this happens, because there's always a possibility that we have a bug and don't handle correctly this extremely quick change of sessions.

cy238 commented 4 years ago

Hi @djee-ms,

thanks for the analysis of those huge logs.

The double negotiation is due to a quirk in our signalling logic. Both clients just keep sending offers (and answers to them) until they're fully connected, in case an sdp message gets dropped. Obviously not the cleanest way, but it doesn't seem to break anything. I disabled that part for my testing now though. The iOS video still does not show up on UWP, eventhough there's no longer a quick stream restart. I couldn't find any big differences in the logs (except for the missing 2nd negotiation), but in case it helps: Log: UWP -> Android Log: UWP->iOS

As for the WPA output: I couldn't get any visible data even with the PDB files included. I enabled Copy PDB files, Development Build and Script Debugging in the Unity build dialog and ran the VS solution in Debug configuration. I also confirmed that the PDB files do actually get copied to the VS project. Any idea where I messed up?

KarthikRichie commented 3 years ago

Hello cy238,

I'm facing a similar issue as yours. Were you able to fix this ?