Open SergioMasson opened 4 years ago
SergioMasson
commented
4 years ago Do you guys think it could be some settings that we are doing wrong? Nothing has changed in the our web application from when we were using Mixed reality WebRTC 1.0.3. It there soming we have to change in order to call version 2.0?
arufolo-wayahealth
commented
4 years ago +1 We are experiencing the same exact issue. Have you made any progress on solving the problem?
We are receiving the ontrack() call back in the web browser, but the "streams" property on the event object is an empty array. I can also confirm this was properly working on version 1.
I need to use version 2.0+ of this repository because we need the scene camera streaming functionality that was introduced.
jmorgan-wellovate
commented
4 years ago Experiencing the same issue as well. Any help here whatsoever would be appreciated!
Marcus757
commented
4 years ago I am running into the same problem. Hopefully a fix will be provided soon.
SergioMasson
commented
4 years ago After taking a look to the offer and answer messages sent from Mixed reality WebRTC, I have found differences on the "msid-semantic" attribute. When using version 1.0.3 this attribute was "a=msid-semantic: WMS local_av_stream". However, on 2.0 the local_av_stream is gone "a=msid-semantic: WMS". I`m not an expert in WebRTC, but it seam that this attribute should be optional and it is only a group identifier. By simply placing it manually on the "offer" and "answer" messages I got it to work temporarily on version 2.0 using the code bellow:
private string EnsureSemanticGrouping(string sdpMessage)
{
//Checks if the msid semantic group has been set in the message.
if (!sdpMessage.Contains("msid-semantic: WMS\r"))
return sdpMessage;
return sdpMessage.Replace("msid-semantic: WMS\r", "msid-semantic: WMS local_av_stream\r").Replace("msid:-", "msid:-local_av_stream");
}Thank you @SergioMasson this worked for chrome! This helps tremendously and should be introduced in a future PR
travelhawk
commented
3 years ago That worked for me, too. Thank you!! It's just sad that it took me so long to find the problem and this issue seems quite big and is not addressed yet....
jmreyesgarcia
commented
2 years ago Hi there,
We are also developing an app in order to show video feed and audio from a Hololens 2 in a browser and we are still struggling with the signalling. It seems that there are some diferences in the SDP offer/answer protocol between de MixedReality-WebRTC SDK and the Mozilla WebRTC API. Right now we are using this very simple signalling server node-dss, and using the simpler WebRTC for PC before trying again with the Hololens 2 client. I'm borrowing your nice format as a template @SergioMasson, I think it is very easy to understand.
Expected behavior
The call should have video and audio in both directions.
Actual behavior
The peers don't connect. Unity client shows:
Cannot apply remote description: Values does not fall within the expected range.
SDP messages
SDP offer message sent from Unity editor to browser, and answer message sent from the browser to Unity.
Environment
Peer 1:
Peer 2:
Additional context
For signalling we are using node-dss. When performing a call from Unity editor to another Unity editor/PC Compilation/UWP Compilation everything works fine. Also when we try from a web client to the web client (either Firefox or Chrome)
It seems that we are having trouble trying to communicate Unity 3D PC/UWP client and web. ¿Could you give us any clue to follow? We just applied the function you developed @SergioMasson to the Unity Client SDP offer/answer:
private string EnsureSemanticGrouping(string sdpMessage)
{
//Checks if the msid semantic group has been set in the message.
if (!sdpMessage.Contains("msid-semantic: WMS\r"))
return sdpMessage;
return sdpMessage.Replace("msid-semantic: WMS\r", "msid-semantic: WMS local_av_stream\r").Replace("msid:-", "msid:-local_av_stream");
}and the browser peer and Unity3D peer are not stablishing a connection yet.
The main error we are receiving is this:
Cannot apply remote description: Value does not fall within the expected range.
¿Would you have any clue, or a path to to keep investigating in order to figure it out?
Davidwang007
commented
2 years ago Have you solved it yet ? I have the same problem,please show me the right direction,thank you.
Describe the bug I`m working on a project that uses WebRTC to promote video calls between Hololens and web browser users. Calls with video and audio were working fine on version 1.0.3. However, we are currently working on upgrading the projeto to version 2.0 and for some reason the web browser (both chrome and Firefox) fail to receive media from Unity/Hololens.
Doing some debugging we have noticed that the rtcPeerConnection.ontrack event fired on the browser after receiving the offer from unity prints the stream object as "undefined" when using MixedReality-WebRTC 2.0. When using version 1.0.3 that was printed as [object MediaStream].
Expected behavior The call should have video and audio in both directions.
SDP messages SDP offer message sent from Unity editor to chrome, and answer message sent from chrome to Unity.
Offer message
v=0\r\no=- 1044938697899636485 2 IN IP4 127.0.0.1\r\ns=-\r\nt=0 0\r\na=group:BUNDLE 0 1\r\na=msid-semantic: WMS\r\nm=audio 9 UDP/TLS/RTP/SAVPF 111 103 104 9 102 0 8 106 105 13 110 112 113 126\r\nc=IN IP4 0.0.0.0\r\na=rtcp:9 IN IP4 0.0.0.0\r\na=ice-ufrag:FRM4\r\na=ice-pwd:vJTJFaMmV4vjUAA1fIlEOaZN\r\na=ice-options:trickle\r\na=fingerprint:sha-256 E3:A6:A7:58:F1:95:B0:74:DC:CF:33:2B:AF:D2:F7:2A:E7:2E:3E:22:F4:6F:88:9C:0E:49:F9:55:F4:2F:79:40\r\na=setup:actpass\r\na=mid:0\r\na=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level\r\na=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid\r\na=sendrecv\r\na=msid:- 73edb09e-86b4-491d-8aa1-344b207901c7\r\na=rtcp-mux\r\na=rtpmap:111 opus/48000/2\r\na=rtcp-fb:111 transport-cc\r\na=fmtp:111 minptime=10;useinbandfec=1\r\na=rtpmap:103 ISAC/16000\r\na=rtpmap:104 ISAC/32000\r\na=rtpmap:9 G722/8000\r\na=rtpmap:102 ILBC/8000\r\na=rtpmap:0 PCMU/8000\r\na=rtpmap:8 PCMA/8000\r\na=rtpmap:106 CN/32000\r\na=rtpmap:105 CN/16000\r\na=rtpmap:13 CN/8000\r\na=rtpmap:110 telephone-event/48000\r\na=rtpmap:112 telephone-event/32000\r\na=rtpmap:113 telephone-event/16000\r\na=rtpmap:126 telephone-event/8000\r\na=ssrc:1436316144 cname:qZUxAzmV3LsJMQx1\r\na=ssrc:1436316144 msid: 73edb09e-86b4-491d-8aa1-344b207901c7\r\na=ssrc:1436316144 mslabel:\r\na=ssrc:1436316144 label:73edb09e-86b4-491d-8aa1-344b207901c7\r\nm=video 9 UDP/TLS/RTP/SAVPF 96 97 98 99 100 101 127 124 125\r\nc=IN IP4 0.0.0.0\r\na=rtcp:9 IN IP4 0.0.0.0\r\na=ice-ufrag:FRM4\r\na=ice-pwd:vJTJFaMmV4vjUAA1fIlEOaZN\r\na=ice-options:trickle\r\na=fingerprint:sha-256 E3:A6:A7:58:F1:95:B0:74:DC:CF:33:2B:AF:D2:F7:2A:E7:2E:3E:22:F4:6F:88:9C:0E:49:F9:55:F4:2F:79:40\r\na=setup:actpass\r\na=mid:1\r\na=extmap:2 urn:ietf:params:rtp-hdrext:toffset\r\na=extmap:3 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time\r\na=extmap:4 urn:3gpp:video-orientation\r\na=extmap:5 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01\r\na=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay\r\na=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type\r\na=extmap:8 http://www.webrtc.org/experiments/rtp-hdrext/video-timing\r\na=extmap:10 http://tools.ietf.org/html/draft-ietf-avtext-framemarking-07\r\na=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid\r\na=sendrecv\r\na=msid:- 5f3093ea-cd98-4009-8107-8e1c03dd90d4\r\na=rtcp-mux\r\na=rtcp-rsize\r\na=rtpmap:96 VP8/90000\r\na=rtcp-fb:96 goog-remb\r\na=rtcp-fb:96 transport-cc\r\na=rtcp-fb:96 ccm fir\r\na=rtcp-fb:96 nack\r\na=rtcp-fb:96 nack pli\r\na=rtpmap:97 rtx/90000\r\na=fmtp:97 apt=96\r\na=rtpmap:98 VP9/90000\r\na=rtcp-fb:98 goog-remb\r\na=rtcp-fb:98 transport-cc\r\na=rtcp-fb:98 ccm fir\r\na=rtcp-fb:98 nack\r\na=rtcp-fb:98 nack pli\r\na=fmtp:98 x-google-profile-id=0\r\na=rtpmap:99 rtx/90000\r\na=fmtp:99 apt=98\r\na=rtpmap:100 multiplex/90000\r\na=rtcp-fb:100 goog-remb\r\na=rtcp-fb:100 transport-cc\r\na=rtcp-fb:100 ccm fir\r\na=rtcp-fb:100 nack\r\na=rtcp-fb:100 nack pli\r\na=fmtp:100 acn=VP9;x-google-profile-id=0\r\na=rtpmap:101 rtx/90000\r\na=fmtp:101 apt=100\r\na=rtpmap:127 red/90000\r\na=rtpmap:124 rtx/90000\r\na=fmtp:124 apt=127\r\na=rtpmap:125 ulpfec/90000\r\na=ssrc-group:FID 796213528 1744200706\r\na=ssrc:796213528 cname:qZUxAzmV3LsJMQx1\r\na=ssrc:796213528 msid: 5f3093ea-cd98-4009-8107-8e1c03dd90d4\r\na=ssrc:796213528 mslabel:\r\na=ssrc:796213528 label:5f3093ea-cd98-4009-8107-8e1c03dd90d4\r\na=ssrc:1744200706 cname:qZUxAzmV3LsJMQx1\r\na=ssrc:1744200706 msid: 5f3093ea-cd98-4009-8107-8e1c03dd90d4\r\na=ssrc:1744200706 mslabel:\r\na=ssrc:1744200706 label:5f3093ea-cd98-4009-8107-8e1c03dd90d4\r\nAnswer message
v=0\r\no=- 3646397425678593421 2 IN IP4 127.0.0.1\r\ns=-\r\nt=0 0\r\na=group:BUNDLE 0 1\r\na=msid-semantic: WMS CTOoJEsouey3Vg1GBRoUf7DmZCe4RjSOnwJT\r\nm=audio 9 UDP/TLS/RTP/SAVPF 111 103 104 9 0 8 106 105 13 110 112 113 126\r\nc=IN IP4 0.0.0.0\r\na=rtcp:9 IN IP4 0.0.0.0\r\na=ice-ufrag:feAL\r\na=ice-pwd:Tb1P3DbijT5tQ3t7aU0tIPXF\r\na=ice-options:trickle\r\na=fingerprint:sha-256 33:36:77:DD:95:DB:65:32:82:20:87:2D:F3:E2:8F:98:7C:97:A4:41:5E:88:A7:AC:80:E0:D6:D0:99:24:67:0E\r\na=setup:active\r\na=mid:0\r\na=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level\r\na=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid\r\na=sendrecv\r\na=msid:CTOoJEsouey3Vg1GBRoUf7DmZCe4RjSOnwJT c04f73c5-1629-4642-bd3d-58a0070b5307\r\na=rtcp-mux\r\na=rtpmap:111 opus/48000/2\r\na=rtcp-fb:111 transport-cc\r\na=fmtp:111 minptime=10;useinbandfec=1\r\na=rtpmap:103 ISAC/16000\r\na=rtpmap:104 ISAC/32000\r\na=rtpmap:9 G722/8000\r\na=rtpmap:0 PCMU/8000\r\na=rtpmap:8 PCMA/8000\r\na=rtpmap:106 CN/32000\r\na=rtpmap:105 CN/16000\r\na=rtpmap:13 CN/8000\r\na=rtpmap:110 telephone-event/48000\r\na=rtpmap:112 telephone-event/32000\r\na=rtpmap:113 telephone-event/16000\r\na=rtpmap:126 telephone-event/8000\r\na=ssrc:3702220422 cname:sTeiwPcf9UjxBLfU\r\nm=video 9 UDP/TLS/RTP/SAVPF 96 97 98 99 127 124 125\r\nc=IN IP4 0.0.0.0\r\na=rtcp:9 IN IP4 0.0.0.0\r\na=ice-ufrag:feAL\r\na=ice-pwd:Tb1P3DbijT5tQ3t7aU0tIPXF\r\na=ice-options:trickle\r\na=fingerprint:sha-256 33:36:77:DD:95:DB:65:32:82:20:87:2D:F3:E2:8F:98:7C:97:A4:41:5E:88:A7:AC:80:E0:D6:D0:99:24:67:0E\r\na=setup:active\r\na=mid:1\r\na=extmap:2 urn:ietf:params:rtp-hdrext:toffset\r\na=extmap:3 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time\r\na=extmap:4 urn:3gpp:video-orientation\r\na=extmap:5 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01\r\na=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay\r\na=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type\r\na=extmap:8 http://www.webrtc.org/experiments/rtp-hdrext/video-timing\r\na=extmap:10 http://tools.ietf.org/html/draft-ietf-avtext-framemarking-07\r\na=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid\r\na=sendrecv\r\na=msid:CTOoJEsouey3Vg1GBRoUf7DmZCe4RjSOnwJT 9a68cf1f-068c-4186-b75a-a9f034a1d158\r\na=rtcp-mux\r\na=rtcp-rsize\r\na=rtpmap:96 VP8/90000\r\na=rtcp-fb:96 goog-remb\r\na=rtcp-fb:96 transport-cc\r\na=rtcp-fb:96 ccm fir\r\na=rtcp-fb:96 nack\r\na=rtcp-fb:96 nack pli\r\na=rtpmap:97 rtx/90000\r\na=fmtp:97 apt=96\r\na=rtpmap:98 VP9/90000\r\na=rtcp-fb:98 goog-remb\r\na=rtcp-fb:98 transport-cc\r\na=rtcp-fb:98 ccm fir\r\na=rtcp-fb:98 nack\r\na=rtcp-fb:98 nack pli\r\na=fmtp:98 profile-id=0\r\na=rtpmap:99 rtx/90000\r\na=fmtp:99 apt=98\r\na=rtpmap:127 red/90000\r\na=rtpmap:124 rtx/90000\r\na=fmtp:124 apt=127\r\na=rtpmap:125 ulpfec/90000\r\na=ssrc-group:FID 2169364064 3023599391\r\na=ssrc:2169364064 cname:sTeiwPcf9UjxBLfU\r\na=ssrc:3023599391 cname:sTeiwPcf9UjxBLfU\r\nEnvironment
Peer 1:
Peer 2:
Additional context For signaling we are using a costume solution based on WebSockets. However, all signaling events seam to be happening in the correct order. When performing a call from Unity editor to another Unity editor everything works fine.