search

microsoft / MixedReality-WebRTC

MixedReality-WebRTC is a collection of components to help mixed reality app developers integrate audio and video real-time communication into their application and improve their collaborative experience
https://microsoft.github.io/MixedReality-WebRTC/
MIT License
908 stars 282 forks source link

Playing video from remote source #88

Open gamedolphin opened 4 years ago

gamedolphin commented 4 years ago

My final goal is to stream UE4 pixel streaming output (which uses WebRTC) directly into Unity using this wonderful library. I have figured out the signalling by investigating the client side javascript of the pixel streaming web server. The offer sdp from unity is this -> 

o=- 6539762120684278701 2 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE 0 1
a=msid-semantic: WMS
m=audio 9 UDP/TLS/RTP/SAVPF 111 103 104 9 102 0 8 106 105 13 110 112 113 126
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:GsFB
a=ice-pwd:YYdHkh1e5pwDBWOoPoUwyKZ3
a=ice-options:trickle
a=fingerprint:sha-256 DB:C0:BE:0E:56:C1:B0:A8:F2:99:2C:2E:CF:09:EC:32:EC:7E:F1:59:D2:34:D7:25:87:03:65:5C:CC:01:62:FC
a=setup:actpass
a=mid:0
a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level
a=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid
a=recvonly
a=rtcp-mux
a=rtpmap:111 opus/48000/2
a=rtcp-fb:111 transport-cc
a=fmtp:111 minptime=10;useinbandfec=1
a=rtpmap:103 ISAC/16000
a=rtpmap:104 ISAC/32000
a=rtpmap:9 G722/8000
a=rtpmap:102 ILBC/8000
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:106 CN/32000
a=rtpmap:105 CN/16000
a=rtpmap:13 CN/8000
a=rtpmap:110 telephone-event/48000
a=rtpmap:112 telephone-event/32000
a=rtpmap:113 telephone-event/16000
a=rtpmap:126 telephone-event/8000
m=video 9 UDP/TLS/RTP/SAVPF 96 97 98 99 100 101 127 124 125
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:GsFB
a=ice-pwd:YYdHkh1e5pwDBWOoPoUwyKZ3
a=ice-options:trickle
a=fingerprint:sha-256 DB:C0:BE:0E:56:C1:B0:A8:F2:99:2C:2E:CF:09:EC:32:EC:7E:F1:59:D2:34:D7:25:87:03:65:5C:CC:01:62:FC
a=setup:actpass
a=mid:1
a=extmap:2 urn:ietf:params:rtp-hdrext:toffset
a=extmap:3 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:4 urn:3gpp:video-orientation
a=extmap:5 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay
a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type
a=extmap:8 http://www.webrtc.org/experiments/rtp-hdrext/video-timing
a=extmap:10 http://tools.ietf.org/html/draft-ietf-avtext-framemarking-07
a=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid
a=recvonly
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:96 VP8/90000
a=rtcp-fb:96 goog-remb
a=rtcp-fb:96 transport-cc
a=rtcp-fb:96 ccm fir
a=rtcp-fb:96 nack
a=rtcp-fb:96 nack pli
a=rtpmap:97 rtx/90000
a=fmtp:97 apt=96
a=rtpmap:98 VP9/90000
a=rtcp-fb:98 goog-remb
a=rtcp-fb:98 transport-cc
a=rtcp-fb:98 ccm fir
a=rtcp-fb:98 nack
a=rtcp-fb:98 nack pli
a=fmtp:98 x-google-profile-id=0
a=rtpmap:99 rtx/90000
a=fmtp:99 apt=98
a=rtpmap:100 multiplex/90000
a=rtcp-fb:100 goog-remb
a=rtcp-fb:100 transport-cc
a=rtcp-fb:100 ccm fir
a=rtcp-fb:100 nack
a=rtcp-fb:100 nack pli
a=fmtp:100 acn=VP9;x-google-profile-id=0
a=rtpmap:101 rtx/90000
a=fmtp:101 apt=100
a=rtpmap:127 red/90000
a=rtpmap:124 rtx/90000
a=fmtp:124 apt=127
a=rtpmap:125 ulpfec/90000

But the pixel streaming server fails to send an answer, and shuts down the connection with the message Failed to set remote offer sdp: Failed to set remote video description send parameters..

The offer sdp from the browser (which seems to work and the pixel streaming server returns an answer promptly) is as follows :

o=- 3955894584286021740 2 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE 0 1 2
a=msid-semantic: WMS stream_id
m=audio 9 UDP/TLS/RTP/SAVPF 111 110
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:QBL2
a=ice-pwd:69soRBZjafxCU+yBjux/ajin
a=ice-options:trickle
a=fingerprint:sha-256 7A:98:B6:98:5C:37:79:B7:87:E0:DD:51:48:4A:F1:44:0A:7C:1B:07:B8:1A:53:B2:62:83:0B:E5:1A:E7:59:8A
a=setup:active
a=mid:0
a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level
a=extmap:3 urn:ietf:params:rtp-hdrext:sdes:mid
a=sendonly
a=msid:stream_id audio_label
a=rtcp-mux
a=rtpmap:111 opus/48000/2
a=rtcp-fb:111 transport-cc
a=fmtp:111 minptime=10;useinbandfec=1
a=rtpmap:110 telephone-event/48000
a=ssrc:3696450137 cname:xiCSXrDBl38Agepp
a=ssrc:3696450137 msid:stream_id audio_label
a=ssrc:3696450137 mslabel:stream_id
a=ssrc:3696450137 label:audio_label
m=video 9 UDP/TLS/RTP/SAVPF 125 107 114 115 116
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:QBL2
a=ice-pwd:69soRBZjafxCU+yBjux/ajin
a=ice-options:trickle
a=fingerprint:sha-256 7A:98:B6:98:5C:37:79:B7:87:E0:DD:51:48:4A:F1:44:0A:7C:1B:07:B8:1A:53:B2:62:83:0B:E5:1A:E7:59:8A
a=setup:active
a=mid:1
a=extmap:14 urn:ietf:params:rtp-hdrext:toffset
a=extmap:13 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:12 urn:3gpp:video-orientation
a=extmap:2 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:11 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay
a=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type
a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-timing
a=extmap:3 urn:ietf:params:rtp-hdrext:sdes:mid
a=sendonly
a=msid:stream_id video_label
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:125 H264/90000
a=rtcp-fb:125 goog-remb
a=rtcp-fb:125 transport-cc
a=rtcp-fb:125 ccm fir
a=rtcp-fb:125 nack
a=rtcp-fb:125 nack pli
a=fmtp:125 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e033
a=rtpmap:107 rtx/90000
a=fmtp:107 apt=125
a=rtpmap:114 red/90000
a=rtpmap:115 rtx/90000
a=fmtp:115 apt=114
a=rtpmap:116 ulpfec/90000
a=ssrc-group:FID 3233574813 670277948
a=ssrc:3233574813 cname:xiCSXrDBl38Agepp
a=ssrc:3233574813 msid:stream_id video_label
a=ssrc:3233574813 mslabel:stream_id
a=ssrc:3233574813 label:video_label
a=ssrc:670277948 cname:xiCSXrDBl38Agepp
a=ssrc:670277948 msid:stream_id video_label
a=ssrc:670277948 mslabel:stream_id
a=ssrc:670277948 label:video_label
m=application 9 UDP/DTLS/SCTP webrtc-datachannel
c=IN IP4 0.0.0.0
b=AS:30
a=ice-ufrag:QBL2
a=ice-pwd:69soRBZjafxCU+yBjux/ajin
a=ice-options:trickle
a=fingerprint:sha-256 7A:98:B6:98:5C:37:79:B7:87:E0:DD:51:48:4A:F1:44:0A:7C:1B:07:B8:1A:53:B2:62:83:0B:E5:1A:E7:59:8A
a=setup:active
a=mid:2
a=sctp-port:5000

What am I missing? I am quite stuck at what needs to be done. One of my wild guesses is that the unity side does not support the codec (h264) the ue4 side is sending? I am not sure. Setting PeerConnection.Peer.PreferredVideoCodec = "H264"; causes unity to say ArgumentException: Value does not fall within the expected range. when creating the offer.

djee-ms commented 4 years ago

Hi @gamedolphin

I am not familiar with "pixel streaming web server" of UE4 (do you have a link please, out of curiosity?)

Your first SDP offer doesn't list H.264 as a supported video codec, so it makes sense that you get an ArgumentException when trying to force it. I don't remember in detail how it works on Desktop nor why H.264 is not available from the Google software codec implementation, but I know the Media Foundation one provided by webrtc-uwp-sdk is only available for UWP, so can't be used in the Unity editor.

Now for the issue, my (wild) guess is also that there is no compatible codec between the two peers. To confirm that it would be interesting to see the actual answer message in the case where it works, to see what video codecs the server supports. If it supports only H.264 then that's why it doesn't work.

gamedolphin commented 4 years ago

Hey! Thank you for the prompt reply. Will inserting a a=rtpmap:125 H264/90000 work? 😬 Anyway, for reference, UE4 Pixel Streaming is here - https://docs.unrealengine.com/en-US/Platforms/PixelStreaming/index.html

gamedolphin commented 4 years ago

This is the answer for the browser when connecting using chrome -

v=0
o=- 1837563472811065345 2 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE 0 1 2
a=msid-semantic: WMS stream_id
m=audio 9 UDP/TLS/RTP/SAVPF 111 110
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:TmiO
a=ice-pwd:8Pgt6p99jjFLhvFQk3KB09bw
a=ice-options:trickle
a=fingerprint:sha-256 D8:34:86:F2:C9:EF:B3:5A:DC:B4:36:3B:3B:F6:9D:A2:87:35:E8:06:93:EC:D5:94:E7:72:F3:97:B0:E4:CC:1F
a=setup:active
a=mid:0
a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level
a=extmap:3 urn:ietf:params:rtp-hdrext:sdes:mid
a=sendonly
a=msid:stream_id audio_label
a=rtcp-mux
a=rtpmap:111 opus/48000/2
a=rtcp-fb:111 transport-cc
a=fmtp:111 minptime=10;useinbandfec=1
a=rtpmap:110 telephone-event/48000
a=ssrc:1620706015 cname:gQpgtB6hMAKYBLaq
a=ssrc:1620706015 msid:stream_id audio_label
a=ssrc:1620706015 mslabel:stream_id
a=ssrc:1620706015 label:audio_label
m=video 9 UDP/TLS/RTP/SAVPF 125 107 114 115 116
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:TmiO
a=ice-pwd:8Pgt6p99jjFLhvFQk3KB09bw
a=ice-options:trickle
a=fingerprint:sha-256 D8:34:86:F2:C9:EF:B3:5A:DC:B4:36:3B:3B:F6:9D:A2:87:35:E8:06:93:EC:D5:94:E7:72:F3:97:B0:E4:CC:1F
a=setup:active
a=mid:1
a=extmap:14 urn:ietf:params:rtp-hdrext:toffset
a=extmap:13 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:12 urn:3gpp:video-orientation
a=extmap:2 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:11 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay
a=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type
a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-timing
a=extmap:3 urn:ietf:params:rtp-hdrext:sdes:mid
a=sendonly
a=msid:stream_id video_label
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:125 H264/90000
a=rtcp-fb:125 goog-remb
a=rtcp-fb:125 transport-cc
a=rtcp-fb:125 ccm fir
a=rtcp-fb:125 nack
a=rtcp-fb:125 nack pli
a=fmtp:125 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e033
a=rtpmap:107 rtx/90000
a=fmtp:107 apt=125
a=rtpmap:114 red/90000
a=rtpmap:115 rtx/90000
a=fmtp:115 apt=114
a=rtpmap:116 ulpfec/90000
a=ssrc-group:FID 611003714 425136751
a=ssrc:611003714 cname:gQpgtB6hMAKYBLaq
a=ssrc:611003714 msid:stream_id video_label
a=ssrc:611003714 mslabel:stream_id
a=ssrc:611003714 label:video_label
a=ssrc:425136751 cname:gQpgtB6hMAKYBLaq
a=ssrc:425136751 msid:stream_id video_label
a=ssrc:425136751 mslabel:stream_id
a=ssrc:425136751 label:video_label
m=application 9 UDP/DTLS/SCTP webrtc-datachannel
c=IN IP4 0.0.0.0
b=AS:30
a=ice-ufrag:TmiO
a=ice-pwd:8Pgt6p99jjFLhvFQk3KB09bw
a=ice-options:trickle
a=fingerprint:sha-256 D8:34:86:F2:C9:EF:B3:5A:DC:B4:36:3B:3B:F6:9D:A2:87:35:E8:06:93:EC:D5:94:E7:72:F3:97:B0:E4:CC:1F
a=setup:active
a=mid:2
a=sctp-port:5000
gamedolphin commented 4 years ago

Minor update -> replacing a=rtpmap:98 VP9/90000 with a=rtpmap:98 H264/90000 along with the setting a=fmtp:98 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e033 seems to have convinced the remote client to finally send the video track and the webrtc connection state gets updated to connected. BUT OnTrack is not called in the unity editor. I have been digging into the c++ code to see what exactly triggers it.

djee-ms commented 4 years ago

Are you replacing in the answer only? If so then the peers won't be in sync because the offering side will not know about your change, which I assume you do after calling SetLocalDescription().

I am confused by your SDP messages. You pasted in your first message, second block an "offer from the browser", which implies the browser initiates the connection and the answer comes from the UE4 pixel server. Then in a later message you pasted the "answer for the browser"; is that the one coming from the UE4 server? Because they look suspiciously similar (only some password/fingerprint differ), and both list e.g. rtx/90000 which is also offered by Unity (see your first block), so this codec at least should be compatible with all peers and should be selected as fallback if H.264 or others are not supported by all peers.

gamedolphin commented 4 years ago

Im sorry for the late reply. Yes "answer for browser" is the reply from the ue4 server. I've been exploring other options to transcode the stream to vp8/vp9, with not much luck.

Is there a way to replace the webrtc base plugin to use a custom compiled one with h264 support on the editor?

djee-ms commented 4 years ago

There are issues with that. H.264 supports on UWP comes from code in the WebRTC UWP project, and doesn't work with Desktop as is. Alternatively, Google has some H.264 software encoder, but it needs ffmepg I think, and is compiled out by default due to licensing limitations. So you would have to basically recompile everything from scratch, including the Google impementation, the WebRTC UWP project, and this project, and make somewhat significant changes to all of them. That sounds like a prohibitive amount of work, and we cannot offer any support for it nor guarantee it will even work. So the basic answer is no, there is no simple way to do that at the moment.

It seems the UE4 offer only contains H.264, so the two are just not compatible. You could have some implementation for UWP in Unity, which would work, but I don't see any simple path forward for the Unity Editor and other non-UWP platforms.