Open gamedolphin opened 4 years ago
Hi @gamedolphin
I am not familiar with "pixel streaming web server" of UE4 (do you have a link please, out of curiosity?)
Your first SDP offer doesn't list H.264 as a supported video codec, so it makes sense that you get an ArgumentException
when trying to force it. I don't remember in detail how it works on Desktop nor why H.264 is not available from the Google software codec implementation, but I know the Media Foundation one provided by webrtc-uwp-sdk is only available for UWP, so can't be used in the Unity editor.
Now for the issue, my (wild) guess is also that there is no compatible codec between the two peers. To confirm that it would be interesting to see the actual answer message in the case where it works, to see what video codecs the server supports. If it supports only H.264 then that's why it doesn't work.
Hey! Thank you for the prompt reply. Will inserting a a=rtpmap:125 H264/90000
work? 😬 Anyway, for reference, UE4 Pixel Streaming is here - https://docs.unrealengine.com/en-US/Platforms/PixelStreaming/index.html
This is the answer for the browser when connecting using chrome -
v=0
o=- 1837563472811065345 2 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE 0 1 2
a=msid-semantic: WMS stream_id
m=audio 9 UDP/TLS/RTP/SAVPF 111 110
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:TmiO
a=ice-pwd:8Pgt6p99jjFLhvFQk3KB09bw
a=ice-options:trickle
a=fingerprint:sha-256 D8:34:86:F2:C9:EF:B3:5A:DC:B4:36:3B:3B:F6:9D:A2:87:35:E8:06:93:EC:D5:94:E7:72:F3:97:B0:E4:CC:1F
a=setup:active
a=mid:0
a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level
a=extmap:3 urn:ietf:params:rtp-hdrext:sdes:mid
a=sendonly
a=msid:stream_id audio_label
a=rtcp-mux
a=rtpmap:111 opus/48000/2
a=rtcp-fb:111 transport-cc
a=fmtp:111 minptime=10;useinbandfec=1
a=rtpmap:110 telephone-event/48000
a=ssrc:1620706015 cname:gQpgtB6hMAKYBLaq
a=ssrc:1620706015 msid:stream_id audio_label
a=ssrc:1620706015 mslabel:stream_id
a=ssrc:1620706015 label:audio_label
m=video 9 UDP/TLS/RTP/SAVPF 125 107 114 115 116
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:TmiO
a=ice-pwd:8Pgt6p99jjFLhvFQk3KB09bw
a=ice-options:trickle
a=fingerprint:sha-256 D8:34:86:F2:C9:EF:B3:5A:DC:B4:36:3B:3B:F6:9D:A2:87:35:E8:06:93:EC:D5:94:E7:72:F3:97:B0:E4:CC:1F
a=setup:active
a=mid:1
a=extmap:14 urn:ietf:params:rtp-hdrext:toffset
a=extmap:13 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:12 urn:3gpp:video-orientation
a=extmap:2 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:11 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay
a=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type
a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-timing
a=extmap:3 urn:ietf:params:rtp-hdrext:sdes:mid
a=sendonly
a=msid:stream_id video_label
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:125 H264/90000
a=rtcp-fb:125 goog-remb
a=rtcp-fb:125 transport-cc
a=rtcp-fb:125 ccm fir
a=rtcp-fb:125 nack
a=rtcp-fb:125 nack pli
a=fmtp:125 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e033
a=rtpmap:107 rtx/90000
a=fmtp:107 apt=125
a=rtpmap:114 red/90000
a=rtpmap:115 rtx/90000
a=fmtp:115 apt=114
a=rtpmap:116 ulpfec/90000
a=ssrc-group:FID 611003714 425136751
a=ssrc:611003714 cname:gQpgtB6hMAKYBLaq
a=ssrc:611003714 msid:stream_id video_label
a=ssrc:611003714 mslabel:stream_id
a=ssrc:611003714 label:video_label
a=ssrc:425136751 cname:gQpgtB6hMAKYBLaq
a=ssrc:425136751 msid:stream_id video_label
a=ssrc:425136751 mslabel:stream_id
a=ssrc:425136751 label:video_label
m=application 9 UDP/DTLS/SCTP webrtc-datachannel
c=IN IP4 0.0.0.0
b=AS:30
a=ice-ufrag:TmiO
a=ice-pwd:8Pgt6p99jjFLhvFQk3KB09bw
a=ice-options:trickle
a=fingerprint:sha-256 D8:34:86:F2:C9:EF:B3:5A:DC:B4:36:3B:3B:F6:9D:A2:87:35:E8:06:93:EC:D5:94:E7:72:F3:97:B0:E4:CC:1F
a=setup:active
a=mid:2
a=sctp-port:5000
Minor update -> replacing a=rtpmap:98 VP9/90000
with a=rtpmap:98 H264/90000
along with the setting a=fmtp:98 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e033
seems to have convinced the remote client to finally send the video track and the webrtc connection state gets updated to connected. BUT OnTrack is not called in the unity editor. I have been digging into the c++ code to see what exactly triggers it.
Are you replacing in the answer only? If so then the peers won't be in sync because the offering side will not know about your change, which I assume you do after calling SetLocalDescription()
.
I am confused by your SDP messages. You pasted in your first message, second block an "offer from the browser", which implies the browser initiates the connection and the answer comes from the UE4 pixel server. Then in a later message you pasted the "answer for the browser"; is that the one coming from the UE4 server? Because they look suspiciously similar (only some password/fingerprint differ), and both list e.g. rtx/90000
which is also offered by Unity (see your first block), so this codec at least should be compatible with all peers and should be selected as fallback if H.264 or others are not supported by all peers.
Im sorry for the late reply. Yes "answer for browser" is the reply from the ue4 server. I've been exploring other options to transcode the stream to vp8/vp9, with not much luck.
Is there a way to replace the webrtc base plugin to use a custom compiled one with h264 support on the editor?
There are issues with that. H.264 supports on UWP comes from code in the WebRTC UWP project, and doesn't work with Desktop as is. Alternatively, Google has some H.264 software encoder, but it needs ffmepg I think, and is compiled out by default due to licensing limitations. So you would have to basically recompile everything from scratch, including the Google impementation, the WebRTC UWP project, and this project, and make somewhat significant changes to all of them. That sounds like a prohibitive amount of work, and we cannot offer any support for it nor guarantee it will even work. So the basic answer is no, there is no simple way to do that at the moment.
It seems the UE4 offer only contains H.264, so the two are just not compatible. You could have some implementation for UWP in Unity, which would work, but I don't see any simple path forward for the Unity Editor and other non-UWP platforms.
My final goal is to stream UE4 pixel streaming output (which uses WebRTC) directly into Unity using this wonderful library. I have figured out the signalling by investigating the client side javascript of the pixel streaming web server. The offer sdp from unity is this ->
But the pixel streaming server fails to send an answer, and shuts down the connection with the message
Failed to set remote offer sdp: Failed to set remote video description send parameters.
.The offer sdp from the browser (which seems to work and the pixel streaming server returns an answer promptly) is as follows :
What am I missing? I am quite stuck at what needs to be done. One of my wild guesses is that the unity side does not support the codec (h264) the ue4 side is sending? I am not sure. Setting
PeerConnection.Peer.PreferredVideoCodec = "H264";
causes unity to sayArgumentException: Value does not fall within the expected range.
when creating the offer.