MixedReality-WebRTC is a collection of components to help mixed reality app developers integrate audio and video real-time communication into their application and improve their collaborative experience
Describe the bug
If I produce RGBA Images either via already existing VideoSources or a self made custom one and send them into WebRTC, the native peer always returns Transceivers with I420a formatted RemoteTracks. This results in a wrong Pairing Process between the two clients which again prevents the answering PeerConnection from displaying the RemoteTrack because it´s pixel format is wrong.
To Reproduce
Add any ARBG Video Source into your Project, with a LocalOnly Signaler and set a debug point at the receiving PeerConnection in Line 487. Here you will see that the remote tracks of the Transceivers are always I420a formatted.
Expected behavior
I would either expect that the native Peer returns the correct Format automatically or that there is a possibility to configure the PixelFormat type for example within TransceiverInitSettings within the StartConnection() method of PeerConnection.
SDP messagesSDP messages are extremely helpful to diagnose a wide range of issues from connection not being established to audio/video issues and codecs issues.
EnvironmentPlease fill the information for each peer if different
MR-WebRTC version: Package version 2.0.2
Platform: Windows Desktop, UWP, Unity Editor
Architecture: x64, ARM
Unity version (if applicable): 2019.4.7f1
Target device: Windows Desktop, HoloLens 2
Additional context
I tried to set up the WebRTC stream from Desktop client to Desktop client via LocalHost and from Desktop client to HoloLens2 via USB (UsbNcm). The across mentioned SDP Messages are generated in the LocalHost Approach.
Describe the bug If I produce RGBA Images either via already existing VideoSources or a self made custom one and send them into WebRTC, the native peer always returns Transceivers with I420a formatted RemoteTracks. This results in a wrong Pairing Process between the two clients which again prevents the answering PeerConnection from displaying the RemoteTrack because it´s pixel format is wrong.
To Reproduce Add any ARBG Video Source into your Project, with a LocalOnly Signaler and set a debug point at the receiving PeerConnection in Line 487. Here you will see that the remote tracks of the Transceivers are always I420a formatted.
Expected behavior I would either expect that the native Peer returns the correct Format automatically or that there is a possibility to configure the PixelFormat type for example within TransceiverInitSettings within the StartConnection() method of PeerConnection.
SDP messages SDP messages are extremely helpful to diagnose a wide range of issues from connection not being established to audio/video issues and codecs issues.
Offer message
v=0 o=- 2076696841371316938 4 IN IP4 127.0.0.1 s=- t=0 0 a=group:BUNDLE 0 a=msid-semantic: WMS m=video 9 UDP/TLS/RTP/SAVPF 96 97 98 99 100 101 127 124 125 c=IN IP4 0.0.0.0 a=rtcp:9 IN IP4 0.0.0.0 a=ice-ufrag:wzDB a=ice-pwd:xFUlepVqZ3YPb2yU2H83T3qi a=ice-options:trickle a=fingerprint:sha-256 3A:3B:E4:9A:E3:BF:0D:AC:35:19:52:E0:78:5C:A8:60:6A:3A:7F:D7:EF:B1:6D:A3:0B:8C:18:00:D7:9B:49:5A a=setup:actpass a=mid:0 a=extmap:2 urn:ietf:params:rtp-hdrext:toffset a=extmap:3 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time a=extmap:4 urn:3gpp:video-orientation a=extmap:5 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01 a=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type a=extmap:8 http://www.webrtc.org/experiments/rtp-hdrext/video-timing a=extmap:10 http://tools.ietf.org/html/draft-ietf-avtext-framemarking-07 a=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid a=sendrecv a=msid:- f39884f9-879a-4422-b901-a3b06c7839d2 a=rtcp-mux a=rtcp-rsize a=rtpmap:96 VP8/90000 a=rtcp-fb:96 goog-remb a=rtcp-fb:96 transport-cc a=rtcp-fb:96 ccm fir a=rtcp-fb:96 nack a=rtcp-fb:96 nack pli a=rtpmap:97 rtx/90000 a=fmtp:97 apt=96 a=rtpmap:98 VP9/90000 a=rtcp-fb:98 goog-remb a=rtcp-fb:98 transport-cc a=rtcp-fb:98 ccm fir a=rtcp-fb:98 nack a=rtcp-fb:98 nack pli a=fmtp:98 x-google-profile-id=0 a=rtpmap:99 rtx/90000 a=fmtp:99 apt=98 a=rtpmap:100 multiplex/90000 a=rtcp-fb:100 goog-remb a=rtcp-fb:100 transport-cc a=rtcp-fb:100 ccm fir a=rtcp-fb:100 nack a=rtcp-fb:100 nack pli a=fmtp:100 acn=VP9;x-google-profile-id=0 a=rtpmap:101 rtx/90000 a=fmtp:101 apt=100 a=rtpmap:127 red/90000 a=rtpmap:124 rtx/90000 a=fmtp:124 apt=127 a=rtpmap:125 ulpfec/90000 a=ssrc-group:FID 3960815271 3466666144 a=ssrc:3960815271 cname:PgSiugwinW0PykZ0 a=ssrc:3960815271 msid: f39884f9-879a-4422-b901-a3b06c7839d2 a=ssrc:3960815271 mslabel: a=ssrc:3960815271 label:f39884f9-879a-4422-b901-a3b06c7839d2 a=ssrc:3466666144 cname:PgSiugwinW0PykZ0 a=ssrc:3466666144 msid: f39884f9-879a-4422-b901-a3b06c7839d2 a=ssrc:3466666144 mslabel: a=ssrc:3466666144 label:f39884f9-879a-4422-b901-a3b06c7839d2Answer message
v=0 o=- 1327232969244664064 2 IN IP4 127.0.0.1 s=- t=0 0 a=group:BUNDLE 0 a=msid-semantic: WMS m=video 9 UDP/TLS/RTP/SAVPF 96 97 98 99 100 101 127 124 125 c=IN IP4 0.0.0.0 a=rtcp:9 IN IP4 0.0.0.0 a=ice-ufrag:LCYd a=ice-pwd:v2lssTZH/TKpeogSJSh9BBaX a=ice-options:trickle a=fingerprint:sha-256 5A:28:C2:C1:F2:94:67:18:8D:A5:92:70:12:49:81:CD:C7:3B:0A:26:61:48:1A:0E:23:4C:82:4C:FA:1E:39:CD a=setup:active a=mid:0 a=extmap:2 urn:ietf:params:rtp-hdrext:toffset a=extmap:3 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time a=extmap:4 urn:3gpp:video-orientation a=extmap:5 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01 a=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type a=extmap:8 http://www.webrtc.org/experiments/rtp-hdrext/video-timing a=extmap:10 http://tools.ietf.org/html/draft-ietf-avtext-framemarking-07 a=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid a=sendrecv a=msid:- a824019e-8ee2-4f2b-99f3-f40694c4b2fc a=rtcp-mux a=rtcp-rsize a=rtpmap:96 VP8/90000 a=rtcp-fb:96 goog-remb a=rtcp-fb:96 transport-cc a=rtcp-fb:96 ccm fir a=rtcp-fb:96 nack a=rtcp-fb:96 nack pli a=rtpmap:97 rtx/90000 a=fmtp:97 apt=96 a=rtpmap:98 VP9/90000 a=rtcp-fb:98 goog-remb a=rtcp-fb:98 transport-cc a=rtcp-fb:98 ccm fir a=rtcp-fb:98 nack a=rtcp-fb:98 nack pli a=fmtp:98 x-google-profile-id=0 a=rtpmap:99 rtx/90000 a=fmtp:99 apt=98 a=rtpmap:100 multiplex/90000 a=rtcp-fb:100 goog-remb a=rtcp-fb:100 transport-cc a=rtcp-fb:100 ccm fir a=rtcp-fb:100 nack a=rtcp-fb:100 nack pli a=fmtp:100 acn=VP9;x-google-profile-id=0 a=rtpmap:101 rtx/90000 a=fmtp:101 apt=100 a=rtpmap:127 red/90000 a=rtpmap:124 rtx/90000 a=fmtp:124 apt=127 a=rtpmap:125 ulpfec/90000 a=ssrc-group:FID 553080981 3209422063 a=ssrc:553080981 cname:3Vlj9y151vITuW24 a=ssrc:3209422063 cname:3Vlj9y151vITuW24Environment Please fill the information for each peer if different
Additional context I tried to set up the WebRTC stream from Desktop client to Desktop client via LocalHost and from Desktop client to HoloLens2 via USB (UsbNcm). The across mentioned SDP Messages are generated in the LocalHost Approach.