search

microsoft / MixedReality-WebRTC

MixedReality-WebRTC is a collection of components to help mixed reality app developers integrate audio and video real-time communication into their application and improve their collaborative experience
https://microsoft.github.io/MixedReality-WebRTC/
MIT License
913 stars 283 forks source link

Remote description not setting when PrefferedVideoCodec = "H264" On HoloLens 2 #779

Open AleksandrShmalko opened 3 years ago

AleksandrShmalko commented 3 years ago

Describe the bug When I call await Peer.SetRemoteDescriptionAsync(message) on client side with settings : PeerConnection.Peer.PreferredVideoCodec = "H264" , There is exception. StackTrace of this exception has method Microsoft.MixedReality.WebRTC.Unity.NodeDssSignale.CO_GetAndProcessFromServer which I was debugging a lot. But it is never called, I was debugging it`s call and method also. So, it is not clear how to debug and fix this problem

With VP8 softwarte decoder it works good, but for hardware H264 I spent a lot of time without result to launch this

To Reproduce Steps to reproduce the behavior:

  1. Set Peer.PrefferedVideoCodec = "H264"
  2. In overrided signaler class, in method OnSdpAnserReceived, call PeerConnection.HandleConnectionMethodAsync(sdpAnswer)
  3. Catch exception

Expected behavior Sdp answer generates and sends to another peer

SDP messages

Offer message "v=0 o=- 6372732482618574686 2 IN IP4 127.0.0.1 s=- t=0 0 a=group:BUNDLE 0 1 2 a=msid-semantic: WMS isar_stream m=audio 9 UDP/TLS/RTP/SAVPF 111 103 104 9 102 0 8 106 105 13 110 112 113 126 c=IN IP4 0.0.0.0 b=AS:20000 a=rtcp:9 IN IP4 0.0.0.0 a=ice-ufrag:bybj a=ice-pwd:AhoVyAHF14F4SR1KS77QMUtR a=ice-options:trickle a=fingerprint:sha-256 FC:35:0A:C0:E8:44:69:5C:40:6C:60:93:26:94:80:21:12:9D:F0:56:BB:97:40:C1:9B:A4:69:1A:0B:CA:94:BF a=setup:actpass a=mid:0 a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level a=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid a=sendrecv a=msid:isar_stream audio_track a=rtcp-mux a=rtpmap:111 opus/48000/2 a=rtcp-fb:111 transport-cc a=fmtp:111 minptime=10;useinbandfec=1 a=rtpmap:103 ISAC/16000 a=rtpmap:104 ISAC/32000 a=rtpmap:9 G722/8000 a=rtpmap:102 ILBC/8000 a=rtpmap:0 PCMU/8000 a=rtpmap:8 PCMA/8000 a=rtpmap:106 CN/32000 a=rtpmap:105 CN/16000 a=rtpmap:13 CN/8000 a=rtpmap:110 telephone-event/48000 a=rtpmap:112 telephone-event/32000 a=rtpmap:113 telephone-event/16000 a=rtpmap:126 telephone-event/8000 a=ssrc:3783151057 cname:LTKdYAtqXSJtBqa+ a=ssrc:3783151057 msid:isar_stream audio_track a=ssrc:3783151057 mslabel:isar_stream a=ssrc:3783151057 label:audio_track m=video 9 UDP/TLS/RTP/SAVPF 96 97 98 99 100 101 127 123 125 122 124 c=IN IP4 0.0.0.0 b=AS:20000 a=rtcp:9 IN IP4 0.0.0.0 a=ice-ufrag:bybj a=ice-pwd:AhoVyAHF14F4SR1KS77QMUtR a=ice-options:trickle a=fingerprint:sha-256 FC:35:0A:C0:E8:44:69:5C:40:6C:60:93:26:94:80:21:12:9D:F0:56:BB:97:40:C1:9B:A4:69:1A:0B:CA:94:BF a=setup:actpass a=mid:1 a=extmap:2 urn:ietf:params:rtp-hdrext:toffset a=extmap:3 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time a=extmap:4 urn:3gpp:video-orientation a=extmap:5 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01 a=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type a=extmap:8 http://www.webrtc.org/experiments/rtp-hdrext/video-timing a=extmap:10 http://tools.ietf.org/html/draft-ietf-avtext-framemarking-07 a=extmap:14 http://www.holo-light.com/404 a=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid a=sendrecv a=msid:isar_stream d3d_videotrack a=rtcp-mux a=rtcp-rsize a=rtpmap:96 H264/90000 a=rtcp-fb:96 goog-remb a=rtcp-fb:96 transport-cc a=rtcp-fb:96 ccm fir a=rtcp-fb:96 nack a=rtcp-fb:96 nack pli a=fmtp:96 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=42100b a=rtpmap:97 rtx/90000 a=fmtp:97 apt=96 a=rtpmap:98 H264/90000 a=rtcp-fb:98 goog-remb a=rtcp-fb:98 transport-cc a=rtcp-fb:98 ccm fir a=rtcp-fb:98 nack a=rtcp-fb:98 nack pli a=fmtp:98 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42100b a=rtpmap:99 rtx/90000 a=fmtp:99 apt=98 a=rtpmap:100 H264/90000 a=rtcp-fb:100 goog-remb a=rtcp-fb:100 transport-cc a=rtcp-fb:100 ccm fir a=rtcp-fb:100 nack a=rtcp-fb:100 nack pli a=fmtp:100 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=42e01f a=rtpmap:101 rtx/90000 a=fmtp:101 apt=100 a=rtpmap:127 H264/90000 a=rtcp-fb:127 goog-remb a=rtcp-fb:127 transport-cc a=rtcp-fb:127 ccm fir a=rtcp-fb:127 nack a=rtcp-fb:127 nack pli a=fmtp:127 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f a=rtpmap:123 rtx/90000 a=fmtp:123 apt=127 a=rtpmap:125 red/90000 a=rtpmap:122 rtx/90000 a=fmtp:122 apt=125 a=rtpmap:124 ulpfec/90000 a=ssrc-group:FID 3668965338 1381498673 a=ssrc:3668965338 cname:LTKdYAtqXSJtBqa+ a=ssrc:3668965338 msid:isar_stream d3d_videotrack a=ssrc:3668965338 mslabel:isar_stream a=ssrc:3668965338 label:d3d_videotrack a=ssrc:1381498673 cname:LTKdYAtqXSJtBqa+ a=ssrc:1381498673 msid:isar_stream d3d_videotrack a=ssrc:1381498673 mslabel:isar_stream a=ssrc:1381498673 label:d3d_videotrack m=application 9 DTLS/SCTP 5000 c=IN IP4 0.0.0.0 b=AS:20000 a=ice-ufrag:bybj a=ice-pwd:AhoVyAHF14F4SR1KS77QMUtR a=ice-options:trickle a=fingerprint:sha-256 FC:35:0A:C0:E8:44:69:5C:40:6C:60:93:26:94:80:21:12:9D:F0:56:BB:97:40:C1:9B:A4:69:1A:0B:CA:94:BF a=setup:actpass a=mid:2 a=sctpmap:5000 webrtc-datachannel 1024"

SDP answer is not configuring because of error

Environment

Additional context Exception description :

spacecheeserocks commented 3 years ago

Not the only possibility, but this likely means the response would have refused to negotiate the H264 codec.
If you are testing between HoloLens and Unity Editor, or HoloLens and Unity PC/Standalone(non UWP), this will happen, because the non-UWP library does not support H264 video.

If you want to test in this case, your options are:

This "Value does not fall within the expected range" exception is unfortunately quite generic, because it's a 'default' message based on an error code from the internal library.

This exception is sometimes thrown when the WebRTC library didn't "like" the SDP answer it received - such as - when the answer 'rejects' a video negotiation and disables it, or when the response for the video line contains a codec that the WebRTC library doesn't support.

The "PreferredVideoCodec = H264" option instructs the library to only offer H264 for outgoing video, and to only accept H264 for incoming video. Since the Editor/PC library doesn't support H264, it disables the video line because there is no negotiable codec.

AleksandrShmalko commented 3 years ago

Thank you for the answer! Yes, I really forced "H264" to the "PrefferedVideoCodec" and it was the problem. Without any settings it works good. Additional and useful question, is there any ways to get time of the decoding frame? I need this time to be sure that I use hardware but not software one.

Matteo-0 commented 3 years ago

Sorry, Can you please tell me how to modify the videocodec, in which script should I look? I cannot find anything.