search

microsoft / MixedReality-WebRTC

MixedReality-WebRTC is a collection of components to help mixed reality app developers integrate audio and video real-time communication into their application and improve their collaborative experience
https://microsoft.github.io/MixedReality-WebRTC/
MIT License
910 stars 283 forks source link

Outgoing video not being received by the destination peer #679

Open anpep opened 3 years ago

anpep commented 3 years ago

Describe the bug I want bidirectional audio/video communication between two peers (one is my computer, the other is the HoloLens 2). I succeeded in getting the HoloLens stream audio and video to my computer, but the A/V stream from my computer does not get to the HoloLens.

In Unity, I've correctly set up the send/receive transceivers as seen in the screenshot: image

In my desktop code, I also set up the transceivers as send/receive:

{
    var microphoneSource = await DeviceAudioTrackSource.CreateAsync(new LocalAudioDeviceInitConfig());
    var microphoneTrack = LocalAudioTrack.CreateFromSource(microphoneSource, new LocalAudioTrackInitConfig {
        trackName = "microphone_track"
    });

    var audioTransceiver = PeerConnection.AddTransceiver(MediaKind.Audio);
    audioTransceiver.DesiredDirection = Transceiver.Direction.SendReceive;
    audioTransceiver.LocalAudioTrack = microphoneTrack;
}

{
    var captureDevices = await DeviceVideoTrackSource.GetCaptureDevicesAsync();
    var webcamSource = await DeviceVideoTrackSource.CreateAsync(new LocalVideoDeviceInitConfig {
        videoDevice = captureDevices.First(dev => dev.name.StartsWith("OBS"))
    });
    var webcamTrack = LocalVideoTrack.CreateFromSource(webcamSource, new LocalVideoTrackInitConfig {
        trackName = "webcam_track"
    });

    var videoTransceiver = PeerConnection.AddTransceiver(MediaKind.Video);
    videoTransceiver.DesiredDirection = Transceiver.Direction.SendReceive;
    videoTransceiver.LocalVideoTrack = webcamTrack;
}

PeerConnection.LocalSdpReadytoSend += OnLocalSdpReadyToSend;
PeerConnection.IceCandidateReadytoSend += OnIceCandidateReadyToSend;
PeerConnection.IceStateChanged += (s) => Log.Info($"ice state: {s}");
PeerConnection.VideoTrackAdded += OnVideoTrackAdded;

await PeerConnection.SetRemoteDescriptionAsync(new SdpMessage { Type = SdpMessageType.Offer, Content = msg.data.sessionDescription });
PeerConnection.CreateAnswer();

However, the SDP answer (the one going from my PC to the HoloLens) has a=recvonly instead of a=sendrecv, which seems strange to me because the transceivers are being created before sending the answer.

Expected behavior The HoloLens getting an incoming audio/video feed.

SDP messages SDP messages are extremely helpful to diagnose a wide range of issues from connection not being established to audio/video issues and codecs issues.

Offer message v=0 o=- 4196501685274162302 2 IN IP4 127.0.0.1 s=- t=0 0 a=group:BUNDLE 0 1 a=msid-semantic: WMS m=video 9 UDP/TLS/RTP/SAVPF 96 97 98 99 100 101 127 124 125 c=IN IP4 0.0.0.0 a=rtcp:9 IN IP4 0.0.0.0 a=ice-ufrag:6rrQ a=ice-pwd:oDzzm8LmH4EkllD/aUVD0XdP a=ice-options:trickle a=fingerprint:sha-256 AE:7C:7A:0E:B6:97:46:DD:48:94:3A:6C:BC:28:17:D6:60:F5:E2:86:00:AB:AF:E0:85:F3:75:BE:88:4F:4D:80 a=setup:actpass a=mid:0 a=extmap:2 urn:ietf:params:rtp-hdrext:toffset a=extmap:3 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time a=extmap:4 urn:3gpp:video-orientation a=extmap:5 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01 a=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type a=extmap:8 http://www.webrtc.org/experiments/rtp-hdrext/video-timing a=extmap:10 http://tools.ietf.org/html/draft-ietf-avtext-framemarking-07 a=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid a=sendrecv a=msid:- 2a9e03ed-b2a7-459e-9000-29fb331b317c a=rtcp-mux a=rtcp-rsize a=rtpmap:96 VP8/90000 a=rtcp-fb:96 goog-remb a=rtcp-fb:96 transport-cc a=rtcp-fb:96 ccm fir a=rtcp-fb:96 nack a=rtcp-fb:96 nack pli a=rtpmap:97 rtx/90000 a=fmtp:97 apt=96 a=rtpmap:98 VP9/90000 a=rtcp-fb:98 goog-remb a=rtcp-fb:98 transport-cc a=rtcp-fb:98 ccm fir a=rtcp-fb:98 nack a=rtcp-fb:98 nack pli a=fmtp:98 x-google-profile-id=0 a=rtpmap:99 rtx/90000 a=fmtp:99 apt=98 a=rtpmap:100 multiplex/90000 a=rtcp-fb:100 goog-remb a=rtcp-fb:100 transport-cc a=rtcp-fb:100 ccm fir a=rtcp-fb:100 nack a=rtcp-fb:100 nack pli a=fmtp:100 acn=VP9;x-google-profile-id=0 a=rtpmap:101 rtx/90000 a=fmtp:101 apt=100 a=rtpmap:127 red/90000 a=rtpmap:124 rtx/90000 a=fmtp:124 apt=127 a=rtpmap:125 ulpfec/90000 a=ssrc-group:FID 1986786933 3633893868 a=ssrc:1986786933 cname:/gdc7e9aCW6c2BOi a=ssrc:1986786933 msid: 2a9e03ed-b2a7-459e-9000-29fb331b317c a=ssrc:1986786933 mslabel: a=ssrc:1986786933 label:2a9e03ed-b2a7-459e-9000-29fb331b317c a=ssrc:3633893868 cname:/gdc7e9aCW6c2BOi a=ssrc:3633893868 msid: 2a9e03ed-b2a7-459e-9000-29fb331b317c a=ssrc:3633893868 mslabel: a=ssrc:3633893868 label:2a9e03ed-b2a7-459e-9000-29fb331b317c m=audio 9 UDP/TLS/RTP/SAVPF 111 103 104 9 102 0 8 106 105 13 110 112 113 126 c=IN IP4 0.0.0.0 a=rtcp:9 IN IP4 0.0.0.0 a=ice-ufrag:6rrQ a=ice-pwd:oDzzm8LmH4EkllD/aUVD0XdP a=ice-options:trickle a=fingerprint:sha-256 AE:7C:7A:0E:B6:97:46:DD:48:94:3A:6C:BC:28:17:D6:60:F5:E2:86:00:AB:AF:E0:85:F3:75:BE:88:4F:4D:80 a=setup:actpass a=mid:1 a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level a=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid a=sendrecv a=msid:- 810acb68-ba4f-40b5-8a75-7c28e26cacfa a=rtcp-mux a=rtpmap:111 opus/48000/2 a=rtcp-fb:111 transport-cc a=fmtp:111 minptime=10;useinbandfec=1 a=rtpmap:103 ISAC/16000 a=rtpmap:104 ISAC/32000 a=rtpmap:9 G722/8000 a=rtpmap:102 ILBC/8000 a=rtpmap:0 PCMU/8000 a=rtpmap:8 PCMA/8000 a=rtpmap:106 CN/32000 a=rtpmap:105 CN/16000 a=rtpmap:13 CN/8000 a=rtpmap:110 telephone-event/48000 a=rtpmap:112 telephone-event/32000 a=rtpmap:113 telephone-event/16000 a=rtpmap:126 telephone-event/8000 a=ssrc:1062114428 cname:/gdc7e9aCW6c2BOi a=ssrc:1062114428 msid: 810acb68-ba4f-40b5-8a75-7c28e26cacfa a=ssrc:1062114428 mslabel: a=ssrc:1062114428 label:810acb68-ba4f-40b5-8a75-7c28e26cacfa
Answer message v=0 o=- 5248938564854636631 2 IN IP4 127.0.0.1 s=- t=0 0 a=group:BUNDLE 0 1 a=msid-semantic: WMS m=video 9 UDP/TLS/RTP/SAVPF 96 97 98 99 100 101 127 124 125 c=IN IP4 0.0.0.0 a=rtcp:9 IN IP4 0.0.0.0 a=ice-ufrag:lmnv a=ice-pwd:xRcMuINw51ho8mzR+Q3Gpja6 a=ice-options:trickle a=fingerprint:sha-256 42:44:3F:B9:46:7C:D8:E8:6B:45:4D:98:C1:BD:6E:63:86:50:96:80:3F:B7:D6:4C:B5:AD:4E:DA:1F:47:3A:F9 a=setup:active a=mid:0 a=extmap:2 urn:ietf:params:rtp-hdrext:toffset a=extmap:3 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time a=extmap:4 urn:3gpp:video-orientation a=extmap:5 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01 a=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type a=extmap:8 http://www.webrtc.org/experiments/rtp-hdrext/video-timing a=extmap:10 http://tools.ietf.org/html/draft-ietf-avtext-framemarking-07 a=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid a=recvonly a=rtcp-mux a=rtcp-rsize a=rtpmap:96 VP8/90000 a=rtcp-fb:96 goog-remb a=rtcp-fb:96 transport-cc a=rtcp-fb:96 ccm fir a=rtcp-fb:96 nack a=rtcp-fb:96 nack pli a=rtpmap:97 rtx/90000 a=fmtp:97 apt=96 a=rtpmap:98 VP9/90000 a=rtcp-fb:98 goog-remb a=rtcp-fb:98 transport-cc a=rtcp-fb:98 ccm fir a=rtcp-fb:98 nack a=rtcp-fb:98 nack pli a=fmtp:98 x-google-profile-id=0 a=rtpmap:99 rtx/90000 a=fmtp:99 apt=98 a=rtpmap:100 multiplex/90000 a=rtcp-fb:100 goog-remb a=rtcp-fb:100 transport-cc a=rtcp-fb:100 ccm fir a=rtcp-fb:100 nack a=rtcp-fb:100 nack pli a=fmtp:100 acn=VP9;x-google-profile-id=0 a=rtpmap:101 rtx/90000 a=fmtp:101 apt=100 a=rtpmap:127 red/90000 a=rtpmap:124 rtx/90000 a=fmtp:124 apt=127 a=rtpmap:125 ulpfec/90000 m=audio 9 UDP/TLS/RTP/SAVPF 111 103 104 9 102 0 8 106 105 13 110 112 113 126 c=IN IP4 0.0.0.0 a=rtcp:9 IN IP4 0.0.0.0 a=ice-ufrag:lmnv a=ice-pwd:xRcMuINw51ho8mzR+Q3Gpja6 a=ice-options:trickle a=fingerprint:sha-256 42:44:3F:B9:46:7C:D8:E8:6B:45:4D:98:C1:BD:6E:63:86:50:96:80:3F:B7:D6:4C:B5:AD:4E:DA:1F:47:3A:F9 a=setup:active a=mid:1 a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level a=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid a=recvonly a=rtcp-mux a=rtpmap:111 opus/48000/2 a=rtcp-fb:111 transport-cc a=fmtp:111 minptime=10;useinbandfec=1 a=rtpmap:103 ISAC/16000 a=rtpmap:104 ISAC/32000 a=rtpmap:9 G722/8000 a=rtpmap:102 ILBC/8000 a=rtpmap:0 PCMU/8000 a=rtpmap:8 PCMA/8000 a=rtpmap:106 CN/32000 a=rtpmap:105 CN/16000 a=rtpmap:13 CN/8000 a=rtpmap:110 telephone-event/48000 a=rtpmap:112 telephone-event/32000 a=rtpmap:113 telephone-event/16000 a=rtpmap:126 telephone-event/8000

Environment Please fill the information for each peer if different

djee-ms commented 3 years ago

which seems strange to me because the transceivers are being created before sending the answer.

Yes that's the issue. This is very unintuitive, but to the best of my knowledge trying to make that work for weeks 1) the standard doesn't allow adding more transceivers in an answer in addition of the ones already present in the offer; and 2) the transceivers need to be created by the caller (create offer side) only, and will be automatically created internally on the callee side (receive offer side) based on the content of the offer. You cannot pre-create some transceiver and hope they will be somehow matched with the ones that the caller advertize in its offer; there is no mechanism that I know of to do that. You need to wait for the "apply remote offer" negotiation step to create those transceivers on the receiving side for you, then you can attach tracks to them and change direction. Whether this can be done before sending the answer or needs a second negotiation, I cannot remember.

anpep commented 3 years ago

which seems strange to me because the transceivers are being created before sending the answer.

Yes that's the issue. This is very unintuitive, but to the best of my knowledge trying to make that work for weeks 1) the standard doesn't allow adding more transceivers in an answer in addition of the ones already present in the offer; and 2) the transceivers need to be created by the caller (create offer side) only, and will be automatically created internally on the callee side (receive offer side) based on the content of the offer. You cannot pre-create some transceiver and hope they will be somehow matched with the ones that the caller advertize in its offer; there is no mechanism that I know of to do that. You need to wait for the "apply remote offer" negotiation step to create those transceivers on the receiving side for you, then you can attach tracks to them and change direction. Whether this can be done before sending the answer or needs a second negotiation, I cannot remember.

Thank you for you answer. I've been looking into the issue and attempted to use a SendOnly video transceiver from my desktop application to the HoloLens application. The SDP answer seems to be just fine but no media gets rendered on the HoloLens. Maybe this has something to do with the HoloLens being the device that initiates the offer and my application being the one who sends the answer? Or perhaps I need to perform a twofold negotiation in which both devices initiate an offer and send an answer? In this case, how do I perform this kind of negotiation while keeping the same WebRTC session on both ends?

I've been looking into the WebRTC API for browsers and this requires no SDP gymnastics. Do browsers handle this transparently to the programmer and has no validity in this scenario?

Thanks for your help

yyanhan commented 3 years ago

have you solved it yet?

thanks

Matteo-0 commented 3 years ago

Please @anpep if you solved the problem, can you share the solution?

anpep commented 3 years ago

@yyanhan @Matteo-0 I'm so sorry for the late answer. I could not solve this issue. Instead, I ended up using Unity and MixedReality-WebRTC for both peers (instead of using a custom Desktop App at one end). Worked flawlessly this way.