microsoft / MixedReality-WebRTC

MixedReality-WebRTC is a collection of components to help mixed reality app developers integrate audio and video real-time communication into their application and improve their collaborative experience
https://microsoft.github.io/MixedReality-WebRTC/
MIT License
898 stars 277 forks source link

Different behaviour between UnityEditor and Hololens: VideoSource stop to send video if other peer want send video #828

Open Qualcuno opened 2 years ago

Qualcuno commented 2 years ago

I developed a communication with Hololens 2 and a webrtc webpage ( passing through an SFU ) and i have 2 operative mode: Hololens ( or UnityEditor ) always offer an audio and a video communication and Web can (A) offer audio and video communication (B) offer just audio communication ( no video needed ).

When i try to start a communication with Unity Editor and the Web, all works fine ( both with (A) and (B) mode works well ) If i try to start a communication with Hololens, something wrong if web want to send video ( (A) mode ) but works well if web not send video ( (B) mode).

In the details, if the web want to send video to Hololens, Hololens not start or stop to send video to web. But this does not appen in Unity Editor! In Editor the same code works well!.

I manipulate the Hololens frame before put it into stream and i saw that no OnIncomingI420Frame or OnIncomingARGB32Frame is called.

I checked if any transceivers or tracks were not properly created with this code private void Update() { if (peer != null && peer.Peer != null) {

        string info = "INFO";

        for (int i = 0; i < peer.Peer.Transceivers.Count; i++)
        {
            Microsoft.MixedReality.WebRTC.Transceiver t = upeer.Peer.Transceivers[i];
            info += "\nTransceiver " + t.MediaKind + " [" + t.MlineIndex + "] " + t.StreamIDs[0] + " " + t.DesiredDirection + " -> " + t.NegotiatedDirection; 
        }

        for (int i = 0; i < peer.Peer.Transceivers.Count; i++)
        {
            Microsoft.MixedReality.WebRTC.Transceiver t = peer.Peer.Transceivers[i];
            if (t.LocalVideoTrack != null)
            {
                info+= "\nLocalVideoTrack " + t.LocalVideoTrack.Name + " enabled: " + t.LocalVideoTrack.Enabled;
            }
        }
       Debug.Log(info);
    }
}

but the result is the same both UnityEditor and Hololens

_Transcevier Kind: Audio, mlineindex[0], streamId: mixedmslabel, Desired: SendReceiver Negotiated: SendRecevie Transcevier Kind: Video, mlineindex[1], streamId: mixedmslabel, Desired: SendReceiver Negotiated: SendRecevie Transcevier Kind: Audio, mlineindex[3], streamId: ....random_Stream_id_1...., Desired: SendReceiver Negotiated: ReceiveOnly Transcevier Kind: Video, mlineindex[4], streamId: ....random_Stream_id_2...., Desired: SendReceiver Negotiated: ReceiveOnly LocalVideoTrack ...random_Track_id_1... enabled: True LocalVideoTrack ...random_Track_id2... enabled: True

( where mixedmslabel is an empty stream presents into SFU and MLineIndex 2 is not presents becouse is the datachannel). I don't know why the MLineIndex 3 and 4 are ReceiveOnly but this output is the same of UnityEditor where web side correctly receive video.

Why VideoSource stop to send video?

Unity 2020.1.17f1 WebRTC 2.0.2