search

microsoft / MixedReality-WebRTC

MixedReality-WebRTC is a collection of components to help mixed reality app developers integrate audio and video real-time communication into their application and improve their collaborative experience
https://microsoft.github.io/MixedReality-WebRTC/
MIT License
913 stars 283 forks source link

Can´t display video stream from SceneVideoSource or UniformColorVideoSource #780

Open SilverLive opened 3 years ago

SilverLive commented 3 years ago

Hi, I can´t display any video stream from SceneVideoSource or UniformColorVideoSource. Is there anything specific with this sources where I have to pay attention to ? I already added the ARGB Material to the corresponding meshRenderers and pairing should be successfully since the webcam-stream I send back is beeing received correctly on the other side.

spacecheeserocks commented 3 years ago

You might want to try using I420 instead of ARGB. Various parts of this library don't actually support ARGB frames at all (despite classes having events/functions for them).

It won't "look right", but even if you poorly convert the ARGB data into I420 data (eg. grayscale only, or just jam the data through as if it was I420), you'll quickly see if you're getting frames or not.

SilverLive commented 3 years ago

I tried to just receive anything and pass it to a I420A-Material-VideoRenderer, but it seems like any missmatch of the sender- and receivers VideoEncoding result in a black image. Even if I define the wrong render-material to my WebCamSource´s LocalVideoRenderer, the resulting image is black.

Since all predefined CustomVideoSource Implementations are using Argb32VideoFrameSorage I hope that MR-WebRTC actually is able to run with those in the correct way.

To find a solution for my problem I again did a deep debugging of the whole mr-WebRTC package. With that I found out that every Transceiver in "_nativePeer.AssociatedTransceivers" (which is used in the Method "HandleConnectionMessageAsync(SdpMessge message)" within "PeerConnection.cs") has the property "RemoteVideoTrack" who´s parameter "FrameEncoding" is always set to I420A . The property is readOnly. This results in the situation, that the pairing is done with a wrong encoded RemoteVideoTrack. This finally makes the remote VideoRenderer try to process each Frame with a I420A Encoding. Now, If I am right with my assumption that every wrong encoded frame results in a black frame, this might be the reason for my problem.

So now to my Question :) How can I make the _nativePeer create Transceivers with an Argb32-encoded remoteVideoTrack? Is there any config-file which I didn´t find so far or anything else that could help? I really appreciate any comment, since I've been attached to the problem for a long time now and I need to find a solution for that.

SilverLive commented 3 years ago

Can´t anybody help me in this?