search

microsoft / MixedReality-WebRTC

MixedReality-WebRTC is a collection of components to help mixed reality app developers integrate audio and video real-time communication into their application and improve their collaborative experience
https://microsoft.github.io/MixedReality-WebRTC/
MIT License
909 stars 283 forks source link

While connected, switch WebcamSource to SceneVideoSource and send to remote peer. #691

Open yasuaki-kan opened 3 years ago

yasuaki-kan commented 3 years ago

I am displaying both of the following videos on the local peer side. A. Camera video captured by WebcamSource B. Scene video captured by SceneVideoSouce

Now I would like to implement the following functions.

  1. When switching between A and B on the local peer side, the remote peer side is notified of the change.
  2. Based on the notification, the remote peer side will switch between A and B videos. I have already implemented the functionality of 1. What I want to know is how to implement 2.

I think it corresponds to 2 in your comment below, but I don't know how to implement it specifically. https://github.com/microsoft/MixedReality-WebRTC/issues/139#issuecomment-562630215

dgtvan commented 3 years ago

According to my understanding about the 2nd approach, I beleive @djee-ms does not mean to take any action on the remote peer. There is ony one video streaming channel between peers, you can simply change the video source on one peer, and another peer does not need to care about that.

Please take a look at my illustration: download

I have not tried this approach so I can not give you code sample.

djee-ms commented 3 years ago

That's the idea @ThaiVan but your diagram should be the other way around, webcam and scene are video sources (the thing that produces frames), which are producing frames for a video track (the frame container abstraction), which is attached to a transceiver (the transport and encoding abstraction) to be sent to the remote peer.

|--------------------local peer-------------------|                        |-----------remote peer-----------|
webcam source ----> sender track ----> transceiver ======= network =======> transceiver ----> receiver track

If you want to switch between the two sources, conceptually you can create a track per source, and hot-swap the track on the transceiver. In C++ and C# this is done by assigning the transceiver's LocalVideoTrack property.

In Unity this is done slightly differently, because of the higher-level abstraction, by setting the MediaTrackSource property, which will automatically delete the track associated with the old source, and create a new track for the new source (so you can't keep the tracks alive since MediaLine is managing them for you, but a track is lightweight so that shouldn't be a problem).

Note that this way there is no need to notify anyone; in fact the remote peer cannot know about the change, so if your app needs to know about it then you need separate logic, for example using data channels.