Get MediaStream from LocalVideoStream and RemoteVideoStream

[StefanoGermani]

I'm using the JS SDK of Azure Communication Service to create a video conference on a web app.

I need to merge video streams from the different partecipants, add creative effects and restream the result somewhere else.

I would like to use video-stream-merger but I cannot find a way to access the low level MediaStream from LocalStreamVideo or RemoteStreamVideo to merge them.

Is there a way to do it?

[LaithRodan]

RAW media access support is part of the backlog and develpomnet is on going. No committed ETA date as of now for JS SDK,

[tompi]

Need this on iOS and android. Would like to do realtime image analysis while streaming video. We had an existing algorithm we had to turn off in favor of acs

[LaithRodan]

Our backlog item includes RAW media access web (JS), Android, iOS and Windows calling SDK. @StefanoGermani , @tompi : for your specific use case, how do you define a RAW media stream? Accessing local RAW media should be straight forward, accessing remote RAW media depends on server side coding and multiplexing.

[tompi]

@LaithRodan yeah, i mean getting byte arrays of local camera feed. Only local camera access is relevant for my use case

[StefanoGermani]

@LaithRodan In my use case, I would like access to the MediaStream of both LocalVideoStream and RemoteVideoStream.

[taskbit]

@LaithRodan We also need a way to manipulate/replace the underlying MediaStreamfor audio & video. See the example below for providing audio gain control. This is currently not possible with ACS.

   private async getAudioStream(deviceId: string) {
      const stream = await navigator.mediaDevices.getUserMedia({ video: false, audio: { deviceId } });
      const ctx = new AudioContext();
      const source = ctx.createMediaStreamSource(stream);
      const gain = ctx.createGain();
      gain.gain.setValueAtTime(1, ctx.currentTime);
      source.connect(gain);
      const dest = ctx.createMediaStreamDestination();
      gain.connect(dest);
      return dest.stream;
   }

Furthermore we wanted to add ACS to our list of supported call providers, but this seems to be diffult because most WebRTC SDKs are centered around Streams. So our abstractions are based on streams and not devices (like in the ACS SDK)

[AdonaiVera]

I'm interested in this issue, I would like to apply computer vision algorithms and augmented reality to only local camera, but I can't get the video stream to apply this algorithm before send it to ACS. It will be great if we can take the stream to pre-process and then send it.

[tom-lokulus]

Hi, my team and I are investigating the use of ACS as part of a customer service solution we want to bring to market, we have a requirement to composite together multiple video streams and then share them via an ACS call. @LaithRodan or anyone else from Microsoft I don't think there is an alternative way to achieve what we need to do? Alternatively, Could you give us an idea of where this is within your backlog and when / if it's likely to be available for us to use?

[tom-lokulus]

@ProbablePrime - just noticed that you closed this issue down, does this mean that there is no intention of this becoming a feature?

[ProbablePrime]

@tom-lokulus We've migrated all features to https://feedback.azure.com/d365community/forum/81ff6d2b-0c25-ec11-b6e6-000d3a4f0858 where you can track them. We have not removed this from consideration we've just moved its location.