ReplayKit - Use a single token/identity for Broadcast Extension & App

[kushJadia]

Hi, We have implemented screen sharing with Twilio and its working fine. But one of my client wants to share screen and keep continue call. That means in the app there is on going call and user starts sharing screen from control centre (from iPhone). Application and extension (in xCode to share screen you need to add extension) both are different things and they can not use same videoTrack as both have different target.

Question is: How to share screen from extension and keep continue call from app (With same user) ?

[ceaglest]

Hi @kushJadia,

Question is: How to share screen from extension and keep continue call from app (With same user) ?

In this case you would likely need two Participants, with two distinct tokens for the presenter's device.

1. Extension - Share video from ReplayKit.
2. App - Share microphone, and playback remote audio.

If sharing the screen outside of your app is a hard requirement, then this is probably the easiest solution.

Alternatively, it might be possible to share frames across process from the extension to the main app and have the app connect to the Room and share all the Tracks. I have not tried this approach personally, so I can not speak to its feasibility.

Best, Chris

[KrisConrad]

I'll add a +1 for this feature. I've also had the same feature request recently with the same basic requirements.

We'd like to be able to join a video call, then at some point after we join the call share our screen (not just the app window) to the call and continue to hear the other participants on the call.

Unfortunately sharing data between a broadcast upload extension and it's host app is very limited. Too limited to share streaming video for sure.

I'd argue that connecting two separate participants with two distinct tokens would degrade the User Experience since it would cause one actual participant to have appear as two in the call's user list.

I agree the correct way to implement something like this (given the limitations of what the iOS SDK allows) would be to connect in the app for mic and playback of remote audio and from the extension for screen sharing via ReplayKit. Ideally we would have a way to merge the two connections though so they appear as a single participant to the other participants in the call.

[ceaglest]

Hey @KrisConrad,

I hear you, and I wish that I had a single Participant solution to offer at this time.

We'd like to be able to join a video call, then at some point after we join the call share our screen (not just the app window) to the call and continue to hear the other participants on the call.

Since each process (the extension and the app) needs to have a secure connection with our servers there isn't any way for those to be represented as the same Participant in a Room. Our current usage model requires that identities be unique, so when the second process joins with the same identity as the first, the first process will be disconnected.

Unfortunately sharing data between a broadcast upload extension and it's host app is very limited. Too limited to share streaming video for sure.

I need to research this more before claiming defeat. For video samples, the main data we care about is the IOSurface that backs each CVPixelBuffer. The surfaces are designed to be shared across processes between Apple's frameworks (like mediaserverd) and your app, so if there is any way we could funnel the IDs fast enough...

I'd argue that connecting two separate participants with two distinct tokens would degrade the User Experience since it would cause one actual participant to have appear as two in the call's user list.

As a workaround, you could have some designation that is signaled in the extension Participant's identity, or in the Track names that the app could use to merge the two Participants.

I will gather more feedback from internal stakeholders and get back to you.

Regards, Chris

[kushJadia]

@ceaglest @KrisConrad Thanks for your help.

[sathishvgs]

Hey @KrisConrad,

I hear you, and I wish that I had a single Participant solution to offer at this time.

We'd like to be able to join a video call, then at some point after we join the call share our screen (not just the app window) to the call and continue to hear the other participants on the call.

Since each process (the extension and the app) needs to have a secure connection with our servers there isn't any way for those to be represented as the same Participant in a Room. Our current usage model requires that identities be unique, so when the second process joins with the same identity as the first, the first process will be disconnected.

Unfortunately sharing data between a broadcast upload extension and it's host app is very limited. Too limited to share streaming video for sure.

I need to research this more before claiming defeat. For video samples, the main data we care about is the IOSurface that backs each CVPixelBuffer. The surfaces are designed to be shared across processes between Apple's frameworks (like mediaserverd) and your app, so if there is any way we could funnel the IDs fast enough...

I'd argue that connecting two separate participants with two distinct tokens would degrade the User Experience since it would cause one actual participant to have appear as two in the call's user list.

As a workaround, you could have some designation that is signaled in the extension Participant's identity, or in the Track names that the app could use to merge the two Participants.

I will gather more feedback from internal stakeholders and get back to you.

Regards, Chris

Hi @ceaglest ,

I have disconnected from the App room and joined the room via extension for screen sharing. But the problem is, I can't able to listen the other participants AudioTrack from the extension. It would be really helpful If we get any solution.

Thanks Sathish Kumar G

[sathishvgs]

Hi @ceaglest , We are waiting for more than 3 days. So, can you please check it out.

[ceaglest]

Hi @sathishvgs,

Sorry that I missed your earlier message.

I don’t believe that it’s possible to play audio from an extension, or at least we were not successful in doing so yet.

If you want to play remote audio it should be possible to join from both the extension (for your screen) and in your app at the same time. These techniques are demonstrated separately in the replaykit example app but you could combine them if needed.

Your app would need to account for two Participants and display them in some way that is appropriate.

Regards, Chris

[ceaglest]

Hello ReplayKit developers,

While our current object & auth models don't allow several applications to publish Tracks under a single Participant we have made some improvements to how broadcast extensions work in the 2.10.0 release. Your extension can now connect (as a unique Participant) without subscribing to any Tracks.

If your goal is to have screen sharing in a traditional video conference, then the best solution I can offer at this time is to connect one ReplayKit broadcast extension Participant to share the screen, and another app Participant to publish the camera and microphone, and receive audio/video Tracks.

You can try using the extension with 2.10.0 in our updated ReplayKitExample.

Best, Chris