Want to Broadcast my Device screen in the same Room which is connected

[sugeetgoyal]

I want to implement share screen functionality in one to one call using Replay Kit. I have seen into the Replay Kit Example project where the BroadcastExtension is used and it create another Room when user want to share the screen.

In my case I have already a connected room and in the same room I want to share the Device Screen. Please help me to get this done if possible.

Steps to Reproduce

1. Connect Room from video call from main project
2. To share the device screen connect room from Broadcast extension

Expected Behavior

Screen share should happen in the same room without creating another room

Actual Behavior

The existing room is getting disconnected and another room is created for Screen Share

Reproduces How Often

100%

Versions

All relevant version information for the issue.

Video iOS SDK

[e.g. 1.3.12 via CocoaPods]

Xcode

12.4

iOS Version

iOS 14.7.1

iOS Device

Any

[timrozum]

Hi @sugeetgoyal I have a few questions:

1. So you have a single app that has the main app connect the user to the video room and then the broadcast extension is used to share the screen to the same video room correct?
2. Just to make sure, have you verified that the main app and the broadcast extension are using the same room name in the connect options?
3. How are you generating access tokens for the main app and the broadcast extension? If you are using the same user identity for both access tokens that could be the problem since user identity must be unique for each participant in a room. To make them unique you could use something like Bob for the main app and Bob-screen for the broadcast extension. It is unfortunate that there needs to be 2 room connections on one device but it is the simplest way to solve this given the main app and the broadcast extension run in separate processes.

Hopefully this helps!

[sugeetgoyal]

Hi Tim

Thanks for the reply.

1. Yes
2. Yes I want to use the same Room so I am giving the same name
3. I am using the same access token since I have one one access token for one user to join any room, this access token I am fetching from API call.

I don't have to fetch 2 access tokens for the same user in my app. Is there a way to link the screen share track into the existing room of the user ?

Thanks & Regards *Sugeet Goyal (*iOS Developer) Mobile- 9739535002

On Sat, Dec 4, 2021 at 2:54 AM Tim Rozum @.***> wrote:

Hi @sugeetgoyal https://github.com/sugeetgoyal I have a few questions:

1. So you have a single app that has the main app connect the user to the video room and then the broadcast extension is used to share the screen to the same video room correct?
2. Just to make sure, have you verified that the main app and the broadcast extension are using the same room name in the connect options?
3. How are you generating access tokens for the main app and the broadcast extension? If you are using the same user identity for both access tokens that could be the problem since user identity must be unique for each participant in a room. To make them unique you could use something like Bob for the main app and Bob-screen for the broadcast extension. It is unfortunate that there needs to be 2 room connections on one device but it is the simplest way to solve this given the main app and the broadcast extension run in separate processes.

Hopefully this helps!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/twilio/twilio-video-ios/issues/217#issuecomment-985837982, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACLIEWZ3LZYEI7FSJM63R3LUPEYSVANCNFSM5JKTAX2A .

[timrozum]

Ok I think probably what is happening is that when the broadcast extension participant connects to the video room the main app participant is disconnected because they have the same user identity.

I know in reality it is the same user but because there are 2 connections to the room there are basically 2 video room participants connecting from the same device.

One option is to use different access tokens with different identities like I mentioned above. And then for other users receiving the screen share you could have code that basically consolidates the 2 participants into 1 in the UI based on info encoded in the user identities.

There is option 2 but it is complex. Instead of the broadcast extension connecting to the video room it could send video frames to the main app and then the main app could upload the frames with a new video track on the already connected participant. The challenge is that inter-process communication is not easy and we are not able to share source code for an example. I would probably only recommend this solution if you run into performance or quality issues on some large screen devices like iPad due to memory limitations of broadcast extensions. This is the only way to work around those performance issues. Otherwise I would probably recommend the solution in the quickstart.

If you want more info on option 2 let me know and I can send you a few more details to point you in the right direction. But unfortunately we don't have a full example.

[sugeetgoyal]

Can you please provide more info on the Option 2. It will be good if we can get some example on it. Our requirement is to share the screen using same access token in a same room.

[timrozum]

@sugeetgoyal I have emailed you some details for this that should point you in the right direction. I cannot provide example code for this solution.

[sugeetgoyal]

Thanks, I'll check it out.

On Mon, 6 Dec 2021 at 9:47 PM, Tim Rozum @.***> wrote:

@sugeetgoyal https://github.com/sugeetgoyal I have emailed you some details for this that should point you in the right direction. I cannot provide example code for this solution.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/twilio/twilio-video-ios/issues/217#issuecomment-986928163, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACLIEWYU7KO6S3P23OJFYP3UPTOZFANCNFSM5JKTAX2A .

-- Thanks & Regards *Sugeet Goyal (*iOS Developer) Mobile- 9739535002

[sugeetgoyal]

Hello, can we receive remote participant audio while screen sharing using broadcast extension method ?

[timrozum]

Yes you should be able to receive remote participant audio in the main app while the broadcast extension is capturing the screen.

[sugeetgoyal]

Hello, I have implemented the screen sharing using the Option 1 suggested here in the comments above. Now I want to set the screen share track priority to high and I tried below things but it is not working. Can you please help me on this ?

In my App Extension SampleHandler.swift class I have done following code :

    screenTrack = LocalVideoTrack(source: videoSource!,
                                  enabled: true,
                                  name: "Screen")
    videoSource!.requestOutputFormat(outputFormat)

    let connectOptions = ConnectOptions(token: accessToken) { (builder) in
        builder.videoTracks = [self.screenTrack!]
        builder.preferredVideoCodecs = [SampleHandler.kVideoCodec]
        builder.encodingParameters = encodingParams
        builder.isAutomaticSubscriptionEnabled = false
        builder.roomName = roomName
    }
    room = TwilioVideoSDK.connect(options: connectOptions, delegate: self)
    room?.localParticipant?.localVideoTracks.first?.priority = .high

Also I have set the priority in RoomDelegate

func roomDidConnect(room: Room) {        
    room.localParticipant?.localVideoTracks.first?.priority = .high

}

Also I have set the priority in LocalParticipantDelegate

func localParticipantDidPublishVideoTrack(participant: LocalParticipant, videoTrackPublication: LocalVideoTrackPublication) {
    print("localParticipantDidPublishVideoTrack")

    participant.localVideoTracks.first?.priority = .high
    videoTrackPublication.priority = .high
}

None of the above is working

[timrozum]

It seems like this should work. How are you determining if it is working or not?

@piyushtank thoughts?

[sugeetgoyal]

@timrozum I checked in my remote participants who is sharing screen.

func didSubscribeToVideoTrack(videoTrack: RemoteVideoTrack, publication: RemoteVideoTrackPublication, participant: RemoteParticipant) { participant.remoteVideoTracks.first?.publishPriority }

It shows Standard Priority only.

[sugeetgoyal]

@timrozum I was looking for the help to set the screen broadcast video track priority to high. Please suggest me anything which can be done to do that. In Android we are able to set and working.

In case of iOS since the priority is not getting set to High its not showing the screen shared track in correct container of web App.

[timrozum]

@sugeetgoyal can you collect debug logs from the video SDK when this happens? See TwilioVideoSDK.setLogLevel. Thanks.

[timrozum]

@sugeetgoyal I'm going to check with some others about using track priority on the local participant.

However in our video reference app we use track priority so that the server can optimize bandwidth allocations for remote video tracks. So that the server can optimize video that a particular device is displaying. And then we use things like track name (to detect screen track) and dominant speaker status to determine what video track should be displayed in the UI. Here is some code that handles that:

https://github.com/twilio/twilio-video-app-ios/blob/c0b4055bca445e868c84b8bbd9da6f604865e40e/VideoApp/VideoApp/Stores/MainParticipant/MainParticipantStore.swift#L24-L28

[sugeetgoyal]

@timrozum Thanks a lot for being with me for so long here. Thanks for your support.

As mentioned in above code sample in the comments by me, I tried the same way. But don't know When I set priority for the local participant why it is still not updated and not reflecting to the remote participants.

Just after connecting to the room, when the roomDidConnect calls I am setting the room localparticipant track priority to high. Do I need to publish the track again which is already added in participant video tracks again? Please let me know if I am missing anything here? Also please check with the others.

[sugeetgoyal]

Hello @timrozum any update on this ?

[sugeetgoyal]

Hello @timrozum, as you said before did you get a chance to check this issue with others. We are not able to move forward with our screen sharing feature because of this issue. Please help us out to make it done.

Thanks

[timrozum]

@sugeetgoyal I reached out to other SDK engineers and they said they would need to investigate this more. They are tracking this internally but I can't really provide a timeline for more information. It seems that adjusting the priority of a local track may not be a common solution.

The best recommendation I can provide at this time is to use track name instead of track priority to determine how to display each track. The track name would describe what the track was used for such as camera or screen and then there could be logic in the app to handle tracks accordingly in the UI.

[wolfspy]

@sugeetgoyal We are not able to reproduce the problem as you described it. We verified the SDK is behaving as expected in this project and the Quickstart project as well https://github.com/twilio/video-quickstart-ios (it might be easier for you to test this bit of code in that project), so let's go through the working process.

1. Join a room with your first device and set the LocalVideoTrack priority after publishing the video track so after room connection succeeds as you we're doing it already is fine:

   func roomDidConnect(room: Room) {
   // your implementation details        
   room.localParticipant?.localVideoTracks.first?.priority = .high
   }

And setup your logging for the second device:


func didSubscribeToVideoTrack(videoTrack: RemoteVideoTrack, publication: RemoteVideoTrackPublication, participant: RemoteParticipant) {
    // your implementation details
    print("***** didSubscribeToVideoTrack publishPriority: \(String(describing: participant.remoteVideoTracks.first?.publishPriority))")
}

2. While debugging on a second device, join the same room and notice the debug output: 
***** didSubscribeToVideoTrack publishPriority: Optional(__C.TVITrackPriority(_rawValue: TVITrackPriorityHigh))

You can also actively monitor video track publish priority changes by implementing the following delegate callback on your RemoteParticipantDelegate object:

func remoteParticipantDidChangeVideoTrackPublishPriority(participant: TwilioVideo.RemoteParticipant, priority: Track.Priority, publication: RemoteVideoTrackPublication) { }

Then you can try toggling priority on the source device and watching for that callback on the other participant devices.

Note that setting the video track priority of a RemoteParticipant video track DOES NOT trigger the same delegate callback to be fired for all other participants and the priority WILL NOT propagate to other devices.

If for some reason it still doesn’t work, please open a new GitHub issue specific to this problem and provide a sample project that demonstrates the problem.

[manishmg]

Hi @sugeetgoyal @timrozum (https://github.com/sugeetgoyal) , can you help me in screen share, I am using twillio's video call and facing the same issue you faced. From the extension, if I join then it is rejoining again and I am not getting other users' videos In the main app. Can you please suggest how you achieved this?

[sugeetgoyal]

Hello @manishmg We have dropped the screen share feature in our app so I have not completed that research on resolution. May be @timrozum can help here.

[arbazkhan9393]

@manishmg i have solved that issue

[VeriKlickDev]

Is the issue resolved? is there a confirmation before you can close this ticket?

On Wed, Dec 14, 2022 at 8:06 AM arbazkhan9393 @.***> wrote:

@manishmg https://github.com/manishmg i have solved that problem

— Reply to this email directly, view it on GitHub https://github.com/twilio/twilio-video-ios/issues/217#issuecomment-1351320890, or unsubscribe https://github.com/notifications/unsubscribe-auth/A22GYZ4X4D6U2VPUKT2NGTTWNHA4TANCNFSM5JKTAX2A . You are receiving this because you commented.Message ID: @.***>

-- Thanks, VeriKlick Support

[arbazkhan9393]

i am not confirmed yet about it because i am still facing issue. i have done it using the first option mention by @timrozum. when i connect to meeting(1 to 1). i use token to connect to call. when i want to share screen i disconnect from that call and join the another room in broadcast extension with same token(same identity). i have done it using observers.

[arbazkhan9393]

@timrozum Hello, I am unable receive remote participant audio while screen sharing using broadcast extension method. and sometimes to remote participant unable to receive my audio