Share video and screen share at a same time.

[Niki-Mehta21]

Before filing an issue please check that the issue is not already addressed by the following:

• Video Guides
• Changelog.

If this is an issue with the QuickStart itself, file it here. If this is an issue with the SDK or how to do something with the SDK please use twilio-video-ios instead.

Please ensure that you are not sharing any Personally Identifiable Information(PII) or sensitive account information (API keys, credentials, etc.) when reporting an issue.

Description

[Description of the issue]

Steps to Reproduce

1. [Step one]
2. [Step two]
3. [Insert as many steps as needed]

Code

// Code that helps reproduce the issue

Expected Behavior

[What you expect to happen]

Actual Behavior

[What actually happens]

Reproduces How Often

[What percentage of the time does it reproduce?]

Logs

Debug level logs are helpful when investigating issues. To enable debug level logging, add the following code to your application:

TwilioVideoSDK.setLogLevel(.debug)

// Log output when the issue occurs

Versions

All relevant version information for the issue.

Video iOS SDK

[e.g. 1.3.12 via CocoaPods]

Xcode

[e.g. 9.2]

iOS Version

[e.g. 11.2.6]

iOS Device

[e.g. iPhone 8 Plus]

[timrozum]

@Niki-Mehta21 can you provide more details about this issue?

It is possible to share camera video anytime the app is in the foreground. But the SDK does not yet support camera access when the app is in the background. Is this your request?

It is also possible to capture video of the app screen when the app is in the foreground. See AppScreenSource.

If you want to capture the screen of the entire system, you could implement a broadcast extension like this example. This will capture the screen of the device even when the app is in the background.

[Niki-Mehta21]

I want to share my video and screen both at a same time , means remote user can able to see my video preview and screen also when I shared.

When I tapped on screenshare , video preview gets stopped , So I publish another video track . So I want detail how to implement that remote user can see our video and screen both at a same time

func ScreenShare(){
        if isScreenShared == true{
            isScreenShared = false
            stopConference(error: nil)

        }else{
           isScreenShared = true
            localParticipant!.publishVideoTrack(screenTrack!)
           startConference()
        }
    }

[RaviTimUI]

Hi @timrozum I am too facing a similar issue. Let me break down the steps for you.

1. We join a room by ataching localVideoTrack and localAudioTrack. Both the video preview and audio works perfectly fine on the remote user's end i.e the 2nd user is able to see the video preview shared by the first user.
2. Later at some point in time, we try to share the screen using AppScreenSource, but it doesn't trigger the screen sharing though we receive the prompt from the OS to allow screen sharing.
3. We came across this link (https://stackoverflow.com/questions/64416384/screen-sharing-using-twilio-in-ios) in stackoverflow which states that if we want to share screen we need to stop publishing the localVideoTrack. But this is something which is not our use-case. We want to share the localVideoTrack as well as the localScreenSharingTrack together to that the other user is able to see the video preview as well as the screen sharing preview.

Code for the local video preview `func startPreview(completionHandler: @escaping (Bool) -> (Void)){

    let frontCamera = CameraSource.captureDevice(position: .front)
    let backCamera = CameraSource.captureDevice(position: .back)

    if (frontCamera != nil || backCamera != nil) {

        let options = CameraSourceOptions { (builder) in
            if #available(iOS 13.0, *) {
                // Track UIWindowScene events for the key window's scene.
                // The example app disables multi-window support in the .plist (see UIApplicationSceneManifestKey).
                builder.orientationTracker = UserInterfaceTracker(scene: UIApplication.shared.keyWindow!.windowScene!)
            }
        }
        // Preview our local camera track in the local video preview view.
        camera = CameraSource(options: options, delegate: self)
        localVideoTrack = LocalVideoTrack(source: camera!, enabled: true, name: "Camera")

        // Add renderer to video track for local preview
        localVideoTrack!.addRenderer(self.localPersonView)
        logMessage(messageText: "Video track created")
        camera!.startCapture(device: frontCamera != nil ? frontCamera! : backCamera!) { (captureDevice, videoFormat, error) in
            if let error = error {
                self.logMessage(messageText: "Capture failed with error.\ncode = \((error as NSError).code) error = \(error.localizedDescription)")
                completionHandler(false)
            } else {
                self.localPersonView.shouldMirror = (captureDevice.position == .front)
                completionHandler(true)
            }
        }
    }
    else {
        self.logMessage(messageText:"No front or back capture device found!")
        completionHandler(false)
    }
}`

Code to connect to the room ` func connectRoom() { let connectOptions = ConnectOptions(token: token) { [self] (builder) in builder.roomName = roomName

        if let audioTrack = self.localAudioTrack {
            builder.audioTracks = [ audioTrack ]
        }else{
            print("Local audioTrack not available")
        }
        if let dataTrack = self.localDataTrack {
            builder.dataTracks = [ dataTrack ]
        }
        if let videoTrack = localVideoTrack {
            builder.videoTracks = [ videoTrack ]
        }else{
            print("Local videoTrack not available")
        }
    }
    room = TwilioVideoSDK.connect(options: connectOptions, delegate: self)
}

` Code to start the screen sharing

` private func startConference() { // Start recording the screen. let options = AppScreenSourceOptions() { builder in builder.screenContent = VideoViewController.kDownscaleBuffers ? .video : .default }

    appScreenSource = AppScreenSource(options: options, delegate: self)

    screenTrack = LocalVideoTrack(source: appScreenSource!,
                                  enabled: true,
                                  name: "Screen-Share")

    appScreenSource?.startCapture { error in
        if error != nil {
            print("Screen capture error: ", error!)
        } else {
            print("Screen capture started.")
            self.room?.localParticipant?.publishVideoTrack(self.screenTrack!)
        }
        if error != nil {
            self.appScreenSource = nil
            self.screenTrack = nil
        }
    }
}

`

Please guide us more on how we can achieve this.

[RaviTimUI]

@timrozum @maxterry @bmctigue Can anyone of you please help us on this?

[timrozum]

@RaviTimUI @Niki-Mehta21 it should be possible to have a camera track and screen track at the same time without issue.

@RaviTimUI I looked at your code and did not find any problems.

I will test this myself and let you know what I find.

[timrozum]

@RaviTimUI @Niki-Mehta21 I have confirmed in a reference app that I can publish the screen and camera at the same time.

Here are the changes I made to test this: https://github.com/twilio/twilio-video-app-ios/compare/spike/screen-and-camera-capture?expand=1

Note that I used a reference app in a different repo and not the quickstart. The code in this other reference app repo is closer to a production app than the quickstarts so you may find the example useful.

Here is a screenshot of the test:

Alice is on an iPhone with camera enabled and screen shared. I took the screenshot for user Bob from a web app that is connected to the same room.

Hopefully this fully working example helps you to get this working in your app.

[timrozum]

@RaviTimUI @Niki-Mehta21 also I posted similar explanation on stack overflow: https://stackoverflow.com/questions/64416384/screen-sharing-using-twilio-in-ios/72904394#72904394

Appreciate an upvote if you are able to. 😄

[timrozum]

Going to close this but let me know if there are more questions.