[Question] Multiple Video Tracks for a Remote Participant

[astaranowicz]

Hi,

Is it possible to have multiple video tracks from a single participant.
As an example: 1) To stream the front camera and the back camera at the same time. 2) To stream the front/back cameras with different FPS and Dimensions.

[ceaglest]

Hi @astaranowicz,

I don't see any problem with publishing multiple video tracks, except that TVICameraSource doesn't support capturing from multiple AVCaptureDevices. If you want to write your own TVIVideoSource that operates AVCaptureSession with dual devices you could probably achieve what you are looking for.

Edit: Actually it would need to be one "source" per sink. You could have some sort of composite source that exposes two TVIVideoSources (one per TVILocalVideoTrack).

Best, Chris

[ceaglest]

Hey @astaranowicz,

I came across some good sample code to capture from multiple devices. Have you seen the AVMultiCamPip example?

It sounds like you want two Tracks not one, so the composition portion of AVMultiCamPip can be ignored.

Best, Chris

[astaranowicz]

Great. I went down the route to have 2 CameraSources (each with a sink) with 2 Video Tracks. What I was confused about was that the remote participant could only have 1 video, 1 audio, and 1 data track. So I didn't think that the remote participant can subscribe to multiple video tracks. What I realized is that the local participant can create as many tracks as they want, and the remote participant can subscribe to multiple, but mainly show just 1 track at at time.

Thank you for the help and example, Aaron

[mattrosemeier2020]

@astaranowicz (cc: @ceaglest ) would you mind showing me how you published multiple video tracks? I modified the VideoQuickStart iOS project and I have two CameraSource objects and am creating an array of two LocalVideoTrack objects:

           let options1 = CameraSourceOptions { (builder) in
                if #available(iOS 13.0, *) {
                    // Track UIWindowScene events for the key window's scene.
                    // The example app disables multi-window support in the .plist (see UIApplicationSceneManifestKey).
                    builder.orientationTracker = UserInterfaceTracker(scene: UIApplication.shared.keyWindow!.windowScene!)
                }
            }

            let options2 = CameraSourceOptions { (builder) in
                if #available(iOS 13.0, *) {
                    // Track UIWindowScene events for the key window's scene.
                    // The example app disables multi-window support in the .plist (see UIApplicationSceneManifestKey).
                    builder.orientationTracker = UserInterfaceTracker(scene: UIApplication.shared.keyWindow!.windowScene!)
                }
            }

            // Preview our local camera track in the local video preview view.
            camera1 = CameraSource(options: options1, delegate: self)
            camera2 = CameraSource(options: options2, delegate: self)

            let tracksArray: [LocalVideoTrack] = [
                LocalVideoTrack(source: camera1!, enabled: true, name: "camera1")!,
                LocalVideoTrack(source: camera2!, enabled: true, name: "camera2")!
            ]

            self.localVideoTracks = tracksArray

I made an additional previewView @IBOutlet type of VideoView so now I have two:

            localVideoTracks![0].addRenderer(self.previewView1)
            localVideoTracks![1].addRenderer(self.previewView2)

The problem occurs when I try to call .startCapture() on the second camera source. Whatever one gets called first works and renders, but not the second one. I am running an iPhone 11 Pro Max iOS13 so I know dual camera capture at the same time is supported:

camera1!.startCapture(device: frontCamera!) { (captureDevice, videoFormat, error) in
                if let error = error {
                    self.logMessage(messageText: "Capture failed with error.\ncode = \((error as NSError).code) error = \(error.localizedDescription)")
                } else {
                    self.previewView1.shouldMirror = (captureDevice.position == .front)
                }
            }

            camera2!.startCapture(device: backCamera!) { (captureDevice, videoFormat, error) in
                if let error = error {
                    self.logMessage(messageText: "Capture failed with error.\ncode = \((error as NSError).code) error = \(error.localizedDescription)")
                } else {
                    self.previewView2.shouldMirror = (captureDevice.position == .back)
                }
            }

The camera2 startCapture() call fails in the closure with a pretty unspecific error:

ERROR:Twilio:[Platform](0x16db7b000): Fatal runtime capture error: Error Domain=AVFoundationErrorDomain Code=-11800 "The operation could not be completed" UserInfo={NSLocalizedFailureReason=An unknown error occurred (-12780), NSLocalizedDescription=The operation could not be completed, NSUnderlyingError=0x2836b9f80 {Error Domain=NSOSStatusErrorDomain Code=-12780 "(null)"}}, code -11800