Intermittent failure in twilioCameraSource.startCapture on iPhone 14 pro/pro max

[kjanderson2]

Before filing an issue please check that the issue is not already addressed by the following:

• Video Guides
• Changelog

If this is an issue with the SDK itself, file it here. If this is an issue with the QuickStart apps, please use video-quickstart-ios.

Please ensure that you are not sharing any Personally Identifiable Information(PII) or sensitive account information (API keys, credentials, etc.) when reporting an issue.

Description

Unfortunately this is a bit tricky to reproduce but we have a lot of reports of the startCapture failing, particularly on the iPhone 14 pro and pro max. I know these devices have had tricky 3rd party issues with other libraries and SDKs so wanted to bring this to Twilio's attention as well. When it does fail, we obviously do not have the camera which is the primary function of our product, making it useless. Would really love this looked into and hopefully fixed.

Steps to Reproduce

1. We attempt to start a capture of the front-facing camera
2. It fails with an unexpected error

Code

// Code that helps reproduce the issue
func startCapture(with cameraSubject: StreemCameraSubject, completion: @escaping ((_ success: Bool) -> Void)) throws {
            let selectedCamera = CameraSource.captureDevice(position: cameraSubject.avCapturePosition)!
            let videoFormats = CameraSource.supportedFormats(captureDevice: selectedCamera) // ordered by increasing size
            var bestFormat: VideoFormat?
            for format in videoFormats {
                guard let theFormat = format as? VideoFormat else { continue }
                if theFormat.dimensions.width <= 1280 && theFormat.dimensions.height <= 720 {
                    bestFormat = theFormat
                }
            }
            guard let preferredFormat = bestFormat else {
                log.error("Unable to find video format ≤ 1280x720")
                throw VideoError.videoFormatNotAvailable
            }

            // The DispatchQueue stuff here, and below in stopCapture(), is based on various StackOverflow discussions
            // re AVCaptureSession.startRunning(), which is what twilioCameraSource.startCapture() ultimately calls.
            // It may not be necessary (ideally, Twilio is already dealing with these issues),
            // and adding it hasn't helped with the video-startup delay,
            // but I see no harm in keeping it for now, just to be paranoid.
            DispatchQueue.global(qos: .background).async {
                let startCaptureTime = Date()
                twilioCameraSource.startCapture(device: selectedCamera, format: preferredFormat) { [weak self] device, format, error in
                    self?.logTimeSince(startCaptureTime, succeeded: error == nil)

                    guard error == nil else {
                        let errorDescription = "\(String(describing: error))"
                        log.error("Unable to start video track: \(errorDescription)")
                        completion(false)
                        return
                    }

                    DispatchQueue.main.async {
                        completion(true)
                    }
                }
            }

private func logTimeSince(_ startCaptureTime: Date, succeeded: Bool) {
        let time = -startCaptureTime.timeIntervalSinceNow
        let timeToNearestTenth = floor(time * 10 + 0.5) / 10
        let message = "twilioCameraSource.startCapture() \(succeeded ? "SUCCEEDED" : "FAILED") after \(timeToNearestTenth) seconds."
        log.info(message)
        DataLogger.info(message)
    }

func stopCapture() {
        DispatchQueue.global(qos: .background).async { [weak self] in
            self?.twilioCameraSource?.stopCapture()
        }
    }

Also we have a StreemCameraSubject defined as

public extension StreemCameraSubject {

    init?(_ wallCameraSubject: Streem_Rooms_CameraSubject) {
        switch wallCameraSubject {
        case .world:
            self = .world
        case .person:
            self = .person
        case .screen:
            self = .screen
        case .UNRECOGNIZED(_):
            return nil
        }
    }

    var avCapturePosition: AVCaptureDevice.Position {
        switch self {
        case .world:
            return .back
        case .person:
            return .front
        case .screen:
            return .unspecified
        }
    }

    var wallCamera: Streem_Rooms_CameraSubject {
        switch self {
        case .world:
            return .world
        case .person:
            return .person
        case .screen:
            return .screen
        }
    }
}

Expected Behavior

This code usually works and begins the front-facing camera flawlessly. It has worked with this same code for a long time with very little adjustment. It also works for almost all devices. However, on the 14 pro max and 14 pro we sometimes see it fail (again, not every time even for the devices that do experience this).

Actual Behavior

startCapture fails and our app has a black screen instead of the desired video feed.

Reproduces How Often

For devices that experience this issue, we probably see it one out of every 4 or 5 times. So 20-25% of the time.

Logs

Debug level logs are helpful when investigating issues. To enable debug level logging, add the following code to your application:

TwilioVideoSDK.setLogLevel(.debug)

^ We can add this in our debug but not our prod version that gets pushed to clients (and that's where we are seeing it the most) so I can share/update the logs when/if we see the issue in our debug env after adding this.

// Log output when the issue occurs
// This is a log we have put into our code
twilioCameraSource.startCapture() FAILED after 2.1 seconds. | StreemRTCCameraSource:logTimeSince(_:succeeded:):80

// and the following are the two other errors that occur when this issue arises
Failed to initialize camera error = Error Domain=AVFoundationErrorDomain Code=-11800 "The operation could not be completed" UserInfo={NSLocalizedFailureReason=An unknown error occurred (-12780), NSLocalizedDescription=The operation could not be completed, NSUnderlyingError=0x2805ff5d0 {Error Domain=NSOSStatusErrorDomain Code=-12780 "(null)"}} | LocalJobViewNonAR:cameraSourceDidFail(source:error:):394

Unable to start video track: Optional(Error Domain=AVFoundationErrorDomain Code=-11800 "The operation could not be completed" UserInfo={NSLocalizedFailureReason=An unknown error occurred (-12780), NSLocalizedDescription=The operation could not be completed, NSUnderlyingError=0x2805ff2a0 {Error Domain=NSOSStatusErrorDomain Code=-12780 "(null)"}}) | StreemRTCCameraSource:startCapture(with:completion:):63

Versions

All relevant version information for the issue.

Video iOS SDK

TwilioVideo 5.0

Xcode

XCode 14.2

iOS Version

iOS v16.x

iOS Device

iPhone 14 pro and iPhone 14 pro max