Unable to see video stream on the Web from my custom iOS app.

Unable to see video stream on the Web from my iOS app.

Description

In my project I need to stream live videos (conferencing) with some custom camera controls. I had setup a custom AV stack in order to have the full control over my camera (change resolution, fps, etc.). As a streaming solution I've picked Twilio. According to the documentation and some examples, if I use a custom AV stack, then I need to use Sinks.

The problem is with the following: when I stream from iPhone and run a VideoQuickStartiOS example app on the simulator on my Mac, then I can see the video and hear audio from my device, which is absolutely great. But, when I connect to the same room from the Web interface, then I can ONLY hear my audio and video is not visible. Room is configured using Twilio CLI and the interesting fact is, when I connect from the Video Community example app to the room the video is visible on the web. Trying to figure out what is the difference and why it's not working using TVIVideoSink from my device.

Steps to Reproduce

Generate a new room from the React example
Setup a simple AVFoundation stack and use code snippets from bellow
Wonder why Video is visible in the VideoQuickStart app and not on the Web.

Code

//Twilio basic setup
private func setupTwilio() {

        if (audioTrack == nil) {
            audioTrack = LocalAudioTrack(options: nil, enabled: true, name: "Microphone")
        }

        // Create a video track which captures from the camera.
        if (videoTrack == nil) {
            guard let device = activeInput?.device else { return }

            videoTrack = LocalVideoTrack(source: self)

            let config = CameraConfigFactory().makeCameraConfigFactory(captureDevice: device)
            self.requestOutputFormat(config.outputFormat)        }

        // Preparing the connect options with the access token that we fetched (or hardcoded).
        let connectOptions = ConnectOptions(token: accessToken) { (builder) in

            if let audioTrack = self.audioTrack {
                builder.audioTracks = [audioTrack]
            }
            if let videoTrack = self.videoTrack {
                builder.videoTracks = [videoTrack]
            }

            builder.roomName = "Test"
        }

        // Connect to the Room using the options we provided.
        room = TwilioVideoSDK.connect(options: connectOptions, delegate: self)
    }

private func videoOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
    guard let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else {
        debugPrint("Can't create pixel buffer")
        return
    }

    let time = CMSampleBufferGetPresentationTimeStamp(sampleBuffer)
    let frame = VideoFrame(timestamp: time, buffer: pixelBuffer, orientation: VideoOrientation.up)
    sink?.onVideoFrame(frame!)
}

// MARK:- VideoSource
extension CameraController: VideoSource {
    var isScreencast: Bool {
        // We want fluid AR content, maintaining the original frame rate.
        return false
    }

    func requestOutputFormat(_ outputFormat: VideoFormat) {
        if let sink = sink {
            sink.onVideoFormatRequest(outputFormat)
        }
    }
}

//Camera config
struct CameraConfig {
    let outputFormat: VideoFormat
    let inputFormat: VideoFormat
}

class CameraConfigFactory {
    func makeCameraConfigFactory(captureDevice: AVCaptureDevice) -> CameraConfig {
        var targetSize: CMVideoDimensions {
            return CMVideoDimensions(width: 544, height: 480)
        }

        var frameRate: UInt {
            return 24 // With simulcast enabled there are 3 temporal layers, allowing a frame rate of {f, f/2, f/4}
        }

        let cropRatio = CGFloat(targetSize.width) / CGFloat(targetSize.height)
        let preferredFormat = selectVideoFormatBySize(captureDevice: captureDevice, targetSize: targetSize)
        preferredFormat.frameRate = min(preferredFormat.frameRate, frameRate)

        let cropDimensions: CMVideoDimensions

        if preferredFormat.dimensions.width > preferredFormat.dimensions.height {
            cropDimensions = CMVideoDimensions(
                width: Int32(CGFloat(preferredFormat.dimensions.height) * cropRatio),
                height: preferredFormat.dimensions.height
            )
        } else {
            cropDimensions = CMVideoDimensions(
                width: preferredFormat.dimensions.width,
                height: Int32(CGFloat(preferredFormat.dimensions.width) * cropRatio)
            )
        }

        let outputFormat = VideoFormat()
        outputFormat.dimensions = cropDimensions
        outputFormat.pixelFormat = preferredFormat.pixelFormat
        outputFormat.frameRate = 0

        return CameraConfig(outputFormat: outputFormat, inputFormat: preferredFormat)
    }

    private func selectVideoFormatBySize(captureDevice: AVCaptureDevice, targetSize: CMVideoDimensions) -> VideoFormat {
        let supportedFormats = Array(CameraSource.supportedFormats(captureDevice: captureDevice)) as! [VideoFormat]

        // Cropping might be used if there is not an exact match
        for format in supportedFormats {
            guard
                format.pixelFormat == .formatYUV420BiPlanarFullRange &&
                    format.dimensions.width >= targetSize.width &&
                    format.dimensions.height >= targetSize.height
                else {
                    continue
            }

            return format
        }

        fatalError()
    }
}

Expected Behavior

Video should be visible on the web version of the room

Actual Behavior

Video is not visible on the web version of the room. Looks like it is turned OFF, but it's visible in the iOS VideoQuickStart app.

Reproduces How Often

Always

Logs

Debug level logs are helpful when investigating issues. To enable debug level logging, add the following code to your application:

Versions

All relevant version information for the issue.

Video iOS SDK

pod 'TwilioVideo', '4.0.0-beta1'

Xcode

Version 11.7 (11E801a)

iOS Version

iOS 13.7

iOS Device

iPhone X

twilio / video-quickstart-ios