Question Local video failed to start on SwiftUI iOS14

[hyunoosung]

When receiving call, my local video view is working but it doesn't work on starting a new call.

Below is my code for starting a call

func startCall() {
        if AVAudioSession.sharedInstance().recordPermission == .granted {
            let callees:[CommunicationUserIdentifier] = [CommunicationUserIdentifier(identifier: self.callee)]
            print("Calling \(String(describing: callees))")

            self.call = self.callAgent?.call(participants: callees, options: StartCallOptions())
            self.call?.delegate = self
            self.callClient!.getDeviceManager { (deviceManager, error) in
                if error != nil {
                    print("Failed to get device manager")
                } else {
                    self.deviceManager = deviceManager
                    print("Got device manager")
                }
                self.startLocalVideo()
            }
        } else {
            // Ask permissions
            self.requestRecordPermission()
        }
    }

    func startLocalVideo() {
        if AVCaptureDevice.authorizationStatus(for: .video) == .authorized {
            do {
                let firstCamera: VideoDeviceInfo? = self.deviceManager?.getCameraList()![0]
                let localVideoStream = LocalVideoStream(camera: firstCamera)

                localRenderer = try Renderer(localVideoStream: localVideoStream!)
                localRendererView = VideoStreamView(view: (try? localRenderer?.createView())!)

                self.call?.startVideo(stream: localVideoStream) { (error) in
                    if error != nil {
                        print("Local video started successfully")
                    } else {
                        print("Local video failed to start: \(String(describing: error?.localizedDescription))")
                    }
                }

            } catch {
                print(error.localizedDescription)
            }

        } else {
            self.requestDevicePermission()
        }
    }

and the log is as below

Start call
Calling [<AzureCommunication.CommunicationUserIdentifier: 0x280cf18e0>]
Got device manager
2021-02-03 06:20:04.193763+0900 AzureCommunicationVideoCallingSample[1712:584144] <SkyLibPreviewVideo:0x282765490:not bound>: running 0 -> 1
2021-02-03 06:20:04.193972+0900 AzureCommunicationVideoCallingSample[1712:584144] <SkyLibPreviewVideo:0x282765490:not bound>: updateState: running 1
2021-02-03 06:20:04.194087+0900 AzureCommunicationVideoCallingSample[1712:584144] <SkyLibPreviewVideo:0x282765490:not bound>: createSource
2021-02-03 06:20:04.194178+0900 AzureCommunicationVideoCallingSample[1712:584144] <SkyLibPreviewVideo:0x282765490:not bound>: creating source
2021-02-03 06:20:04.194258+0900 AzureCommunicationVideoCallingSample[1712:584144] <SkyLibPreviewVideo:0x282765490:not bound>: preview binding not bound -> waiting for bind
2021-02-03 06:20:04.194414+0900 AzureCommunicationVideoCallingSample[1712:584144] <PreviewVideoRenderedBindingEvent 0x280c5d6c0> created
2021-02-03 06:20:04.194769+0900 AzureCommunicationVideoCallingSample[1712:584144] [ACSRenderer] previewVideoStatusDidChange
2021-02-03 06:20:04.197354+0900 AzureCommunicationVideoCallingSample[1712:584144] <SkyLibPreviewVideo:0x282765490:waiting for bind>: adding video target: <ACSRendererView: 0x137609d80; frame = (0 0; 0 0); layer = <CALayer: 0x280c5e0a0>>
2021-02-03 06:20:04.197617+0900 AzureCommunicationVideoCallingSample[1712:584144] <SkyLibPreviewVideo:0x282765490:waiting for bind>: addVideoTarget:: video target is being added: <ACSRendererView: 0x137609d80; frame = (0 0; 0 0); layer = <CALayer: 0x280c5e0a0>>
2021-02-03 06:20:04.197879+0900 AzureCommunicationVideoCallingSample[1712:584144] <SKPPreviewVideoRenderer 0x280270990> setView: (null) --> <ACSRendererView: 0x137609d80; frame = (0 0; 0 0); layer = <CALayer: 0x280c5e0a0>>
2021-02-03 06:20:04.197954+0900 AzureCommunicationVideoCallingSample[1712:584144] <SkyLibPreviewVideo:0x282765490:waiting for bind>: videoTargets=1
2021-02-03 06:20:04.198024+0900 AzureCommunicationVideoCallingSample[1712:584144] [ACSRenderer] created view for local video
Remote participants count: 1
onRemoteParticipantsUpdated added: Optional([<ACSRemoteParticipant: 0x280cf32a0>])
**Local video failed to start: nil**
2021-02-03 06:20:04.306239+0900 AzureCommunicationVideoCallingSample[1712:584144] <SKPPreviewVideoRenderer 0x280270990> bindingCreated attaching <ACSRendererView: 0x137609d80; frame = (0 0; 0 0); layer = <CALayer: 0x280c5e0a0>>
2021-02-03 06:20:04.306406+0900 AzureCommunicationVideoCallingSample[1712:584144] <SkyLibPreviewVideo:0x282765490:waiting for bind>: preview binding waiting for bind -> bound
2021-02-03 06:20:04.306474+0900 AzureCommunicationVideoCallingSample[1712:584144] <SkyLibPreviewVideo:0x282765490:bound>: updateState: running 1
2021-02-03 06:20:04.306515+0900 AzureCommunicationVideoCallingSample[1712:584144] [ACSRenderer] previewVideoStatusDidChange
2021-02-03 06:20:04.306552+0900 AzureCommunicationVideoCallingSample[1712:584144] [ACSRenderer] onPreviewBindingAttached
onCallStateChanged: "Connecting"
2021-02-03 06:20:04.621138+0900 AzureCommunicationVideoCallingSample[1712:584144] <SKPPreviewVideoRenderer 0x280270990> Size changed to {720, 1280}
2021-02-03 06:20:04.621389+0900 AzureCommunicationVideoCallingSample[1712:584144] [ACSRenderer] didChangeSize
onCallStateChanged: "Ringing"
onCallStateChanged: "Disconnected"
Remote participants count: 0
onRemoteParticipantsUpdated removed: Optional([<ACSRemoteParticipant: 0x280cd5c80>])
Call ended

Did I do something wrong?

[sankum-msft]

To start a video call, you should create LocalVideoStream beforehand and set it inside StartCallOptions that you are passing to start the call. So the code would look something like following -

func startCall() {
    if AVAudioSession.sharedInstance().recordPermission == .granted && AVCaptureDevice.authorizationStatus(for: .video) == .authorized {
        self.callClient!.getDeviceManager { (deviceManager, error) in
            if error != nil {
                print("Failed to get device manager")
            } else {
                self.deviceManager = deviceManager
                print("Got device manager")

                let firstCamera: VideoDeviceInfo? = self.deviceManager?.getCameraList()![0]
                let localVideoStream = LocalVideoStream(camera: firstCamera)

                localRenderer = try Renderer(localVideoStream: localVideoStream!)
                localRendererView = VideoStreamView(view: (try? localRenderer?.createView())!)

                let callees:[CommunicationUserIdentifier] = [CommunicationUserIdentifier(identifier: self.callee)]
                print("Calling \(String(describing: callees))")

                let options = StartCallOptions()!
                options.videoOptions = VideoOptions(localVideoStream: self.localVideoStream!)!
                self.call = self.callAgent?.call(participants: callees, options: options)
                self.call?.delegate = self
            }
            //self.startLocalVideo()
        }
    } else {
        // Ask permissions
        self.requestRecordPermission()
        self.requestDevicePermission()
    }
}

[hyunoosung]

Thank you so much for your help I could finish my code But I encountered some weird situations call.remoteParticipants on onCallStateChanged event doesn't include remoteStream sometimes. also local video doesn't render occasionally

i'm pasting my viewModel here

please have a look and tell me which causes problem

Thanks in advance!

    func onCallStateChanged(_ call: Call!, args: PropertyChangedEventArgs!) {
        print("\n----------------------------------")
        print("onCallStateChanged: \(String(reflecting: call.state.name))")
        print("----------------------------------\n")
        callState = call.state

        if callState == .connected {
            print("remoteParticipants count: \(String(describing: self.call?.remoteParticipants.count))")
            self.remoteParticipants = call?.remoteParticipants

            call?.remoteParticipants.forEach { (remoteParticipant) in
                if remoteParticipant.identity is CommunicationUserIdentifier {
                    let remoteParticipantIdentity = remoteParticipant.identity as! CommunicationUserIdentifier
                    let remoteParticipantIdentifier = remoteParticipantIdentity.identifier
                    print("RemoteParticipant identifier:  \(String(describing: remoteParticipantIdentifier))")
                    print("RemoteParticipant displayName \(String(describing: remoteParticipant.displayName))")

                    print("\nRemoteVideoStream count for \(String(describing: remoteParticipant.displayName)):  \(remoteParticipant.videoStreams.count)")

                    if !remoteParticipant.videoStreams.isEmpty {
                        print("\nBinding remoteVideoStream for \(String(describing: remoteParticipant.displayName))")
                        do {
                            if let remoteVideoStreams = remoteParticipant.videoStreams {
                                if remoteVideoStreams.count > 0 {
                                    self.remoteRenderer = try Renderer(remoteVideoStream: remoteVideoStreams[0])
                                    self.remoteRendererView = VideoStreamView(view: (try? self.remoteRenderer!.createView())!)
                                }
                            }
                        } catch {
                            print("Failed starting remoteVideoStream for \(String(describing: remoteParticipant.displayName)) : \(error.localizedDescription)")
                        }
                    } else {
                        print("RemoteVideoStream for \(String(describing: remoteParticipant.displayName)) not found.")
                    }
                }
            }
        }

        if callState == .disconnected || callState == .none {
            remoteRenderer?.dispose()
            localRenderer?.dispose()
            call?.stopVideo(stream: localVideoStream!) { error in
                if error != nil {
                    print("Failed to stop localVideoStream.")
                } else {
                    print("LocalVideoStream successfully stoped.")
                }
            }
        }
    }

    func onRemoteParticipantsUpdated(_ call: Call!, args: ParticipantsUpdatedEventArgs!) {
        print("\n---------------------------")
        print("onRemoteParticipantsUpdated")
        print("---------------------------\n")

        var callerIdentifier: String?

        if call.callerId is CommunicationUserIdentifier {
            let callerUserIdentifier = call.callerId as! CommunicationUserIdentifier
            callerIdentifier = callerUserIdentifier.identifier
            print("Caller identifier:  \(String(describing: callerIdentifier))")
        }

        if let addedParticipants = args.addedParticipants {
            if !addedParticipants.isEmpty {
                print("addedParticipants: \(String(describing: args.addedParticipants.count))")

                if callerIdentifier != nil {
                    print("callerIdentifier: \(String(describing: callerIdentifier))")
                    if let callerRemoteParticipant = args.addedParticipants.first(where: {($0.identity as! CommunicationUserIdentifier).identifier == callerIdentifier}) {
                        callerDisplayName = callerRemoteParticipant.displayName ?? "No displayName"
                        print("Incoming callerDisplayName: \(String(describing: callerDisplayName))")
                    }
                }
            }
        }

        if let removedParticipants = args.removedParticipants {
            if !removedParticipants.isEmpty {
                print("removedParticipants: \(String(describing: args.removedParticipants.count))")

                if callerIdentifier != nil {
                    print("callerIdentifier: \(String(describing: callerIdentifier))")
                    if let callerRemoteParticipant = args.removedParticipants.first(where: {($0.identity as! CommunicationUserIdentifier).identifier == callerIdentifier}) {
                        let callerDisplayName = callerRemoteParticipant.displayName ?? "No displayName"
                        print("Removed callerDisplayName: \(String(describing: callerDisplayName))")
                    }
                }
            }
        }
    }

    func onParticipantStateChanged(_ remoteParticipant: RemoteParticipant!, args: PropertyChangedEventArgs!) {
        print("onParticipantStateChanged: \(String(describing: remoteParticipant)) - \(String(describing: args))")
    }

    func startCall() {
        if AVAudioSession.sharedInstance().recordPermission == .granted && AVCaptureDevice.authorizationStatus(for: .video) == .authorized {

            let startCallOptions = StartCallOptions()
            if let localVideoStream = self.localVideoStream {
                do {
                    self.localRenderer = try Renderer(localVideoStream: localVideoStream)
                    self.localRendererView = VideoStreamView(view: (try self.localRenderer!.createView()))
                } catch {
                    print("Failed binding localVideoRenderView")
                }

                let videoOptions = VideoOptions(localVideoStream: localVideoStream)
                startCallOptions?.videoOptions = videoOptions

                let callees:[CommunicationUserIdentifier] = [CommunicationUserIdentifier(identifier: self.callee)]
                self.call = self.callAgent?.call(participants: callees, options: startCallOptions)
                self.call?.delegate = self

                self.call?.startVideo(stream: localVideoStream) { error in
                    if (error != nil) {
                        print("LocalVideoStream failed to start")
                    }
                    else {
                        print("LocalVideoStream started successfully")
                    }
                }
            }
        } else {
            // Ask permissions
            self.requestRecordPermission()
            self.requestDevicePermission()
            print("RecordPermissiong or DevicePermission not granted. \n Please try again.")
        }
    }

    func endCall() {
        if let call = call {
            call.hangup(options: HangupOptions(), completionHandler: { (error) in
                if error != nil {
                    print("ERROR: It was not possible to hangup the call.")
                } else {

                }
                print("Call ended")
            })
        }
    }

    func acceptCall() {
        if let incomingCall = call {
            print("Accept call")

            let acceptCallOptions = AcceptCallOptions()
            if let localVideoStream = self.localVideoStream {
                do {
                    self.localRenderer = try Renderer(localVideoStream: localVideoStream)
                    self.localRendererView = VideoStreamView(view: (try self.localRenderer!.createView()))
                } catch {
                    print("Failed binding localVideoRenderView")
                }

                acceptCallOptions!.videoOptions = VideoOptions(localVideoStream: localVideoStream)

                incomingCall.startVideo(stream: localVideoStream) { error in
                    if (error != nil) {
                        print("LocalVideoStream failed to start")
                    }
                    else {
                        print("LocalVideoStream started successfully")
                    }
                }
            }

            incomingCall.accept(options: acceptCallOptions!) { (error) in
                if error != nil {
                    print("Failed to accpet incoming call.")
                } else {
                    print("Incoming call accepted with acceptCallOptions.")
                }
            }
        }
    }

[sankum-msft]

@hyounoo suggested fix should work in your case so I'm closing the issue. Please feel free to re-open if you encounter any issues.

[hyunoosung]

should I create a new ticket?

[sankum-msft]

We both ended up posting at the same time. I've reopened the ticket and looking at your posted code now.

[sankum-msft]

call.remoteParticipants on onCallStateChanged event doesn't include remoteStream sometimes.

Participant videos may take some time to be added under the RemoteParticipant.videoStreams collection. You should add a delegate to RemoteParticipant objects to receive the RemoteParticipant.onVideoStreamsUpdated event which gets fired when remote user stops/starts their video.

Please keep in mind that since there is a delay between SDK raising participant added event and you attaching delegate to that participant, you need to do both -

1. Add a delegate to handle video streams added/removed events that happen in the future, and
2. If there are any videos available under RemoteParticipant.videoStreams when you are attaching that delegate, you should handle them explicitly as you've missed event for those videos.

also local video doesn't render occasionally

This is unexpected. Can you please follow this guide and share log files for a repro so that we can investigate? Please also note down the time frame when it failed to render.

[hyunoosung]

thanks for the help Really appreciate I will follow the guide and post the log tmr Been working straight two days in a row Now im off to bed

[hyunoosung]

Finally, I've managed both localVideoStream and remoteVideoStream rendering working. onRemoteParticipants' StreamUpdated delegate event never occurs for me. So I had to give a 1 ~ 1.5 sec delay on main thread after call accepted or connected No idea how could solve this random remote stream event but with above trick.

Thanks for all the help so far!

[sankum-msft]

Are you querying for remote video streams when handling participant added event?

Please keep in mind that since there is a delay between SDK raising participant added event and you attaching delegate to that participant, you need to do both -

1. Add a delegate to handle video streams added/removed events that happen in the future, and
2. If there are any videos available under RemoteParticipant.videoStreams when you are attaching that delegate, you should handle them explicitly as you've missed event for those videos.

It has to be done in this order only - attach event handler and then query for available video streams.

[hyunoosung]

Are you querying for remote video streams when handling participant added event?

Please keep in mind that since there is a delay between SDK raising participant added event and you attaching delegate to that participant, you need to do both -

1. Add a delegate to handle video streams added/removed events that happen in the future, and
2. If there are any videos available under RemoteParticipant.videoStreams when you are attaching that delegate, you should handle them explicitly as you've missed event for those videos.

It has to be done in this order only - attach event handler and then query for available video streams.

Thanks, I followed your guide and code works with delegate now. Closing this issue now.