search

twilio / video-quickstart-ios

Twilio Video Quickstart for iOS
https://www.twilio.com/docs/api/video
MIT License
465 stars 178 forks source link

Camera Source Stops Working After Phone Call #347

Closed malavagile closed 5 years ago

malavagile commented 5 years ago

Camera Source Stop Working After Receiving a phone call.

I am using Room based connection between 2 users. What happens is when any participant receives a call and if they decline it within 2 seconds then camera working fine and resume correctly but if user takes more than 5 seconds to decline the call then camera preview stops rendering and when I tried to the attached camera to the source then the app crashed below error.

Steps to Reproduce

  1. User B joins a room created by User A
  2. UserA is sharing the video and User B viewing it.
  3. UserA receives a phone call.
  4. UserA declines the call after 5 seconds.
// Code that helps reproduce the issue

// When User A Received A Call

// Disable local tracks
localAudioTrack?.isEnabled = false
localVideoTrack?.isEnabled = false

// Initialize at the time of joining the room
self.cameraSource?.stopCapture()

// When User A Decline A Call
// Enable local tracks
localAudioTrack?.isEnabled = true
localVideoTrack?.isEnabled = true

// Restart Capturing.

if let currentDevice = source.device{
            source.startCapture(with: currentDevice) { (device, format, error) in
            }

        }else if let backCamera = TVICameraSource.captureDevice(for: .back){
            source.startCapture(with: backCamera) { (device, format, error) in
            }
        }

Expected Behavior

When call decline I am resuming the call and camera should start recording and it can be viewed by Other Participants.

Actual Behavior

Camera stops rendering video and preview view stuck at both ends.

Reproduces How Often

This happens every time the user takes more than 2 seconds to decline the call.

Logs

2018-12-28 15:34:37.291736+0530 [4233:1219353] ERROR:TwilioVideo:[Platform]:Could not add device input for deviceId: com.apple.avfoundation.avcapturedevice.built-in_video:1

2018-12-28 15:34:37.292181+0530 [4233:1219353] ERROR:TwilioVideo:[Platform]:Could not update camera device input, error Error Domain=com.twilio.video.capturer.camera Code=1003 "Could not add device input for source." UserInfo={NSLocalizedFailureReason=Could not add device input for source.}

2018-12-28 15:34:37.940837+0530 [4233:1218708] *** Assertion failure in -[TVICameraSource capturePipeline:videoOutputWillStartForDevice:], /Users/distiller/project/TwilioVideo/TwilioVideo/Sources/Objective-C/Implementation/Capture/TVICameraSource.m:507

Versions

All relevant version information for the issue.

Video iOS SDK

2.6 via CocoaPods

Xcode

9.4.1

iOS Version

12.1.2

iOS Device

iPhone 6S Plus

piyushtank commented 5 years ago

@malavagile Thanks for reaching out.

  1. UserA receives a phone call.

Is this step, are you receiving an incoming voip call with in the app? or its a cellular/PSTN call or an voip call from any other app via CallKit? If it is a cellular/PSTN call or an incoming call from any other VoIP app via CallKit, iOS will take care of stopping and starting the camera for you. You should not stop and start explicitly. The Video SDK will invoke interruptStarted and interruptEnded for the camera interruption events.

Let me know if you have any questions.

malavagile commented 5 years ago

It’s cellular call.

I did get the interruption events but it doesn’t resumed video automatically.

Sent from my iPhone

On 29-Dec-2018, at 1:04 AM, Piyush Tank notifications@github.com wrote:

@malavagile Thanks for reaching out.

UserA receives a phone call. Is this step, are you receiving an incoming voip call with in the app? or its a cellular/PSTN call or an voip call from any other app via CallKit? If it is a cellular/PSTN call or an incoming call from any other VoIP app via CallKit, iOS will take care of stopping and starting the camera for you. You should not stop and start explicitly. The Video SDK will invoke interruptStarted and interruptEnded for the camera interruption events.

Let me know if you have any questions.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

piyushtank commented 5 years ago

@malavagile I just tried to reproduce the problem on QuickStart, but could not reproduce it, I tried on iPhone X running iOS 12.1.1. Is it possible for you to try to reproduce the problem on QuickStart and let me know ?

malavagile commented 5 years ago

Well, I am not using VOIP push that's why I haven't use CXProvider class which you have used in QuickStart. I have used CXCallObserver class which notify me when a user received the call and when they declined it.

I saw in quick start demo in which you have disconnect room when the user receives a call so Do I have to disconnect room when I receive incoming call notification.

When I say incoming call I mean it's the cellular call.

Can you let me know what I should do when the user received a cellular call and when they disconnect the call and I want to continue twillio room?

Lastly I did remove all the code I did to stop the capturing but I got below error.

2019-01-02 14:23:49.418092+0530 [8298:2592664] [BoringSSL] nw_protocol_boringssl_get_output_frames(1301) [C15.1:2][0x11fc4b300] get output frames failed, state 8196
2019-01-02 14:23:49.418991+0530 [8298:2592664] [BoringSSL] nw_protocol_boringssl_get_output_frames(1301) [C15.1:2][0x11fc4b300] get output frames failed, state 8196
malavagile commented 5 years ago

Here is the complete logs of Twillio at my end. It may help you to identify issue.

Incoming Call Started
INFO:TwilioVideo:[Platform]:Session interrupted with reason: 1
INFO:TwilioVideo:[Platform]:... Capture pipeline did stop.
DEBUG:TwilioVideo:[Platform]:Teardown video pipeline ...
DEBUG:TwilioVideo:[Platform]:Finished video pipeline teardown.
DEBUG:TwilioVideo:[Platform]:Video pipeline did finish running
DEBUG:TwilioVideo:[Platform]:No background task to stop.
INFO:TwilioVideo:[Platform]:Session interrupted with reason: 1
INFO:TwilioVideo:[Platform]:... Capture pipeline did stop.
DEBUG:TwilioVideo:[Platform]:Teardown video pipeline ...
DEBUG:TwilioVideo:[Platform]:Finished video pipeline teardown.
DEBUG:TwilioVideo:[Platform]:Video pipeline did finish running
INFO:TwilioVideo:[Platform]:Session stopped running
[avas] AVAudioSessionPortImpl.mm:56:ValidateRequiredFields: Unknown selected data source for Port Speaker (type: Speaker)
[avas] AVAudioSessionPortImpl.mm:56:ValidateRequiredFields: Unknown selected data source for Port Speaker (type: Speaker)
DEBUG:TwilioVideo:[Platform]:Default audio device received AVAudioSessionInterruptionTypeBegan.
[avas] AVAudioSessionPortImpl.mm:56:ValidateRequiredFields: Unknown selected data source for Port Speaker (type: Speaker)
[aurioc] 1029: failed: '!pri' (enable 3, outf< 1 ch,  44100 Hz, Int16> inf< 1 ch,  44100 Hz, Int16>)
INFO:TwilioVideo:[Platform]:Session interrupted with reason: 1
INFO:TwilioVideo:[Platform]:... Capture pipeline did stop.
INFO:TwilioVideo:[Platform]:Session interrupted with reason: 1
DEBUG:TwilioVideo:[Platform]:Teardown video pipeline ...
INFO:TwilioVideo:[Platform]:... Capture pipeline did stop.
DEBUG:TwilioVideo:[Platform]:Finished video pipeline teardown.
DEBUG:TwilioVideo:[Platform]:Teardown video pipeline ...
DEBUG:TwilioVideo:[Platform]:Video pipeline did finish running
DEBUG:TwilioVideo:[Platform]:Finished video pipeline teardown.
DEBUG:TwilioVideo:[Platform]:No background task to stop.
DEBUG:TwilioVideo:[Platform]:Video pipeline did finish running
DEBUG:TwilioVideo:[Platform]:No background task to stop.
[aurioc] 1029: failed: '!pri' (enable 3, outf< 1 ch,  44100 Hz, Int16> inf< 1 ch,  44100 Hz, Int16>)
[aurioc] 1029: failed: '!pri' (enable 3, outf< 1 ch,  44100 Hz, Int16> inf< 1 ch,  44100 Hz, Int16>)
[aurioc] 1029: failed: '!pri' (enable 3, outf< 1 ch,  44100 Hz, Int16> inf< 1 ch,  44100 Hz, Int16>)
[aurioc] 1029: failed: '!pri' (enable 3, outf< 1 ch,  44100 Hz, Int16> inf< 1 ch,  44100 Hz, Int16>)
Call Disconnected
INFO:TwilioVideo:[Platform]:Session interruption ended
INFO:TwilioVideo:[Platform]:Session started running
INFO:TwilioVideo:[Platform]:Session interrupted with reason: 3
INFO:TwilioVideo:[Platform]:... Capture pipeline did stop.
DEBUG:TwilioVideo:[Platform]:Teardown video pipeline ...
DEBUG:TwilioVideo:[Platform]:Finished video pipeline teardown.
DEBUG:TwilioVideo:[Platform]:Video pipeline did finish running
DEBUG:TwilioVideo:[Platform]:No background task to stop.
DEBUG:TwilioVideo:[Platform]:Interruption is not recoverable, will not attempt to resume.
INFO:TwilioVideo:[Platform]:Session stopped running
INFO:TwilioVideo:[Platform]:Session interruption ended
INFO:TwilioVideo:[Platform]:Session started running
2019-01-02 18:27:18.361835+0530 DOSL[8879:2682473] running.
DEBUG:TwilioVideo:[Platform]:Video device will start running: <AVCaptureFigVideoDevice: 0x14d66cdb0 [Front Camera][com.apple.avfoundation.avcapturedevice.built-in_video:1]>.
[avas] AVAudioSessionPortImpl.mm:56:ValidateRequiredFields: Unknown selected data source for Port Speaker (type: Speaker)
[avas] AVAudioSessionPortImpl.mm:56:ValidateRequiredFields: Unknown selected data source for Port Speaker (type: Speaker)
DEBUG:TwilioVideo:[Platform]:Default audio device received AVAudioSessionInterruptionTypeEnded.
[avas] AVAudioSessionPortImpl.mm:56:ValidateRequiredFields: Unknown selected data source for Port Speaker (type: Speaker)
[avas] AVAudioSessionPortImpl.mm:56:ValidateRequiredFields: Unknown selected data source for Port Speaker (type: Speaker)
[avas] AVAudioSessionPortImpl.mm:56:ValidateRequiredFields: Unknown selected data source for Port Speaker (type: Speaker)
[avas] AVAudioSessionPortImpl.mm:56:ValidateRequiredFields: Unknown selected data source for Port Speaker (type: Speaker)
[avas] AVAudioSessionPortImpl.mm:56:ValidateRequiredFields: Unknown selected data source for Port Speaker (type: Speaker)
[avas] AVAudioSessionPortImpl.mm:56:ValidateRequiredFields: Unknown selected data source for Port Speaker (type: Speaker)
DEBUG:TwilioVideo:[Core]:Sending a ping message ...
INFO:TwilioVideo:[Signaling]:RESIP::TRANSACTION: Stats message
WARN:TwilioVideo:[Signaling]:RESIP::STATS: RESIP:TRANSACTION
TU summary: 0 TRANSPORT 0 TRANSACTION 0 CLIENTTX 0 SERVERTX 0 TIMERS 0
Transaction summary: reqi 5 reqo 10 rspi 10 rspo 5
Details: INVi 0/S0/F0 INVo 1/S1/F0 ACKi 0 ACKo 1 BYEi 0/S0/F0 BYEo 0/S0/F0 CANi 0/S0/F0 CANo 0/S0/F0 MSGi 0/S0/F0 MSGo 0/S0/F0 OPTi 0/S0/F0 OPTo 0/S0/F0 REGi 0/S0/F0 REGo 0/S0/F0 PUBi 0/S0/F0 PUBo 0/S0/F0 SUBi 0/S0/F0 SUBo 0/S0/F0 NOTi 0/S0/F0 NOTo 0/S0/F0 REFi 0/S0/F0 REFo 0/S0/F0 INFi 5/S5/F0 INFo 8/S8/F0 PRAi 0/S0/F0 PRAo 0/S0/F0 SERi 0/S0/F0 SERo 0/S0/F0 UPDi 0/S0/F0 UPDo 0/S0/F0
Retransmissions: INVx 0 finx 0 nonx 0 BYEx 0 CANx 0 MSGx 0 OPTx 0 REGx 0 PUBx 0 SUBx 0 NOTx 0 REFx 0 INFx 0 PRAx 0 SERx 0 UPDx 0
DEBUG:TwilioVideo:[Core]:Sending a ping message ...
DEBUG:TwilioVideo:[Core]:
Sending outgoing SIP message
UPDATE sip:172.18.17.242:5060 SIP/2.0
piyushtank commented 5 years ago

@malavagile Thanks for the logs and more information.

You should not stop or start explicitly when you receive a new incoming call. The Video SDK takes care of that internally ? Are you not getting an expected behavior when you do not stop and start explicitly when incoming call arrives? The SSL errors you are receiving on console is not related and you can ignore it I think.

Also, you should not disconnect from Room when you receive a new Call. In our CallKit QuickStart Example, it does not call disconnect when a new incoming call arrives. The held method on CXProviderDelegate gets called. We do not take any action when a call gets held, see [this] and [this]. I recommend not calling any camera APIs when interruption happens and let SDK handle the events.

If above does not help, is it possible for you to change the QuickStart code to reproduce the problem and share with us? It would be easy to debug the issue by working on the same code base.

Best, Piyush

malavagile commented 5 years ago

I did figure out the issue... When the cellular call ends and I call the API right after it then the video stopped otherwise it's working fine. Sometimes I got error in Camera Source Delegate in didFailWithError sometime.

Do you have an idea what can be done if we need to restart the capturing when we got this error from the delegate?

Should I keep my API call on a background thread as camera resume process may be run on main thread only which may cause the issue?

piyushtank commented 5 years ago

@malavagile

Do you have an idea what can be done if we need to restart the capturing when we got this error from the delegate?

what errors are you getting when didFailWithError is called ?

Should I keep my API call on a background thread as camera resume process may be run on main thread only which may cause the issue?

You can call SDK's camera APIs from main or background thread. SDK takes care of threading internally.

malavagile commented 5 years ago

Thanks for the reply.

Now I am not doing anything to start and stop the camera preview. Only I receive notification call disconnection then only I just call my Rest API and Reestablish Socket Connection ( I am using the socket for chat purpose only ).

Do you think the socket connection may cause this issue?

This is the error from delegate : 

Camera device failed with error : Error Domain=AVFoundationErrorDomain Code=-11800 "The operation could not be completed" UserInfo={NSLocalizedFailureReason=An unknown error occurred (-12780), NSLocalizedDescription=The operation could not be completed, NSUnderlyingError=0x282ecd320 {Error Domain=NSOSStatusErrorDomain Code=-12780 "(null)"}}
malavagile commented 5 years ago

I did figure out the exact scenario when the camera device does not restart automatically.

As per my requirement, I am presenting a screen where both users can share the live video when I present it the first time the code work every time I go to background.

But when dismiss the controller and represent it again for live video share for both user, After that when I go in the background or call occurred the video issue starts.

I did check the memory graph where I found that even my class deinit successfully these things remain in the memory.

Have a look at this image below. screenshot 2019-01-09 at 4 46 12 pm

malavagile commented 5 years ago

Finally, I solve the issue.

The issue was when my class deinit it doesn't remove reference with localVideoTrack and localAudioTrack so when I dismiss my controller, Twillio Classes remains in the memory.

I wrote below method which removes the reference and it solved my issue. When user click disconnect then I call this method.

func unprepareLocalMedia() {

        self.localAudioTrack = nil
        self.localVideoTrack = nil
        self.localDataTrack = nil

        self.room = nil

        // For Twillio 2.6
        self.cameraSource?.stopCapture()
        self.cameraSource?.previewView?.removeFromSuperview()
        self.cameraSource = nil
    }

I suggest that please add this method in github code as well so we can safely close the room when it disconnects.

ceaglest commented 5 years ago

Hi @malavagile,

I'm glad to hear that you were able to move forward.

I wrote below method which removes the reference and it solved my issue. When user click disconnect then I call this method.

Thanks for making this suggestion! Our examples tend to begin using the camera for preview, and then continue to use it even after disconnecting from a Room. As you mentioned, it would be much better if the ViewControllers in our sample code could always be dropped into an existing navigation hierarchy with no issues. We have an internal ticket ISDK-1656 to track this, and will be making updates to our sample code to address lifecycle issues.

Best, Chris