Twilio signaling room error while using broadcast extension with ReplayKit (iOS).

[HRN8891]

Requirement: We are implementing Video Call feature with Screen sharing in iOS Application. We are using the ReplayKit broadcast extension for it

Description

We are using the ReplayKit broadcast extension for share the screen to the connected remote participant. When we start to share screen using the replay kit broadcast extension and user go to the background of the iOS application, it gets disconnected from the room which it has connected for Video Call.

Steps to Reproduce

Lets we have two user User1, User2 and they are connected with same room for video calling. Room Name: CallTest

1. Application-A (User1) and Application-B(User2) are connected to the room CallTest.

2. Both the Application started video calling for the room CallTest.

3. Application-A (User1) started to broadcast extension and connect with the room CallTest with new room identity UserScreen.

4. Now, Application-A has two room instance one for User1 identity and other is for UserScreen identity.

5. Application-A is going to background and after a few minutes User1 is disconnected from the room with error "Signaling connection timed out". This thing happens only when the iOS application stays in the background for 2 to 3 minutes.

We are getting the Error: "Signaling connection timed out" when we launch the application from background to foreground.

Error : Optional(Error Domain=com.twilio.video Code=53002 "Signaling connection timed out" UserInfo={NSLocalizedDescription=Signaling connection timed out})

Note: User1 is got connected in the background if we perform only video calling. As we start the broadcast extension it gets disconnected.

iOS Version: Higher Than 11.0 Xcode: 9.4 Video iOS SDK: 2.0.1 iPhone Device : iPhone-X, iPhone-6.

[piyushtank]

@HRN8891 When application is backgrounded, the video sdk depends on the audio usage if it should terminate the signaling connection. In order to debug the problem, can you provide following information -

1. Which Room instance is throwing 53002 error. Is it the one user1 connected to or the UserScreen ?
2. The audio setup in replay kit extensions. Are you feeding audio samples from microphone or the app audio or both using custom audio device?
3. The audio setup in the app. Are you using custom audio device?
4. Are you publishing the local audio track in both, replaykit extension and the app ?

Best, Piyush

[HRN8891]

Hi @piyushtank. Thanks for the update.

Here is the further details for Twilio ASK setup for App and Extension.

1. When application is in background for few minutes and when i go to foreground "User1" is disconnected from the room with the signalling error.

2. Here is our setup for repalykit in broadcast extension :

==================================================================== class SampleHandler: RPBroadcastSampleHandler, TVIRoomDelegate, TVIVideoCapturer {

public var isScreencast: Bool = true
public var room: TVIRoom?
weak var captureConsumer: TVIVideoCaptureConsumer?
static let kDesiredFrameRate = 30
static let kDownScaledFrameWidth = 540
static let kDownScaledFrameHeight = 960
var tempPixelBuffer: CVPixelBuffer?;
let audioDevice = ExampleCoreAudioDevice()

public var supportedFormats: [TVIVideoFormat] {
    get {
        let screenSize = UIScreen.main.bounds.size
        let format = TVIVideoFormat()
        format.pixelFormat = TVIPixelFormat.formatYUV420BiPlanarFullRange
        format.frameRate = UInt(SampleHandler.kDesiredFrameRate)
        format.dimensions = CMVideoDimensions(width: Int32(screenSize.width), height: Int32(screenSize.height))
        return [format]
    }
}

override func broadcastStarted(withSetupInfo setupInfo: [String: NSObject]?) {

            TwilioVideo.audioDevice = ExampleCoreAudioDevice(audioCapturer: self)

            let accessToken = "UserScreen Access Token"

            let localScreenTrack = TVILocalVideoTrack(capturer: self)
            let h264VideoCodec = TVIH264Codec()
            let localAudioTrack = TVILocalAudioTrack()
            let connectOptions = TVIConnectOptions.init(token: accessToken) { (builder) in

                builder.audioTracks = [localAudioTrack!]
                builder.videoTracks = [localScreenTrack!]

                builder.preferredVideoCodecs = [h264VideoCodec] as! [TVIVideoCodec]

                builder.roomName = "ScreenShare"
            }

            room = TwilioVideo.connect(with: connectOptions, delegate: self)
}

override func broadcastPaused() {
}

override func broadcastResumed() {
}

override func broadcastFinished() {
    room?.disconnect()
}

override func processSampleBuffer(_ sampleBuffer: CMSampleBuffer, with sampleBufferType: RPSampleBufferType) {
    RPScreenRecorder.shared().isMicrophoneEnabled = true
    switch sampleBufferType {
    case RPSampleBufferType.video:
        if ((captureConsumer != nil) && room?.state == .connected) {
            processVideoSampleBuffer(sampleBuffer)
        }
        break
    case RPSampleBufferType.audioApp:
        if (room?.state == .connected) {
            ExampleCoreAudioDeviceRecordCallback(sampleBuffer)
        }
        break
    case RPSampleBufferType.audioMic:
        if (room?.state == .connected) {
            ExampleCoreAudioDeviceRecordCallback(sampleBuffer)
        }
        break
    }
}

func startCapture(_ format: TVIVideoFormat, consumer: TVIVideoCaptureConsumer) {
    captureConsumer = consumer
    captureConsumer!.captureDidStart(true)

    CVPixelBufferCreate(kCFAllocatorDefault,
                        480,//self.width,
        640, //self.height,
        kCVPixelFormatType_420YpCbCr8BiPlanarFullRange,
        nil,
        &tempPixelBuffer)
}

func stopCapture() {
}

// MARK:- Private
func processVideoSampleBuffer(_ sampleBuffer: CMSampleBuffer) {
    //        let  imageBuffer = sampleBuffer.imageBuffer!

    let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)!
    var outPixelBuffer: CVPixelBuffer? = nil

    CVPixelBufferLockBaseAddress(pixelBuffer, []);

    let pixelFormat = CVPixelBufferGetPixelFormatType(pixelBuffer);

    if (pixelFormat != kCVPixelFormatType_420YpCbCr8BiPlanarFullRange) {
        assertionFailure("Extension assumes the incoming frames are of type NV12")
    }

    let status = CVPixelBufferCreate(kCFAllocatorDefault,
                                     SampleHandler.kDownScaledFrameWidth,
                                     SampleHandler.kDownScaledFrameHeight,
                                     pixelFormat,
                                     nil,
                                     &outPixelBuffer);
    if (status != kCVReturnSuccess) {
        print("Failed to create pixel buffer");
    }

    CVPixelBufferLockBaseAddress(outPixelBuffer!, []);

    // Prepare source pointers.
    var sourceImageY = vImage_Buffer(data: CVPixelBufferGetBaseAddressOfPlane(pixelBuffer, 0),
                                     height: vImagePixelCount(CVPixelBufferGetHeightOfPlane(pixelBuffer, 0)),
                                     width: vImagePixelCount(CVPixelBufferGetWidthOfPlane(pixelBuffer, 0)),
                                     rowBytes: CVPixelBufferGetBytesPerRowOfPlane(pixelBuffer, 0))

    var sourceImageUV = vImage_Buffer(data: CVPixelBufferGetBaseAddressOfPlane(pixelBuffer, 1),
                                      height: vImagePixelCount(CVPixelBufferGetHeightOfPlane(pixelBuffer, 1)),
                                      width: vImagePixelCount(CVPixelBufferGetWidthOfPlane(pixelBuffer, 1)),
                                      rowBytes: CVPixelBufferGetBytesPerRowOfPlane(pixelBuffer, 1))

    // Prepare out pointers.
    var outImageY = vImage_Buffer(data: CVPixelBufferGetBaseAddressOfPlane(outPixelBuffer!, 0),
                                  height: vImagePixelCount(CVPixelBufferGetHeightOfPlane(outPixelBuffer!, 0)),
                                  width: vImagePixelCount(CVPixelBufferGetWidthOfPlane(outPixelBuffer!, 0)),
                                  rowBytes: CVPixelBufferGetBytesPerRowOfPlane(outPixelBuffer!, 0))

    var outImageUV = vImage_Buffer(data: CVPixelBufferGetBaseAddressOfPlane(outPixelBuffer!, 1),
                                   height: vImagePixelCount(CVPixelBufferGetHeightOfPlane(outPixelBuffer!, 1)),
                                   width: vImagePixelCount( CVPixelBufferGetWidthOfPlane(outPixelBuffer!, 1)),
                                   rowBytes: CVPixelBufferGetBytesPerRowOfPlane(outPixelBuffer!, 1))

    var error = vImageScale_Planar8(&sourceImageY,
                                    &outImageY,
                                    nil,
                                    vImage_Flags(0));
    if (error != kvImageNoError) {
        print("Failed to down scale luma plane ")
        return;
    }

    error = vImageScale_CbCr8(&sourceImageUV,
                              &outImageUV,
                              nil,
                              vImage_Flags(0));
    if (error != kvImageNoError) {
        print("Failed to down scale chroma plane")
        return;
    }
    CVPixelBufferUnlockBaseAddress(outPixelBuffer!, []);
    CVPixelBufferUnlockBaseAddress(pixelBuffer, []);
    let time = CMSampleBufferGetPresentationTimeStamp(sampleBuffer);
    let frame = TVIVideoFrame(timestamp: time,
                              buffer: outPixelBuffer!,
                              orientation: TVIVideoOrientation.up)

    captureConsumer?.consumeCapturedFrame(frame!)
}

} ExampleCoreAudioDevice is custom audio class for replaykit broadcast extension. ====================================================================

3. We are using app microphone input as app local audio in Application. There is no custom audio setup for the application.

4. Yes. We are publishing local audio track on both replaykit and Application. Here is our setup for application. =====================================================================

func connect() {

    self.prepareLocalMedia()

    let connectOptions = TVIConnectOptions.init(token: accessToken) { (builder) in
        builder.audioTracks = self.localAudioTrack != nil ? [self.localAudioTrack!] : [TVILocalAudioTrack]()
        builder.videoTracks = self.localVideoTrack != nil ? [self.localVideoTrack!] : [TVILocalVideoTrack]()
        // Use the preferred audio codec
        if let preferredAudioCodec = Settings.shared.audioCodec {
            builder.preferredAudioCodecs = [preferredAudioCodec]
        }

        // Use the preferred video codec
        if let preferredVideoCodec = Settings.shared.videoCodec {
            builder.preferredVideoCodecs = [preferredVideoCodec]
        }

        // Use the preferred encoding parameters
        if let encodingParameters = Settings.shared.getEncodingParameters() {
            builder.encodingParameters = encodingParameters
        }

        builder.roomName = "ScreenShare"
    }

    room = TwilioVideo.connect(with: connectOptions, delegate: self)
}

func startPreview() {   
    camera = TVICameraCapturer(source: .frontCamera, delegate: self)
    localVideoTrack = TVILocalVideoTrack.init(capturer: camera!, enabled: true, constraints: nil, name: "Camera")
    if (localVideoTrack == nil) {
        logMessage(messageText: "Failed to create video track")
    } else {
        // Add renderer to video track for local preview
        localVideoTrack!.addRenderer(self.previewView)
        }
}

func prepareLocalMedia() {
    // We will share local audio and video when we connect to the Room.
    // Create an audio track.
    if (localAudioTrack == nil) {
        localAudioTrack = TVILocalAudioTrack.init(options: nil, enabled: true, name: "Microphone")
        if (localAudioTrack == nil) {
            logMessage(messageText: "Failed to create audio track")
        }
    }
    if (localVideoTrack == nil) {
        self.startPreview()
    }
}
func didConnect(to room: TVIRoom) {
    // At the moment, this example only supports rendering one Participant at a time.
}    
func room(_ room: TVIRoom, didDisconnectWithError error: Error?) {
    print("Disconncted from room \(room.name), error = \(String(describing: error))")
}    
func room(_ room: TVIRoom, didFailToConnectWithError error: Error) {
}    
func room(_ room: TVIRoom, participantDidConnect participant: TVIRemoteParticipant) {
}    
func room(_ room: TVIRoom, participantDidDisconnect participant: TVIRemoteParticipant){
}

==================================================================

Here is also some logs which we get when user1 disconnect from the Room.

2018-09-07 15:00:15.607415+0530 ScreenShareTwilio[525:23910] INFO:TwilioVideo:[Signaling]:RESIP::SIP: SipMessage::getContents: got content type (application/room-signaling+json) that is not known, returning as opaque application/octet-stream DEBUG:TwilioVideo:[Core]:onTerminated: code: 53002 msg: Signaling connection timed out explanation: 2018-09-07 15:00:15.657690+0530 ScreenShareTwilio[525:23910] DEBUG:TwilioVideo:[Core]:RoomSignalingImpl: State transition successful: kConnected -> kDisconnecting 2018-09-07 15:00:15.798638+0530 ScreenShareTwilio[525:24564] DEBUG:TwilioVideo:[Core]:Disconnecting 2018-09-07 15:00:15.798766+0530 ScreenShareTwilio[525:24564] DEBUG:TwilioVideo:[Core]:Canceling reconnect retry timer. 2018-09-07 15:00:15.798843+0530 ScreenShareTwilio[525:24564] DEBUG:TwilioVideo:[Core]:AppleReachability::~AppleReachability() 2018-09-07 15:00:15.799538+0530 ScreenShareTwilio[525:24564] INFO:TwilioVideo:[Core]:Close PeerConnectionSignaling's underlying PeerConnection 2018-09-07 15:00:15.856370+0530 ScreenShareTwilio[525:23660] DEBUG:TwilioVideo:[Core]:Discarding ice connection state update because our state is closed. 2018-09-07 15:00:15.901850+0530 ScreenShareTwilio[525:25285] INFO:TwilioVideo:[Platform]:Session interruption ended 2018-09-07 15:00:15.902031+0530 ScreenShareTwilio[525:25285] INFO:TwilioVideo:[Platform]:Session started running 2018-09-07 15:00:15.907563+0530 ScreenShareTwilio[525:23293] DEBUG:TwilioVideo:[Platform]:-[TVIAudioTrack dealloc] 2018-09-07 15:00:15.922973+0530 ScreenShareTwilio[525:25280] DEBUG:TwilioVideo:[Platform]:Video pipeline will start running 2018-09-07 15:00:16.013897+0530 ScreenShareTwilio[525:24508] DEBUG:TwilioVideo:[Platform]:Capture video output dropped a sample buffer. Reason = Discontinuity 2018-09-07 15:00:16.045554+0530 ScreenShareTwilio[525:23661] DEBUG:TwilioVideo:[Platform]:TVIRoom received audioSessionDeactivated. 2018-09-07 15:00:16.048473+0530 ScreenShareTwilio[525:23293] DEBUG:TwilioVideo:[Platform]:Did move to window with size: {0, 0}. Metal content scale factor is now: 0.000 2018-09-07 15:00:16.048709+0530 ScreenShareTwilio[525:23293] DEBUG:TwilioVideo:[Platform]:-[TVIVideoTrack dealloc] 2018-09-07 15:00:16.048803+0530 ScreenShareTwilio[525:23293] DEBUG:TwilioVideo:[Core]:RemoteParticipantSignaling::~RemoteParticipantSignaling(SID = PA044cc1947205289ce9b6b02023b86977) 2018-09-07 15:00:16.049914+0530 ScreenShareTwilio[525:23293] DEBUG:TwilioVideo:[Platform]:-[TVIMetalRenderer dealloc] 2018-09-07 15:00:16.085453+0530 ScreenShareTwilio[525:23910] DEBUG:TwilioVideo:[Core]: Receiving incoming SIP message from infra SIP/2.0 481 Call leg/Transaction does not exist

Via: SIP/2.0/TLS 127.0.0.1;received=182.72.43.194;branch=z9hG4bK-524287-1---2a8e875e3a308453;rport=4148

To: sip:orchestrator@mobile-endpoint.twilio.com;tag=97981637_6772d868_0b37fae1-d04f-469b-96e8-b08829930a07

From: "Ad5Ab107ff75477e9D00BA368BB3aC07" sip:Ad5Ab107ff75477e9D00BA368BB3aC07@mobile-endpoint.twilio.com;tag=c0c728ef

Call-ID: RB0p5AfP2IQ1E1qYmrpZ4g..

CSeq: 11 UPDATE

Server: Twilio

Content-Length: 0 2018-09-07 15:00:16.085583+0530 ScreenShareTwilio[525:23910] DEBUG:TwilioVideo:[Core]:Process UPDATE response with code 481 2018-09-07 15:00:16.085636+0530 ScreenShareTwilio[525:23910] DEBUG:TwilioVideo:[Core]:Received UPDATE status: 481. We will wait for session timer expiry to cleanup the call. 2018-09-07 15:00:16.139646+0530 ScreenShareTwilio[525:24564] INFO:TwilioVideo:[Core]:Done closing the PeerConnection 2018-09-07 15:00:16.145095+0530 ScreenShareTwilio[525:24564] INFO:TwilioVideo:[Core]:PeerConnectionSignaling destroyed 2018-09-07 15:00:16.145446+0530 ScreenShareTwilio[525:24564] DEBUG:TwilioVideo:[Core]:Canceling disconnect timer. 2018-09-07 15:00:16.145513+0530 ScreenShareTwilio[525:24564] DEBUG:TwilioVideo:[Core]:RoomSignalingImpl: State transition successful: kDisconnecting -> kDisconnected 2018-09-07 15:00:16.145566+0530 ScreenShareTwilio[525:24564] INFO:TwilioVideo:[Core]:Shutdown and join the signaling stack's thread. 2018-09-07 15:00:16.145706+0530 ScreenShareTwilio[525:23910] DEBUG:TwilioVideo:[Core]:Shutting down StackThread runloop. Disconnected from room ScreenShare, error = Optional(Error Domain=com.twilio.video Code=53002 "Signaling connection timed out" UserInfo={NSLocalizedDescription=Signaling connection timed out}) 2018-09-07 15:00:25.192674+0530 ScreenShareTwilio[525:23293] INFO:TwilioVideo:[Core]:Invalidating remote media of UserScreen 2018-09-07 15:18:50.694435+0530 ScreenShareTwilio[612:31071] DEBUG:TwilioVideo:[Platform]:Capture video output dropped a sample buffer. Reason = OutOfBuffers 2018-09-07 15:18:50.695188+0530 ScreenShareTwilio[612:31071] DEBUG:TwilioVideo:[Platform]:Capture video output dropped a sample buffer. Reason = OutOfBuffers

Question : Is it possible in twilio to stay connected in background with two room instance (One is for application and other one is for extension)?

[piyushtank]

@HRN8891 Thanks for t he detailed answer.

Since The user1 instance of Room uses TVIDefaultAudioDevice and UserScreen uses the ExampleAudioDevice, when you try to capture audio from both, one of the instance gets the microphone audio, and another gets the audio interruption event. In you case, since user1 connects to Room first, TVIDefaultAudioDevice gets the interruption when UserScreen connects to the Room with ExampleAudioDevice. Due to an underlying signaling bug in video SDK, when the app goes to the background, the Video SDK closes the signaling connection if the audio is deactivated/interrupted. So, if app does not come to foreground before session timer expires (typically ~88 seconds), user gets disconnected from the Room. We are planning to move to a new signaling stack where we should not have this limitation.

In the meantime, is it possible for you to try following workaround - Step 1 - In replay kit extension, do not use ExampleCoreAudioDevice, use the TVIDefaultAudioDevice instead. Set TwilioVideo.audioDevice.enabled = false before connecting to the Room. Step 2 - UserScreen connects to the Room without TVILocalAudioTrack.

Try it out and let me know if you have any questions.

[HRN8891]

Hiii @piyushtank We are implementing above things and let you know if we found anything. We have certain things to ask for the same.

Question 1
What if i will play music or youtube video while screen sharing (From other application or browser)? It will create same behaviour? Because, I have observed same for this.

Question 2 In Replaykit brodcast extension some time its stops with error : "Live Broadcast to App has stopped due to: (null)". At this time we want to disconnect the UserScreen from the room mean while but no delegate of Replay kit triggered. Can you please guide on this?.

[piyushtank]

@HRN8891 sorry for the delay in response.

What if i will play music or youtube video while screen sharing (From other application or browser)? It will create same behaviour? Because, I have observed same for this.

Thats correct you should see the same behavior but the workaround described above should resolve the problem.

In Replaykit brodcast extension some time its stops with error : "Live Broadcast to App has stopped due to: (null)". At this time we want to disconnect the UserScreen from the room mean while but no delegate of Replay kit triggered.

Have you tried callbacks from RPBroadcastActivityViewControllerDelegate and RPBroadcastControllerDelegate? I haven't addressed this scenario in our PR yet but we will try to handle this or add logs to start with in our replay kit sample app. You might have to use REST APIs to cleanup the screen instance of the Room in your app. see this for more information.