Remote video not being sent from iPhone 5 device

[astanton]

Here's my current scenario in a 2-way video chat.

Device 1 iPhone 6 iOS 8.2 local video capture - working remote video capture - displaying white screen for iPhone 5's remote video feed

Device 2 iPhone 5 iOS 9.3 local video capture - working remote video capture - working (can see the iPhone 6 capture feed)

I am also using a custom QBRTCCameraCapture for my local video feed, that you guys helped with and gave me. I've made slight modifications to it to fit the needs I want. I have attached the class that I'm using for it so that you all can check it out.

FaceDetectorCameraCapture.zip

Another thing to note, which I think may be the main cause of this, is that the `AVCaptureVideoDataOutputSampleBufferDelegate captureOutput' stops being called which appears to be happening just as the users are connected to each other. I'm assuming this is part of the problem as well, but I'm not sure where in the line it is.

I have tested this code out on my wife's iPhone 6, which happens to be running iOS 8.0, and everything is working fine. So this is either an iPhone 5 issue or it is an iOS 9 issue. I'm guessing it is a device issue but I'm not sure.

I did use the class that you guys gave me on the sample application, and it worked just fine on both iPhone 5 and iPhone 6, which again confuses me. And I am using this via bridged code since my app is in Swift and this is in Objective C.

[astanton]

A little bit more information...

So I did a drop replacement of the FaceDetectorCamera classes there into the sample video chat app and use that instead of the default QBRTCCameraCapture for the localCameraCapture and everything seems to be working fine.

The iPhone 5 is capturing the video fine and streaming it.

Another issue I've also noticed (on my swift app) is that after the first connection and I close it, when I try to connect again afterwards, it never works. It doesn't actually connect. I have to restart the app to get it to connect and work again. But if I change the `localCameraCapture' back to the default 'QBRTCCameraCapture' class, it works just fine and I can connect immediately after (and video works fine on iPhone 5 as mentioned above).

However when I pulled this class into the sample app, again it worked fine and I could connect multiple times afterwards.

So this is making me wonder if there is something going on with bridging the code from swift possibly? I'm going to try and investigate further and compare logs between the 2 versions and see if I can figure anything out.

[ghost]

Hi @astanton please check - dealloc method in FaceDetectorCamera, if this method don't called, maybe you have retain circle in your app.

[ghost]

@astanton for swift:

deinit {
    // perform the deinitialization
}

[astanton]

@Pro100Andrey so I checked it out and that dealloc method is not being called.

My view controller only has an instance of the FaceDetectorCamera object in it, that I use as the camera for the local video feed.

I then wrote a wrapper function in FaceDetectorCamera so that I could call the dealloc from the swift code (because I am not sure how to call the dealloc method from swift) and put that inside the deinit method.

However, my deinit method is not even being called and I'm not sure why.

So after that, I tried calling the wrapper dealloc method from the viewDidDisappear function in my view controller, however the dealloc method is crashing for me when it's being called. I even tried commenting out everything from the dealloc method other than the super call, and I'm still getting crashes with EXC_BAD_ACCESS messages.

Also, I'm assuming this advice was for the issue where I can't have back-to-back sessions correct?

[ghost]

@astanton Please provide code sample for investigate this issue. Thanks!

[astanton]

Okay so here is some code...

One thing to note as well. This is my view controller class that is a skeleton of what else is in it, I removed a lot of other functionality that is kind of irrelevant to the problems I'm having. However, once I removed a lot of other code, the 'deinit method is now being hit. So I'm guessing that something in my other code is holding a reference to the view controller which isn't letting the deinit method to be called.

However, after the deinit method is being called in swift, it's then calling the dealloc code in the FaceDetectorCamera class, and in there, it's now crashing with the EXC_BAD_ACCESS on the 2nd line in the dealloc method. I tried commenting that out, and it's then crashing on another line in there.

With that said, here's my ViewController class that is stripped down to the bare bones that shows the problem. I've added an end chat button to the middle of the view so you can just end it when pressing the button whenever.

import UIKit

class VideoChatViewController: UIViewController, QBRTCClientDelegate, QBChatDelegate {

    var didInitiateChat : Bool!
    var currentSession : QBRTCSession!

    var yourQuickBloxId : UInt!
    var localVideoView : UIView!
    var localVideoCameraCapture : FaceDetectorCamera!

    var remoteQuickBloxId : UInt!
    var remoteVideoView : QBRTCRemoteVideoView!

    override func viewDidLoad() {
        super.viewDidLoad()

        QBRTCClient.instance().addDelegate(self)
        QBChat.instance().addDelegate(self)
        QBRTCConfig.setDialingTimeInterval(3)
        QBRTCConfig.setAnswerTimeInterval(10)

        initializeLocalVideoView();
        initializeRemoteVideoView();

        let endChatButton = UIButton(frame: CGRect(x: UIScreen.mainScreen().bounds.width/2, y: UIScreen.mainScreen().bounds.height/2, width: 100, height: 20))
        endChatButton.setTitle("End Chat", forState: .Normal)
        endChatButton.setTitleColor(UIColor.blueColor(), forState: .Normal)
        endChatButton.addTarget(self, action: #selector(VideoChatViewController.endChatButtonPressed(_:)), forControlEvents: .TouchUpInside)
        endChatButton.layer.zPosition = 3
        self.view.addSubview(endChatButton)
    }

    deinit {
        print("---- called deinit")
    }

    override func viewWillAppear(animated: Bool) {
        // if this is the person who initiated the chat, call the other person
        if (didInitiateChat == true) {
            currentSession = QBRTCClient.instance().createNewSessionWithOpponents([remoteQuickBloxId], withConferenceType: QBRTCConferenceType.Video)
            currentSession.startCall(nil)
        }
    }

    override func viewDidAppear(animated: Bool) {
        QBRTCClient.initializeRTC()
        QBRTCSoundRouter.instance().initialize()
        QBRTCSoundRouter.instance().currentSoundRoute = .Receiver
    }

    override func viewDidDisappear(animated: Bool) {
        QBRTCClient.deinitializeRTC()
        QBRTCSoundRouter.instance().deinitialize()
    }

    /**
        This function initializes the video feed of your own front camera
    **/
    private func initializeLocalVideoView() {
        let videoFormat = QBRTCVideoFormat()
        videoFormat.frameRate = 30;
        videoFormat.pixelFormat = QBRTCPixelFormat.Format420f
        videoFormat.width = 640;
        videoFormat.height = 480;

        // add the white uiview behind the publisher view
        let whiteBackgroundView = UIView(frame: CGRect(x: 7, y: 20, width: 106, height: 140))
        whiteBackgroundView.layer.zPosition = 3;
        whiteBackgroundView.backgroundColor = UIColor.whiteColor()
        view.addSubview(whiteBackgroundView)

        // setup the face detector camera capture
        let faceDetectorCamera = FaceDetectorCamera(videoFormat: videoFormat, position: .Front, yourQuickBloxId: yourQuickBloxId, remoteQuickBloxId: remoteQuickBloxId)

        // add the local video view over top
        localVideoView = UIView(frame: CGRect(x: 10, y: 23, width: 100, height: 134))
        localVideoView.contentMode = UIViewContentMode.ScaleAspectFill
        localVideoCameraCapture = faceDetectorCamera
        localVideoCameraCapture.previewLayer.frame = localVideoView.bounds
        localVideoCameraCapture.startSession()

        localVideoView.layer.insertSublayer(localVideoCameraCapture.previewLayer, atIndex: 0)
        localVideoView.layer.zPosition = 3

        self.view.addSubview(localVideoView)
    }

    private func initializeRemoteVideoView() {
        remoteVideoView = QBRTCRemoteVideoView(frame: CGRect(x: 0, y: 0, width: UIScreen.mainScreen().bounds.width, height: UIScreen.mainScreen().bounds.height))
        remoteVideoView.contentMode = UIViewContentMode.ScaleAspectFill
        remoteVideoView.layer.zPosition = 2
        remoteVideoView.hidden = true

        // this makes the video layer full screen
        let layer = remoteVideoView.subviews[0].layer as! AVSampleBufferDisplayLayer
        layer.videoGravity = AVLayerVideoGravityResizeAspectFill

        self.view.addSubview(remoteVideoView)
    }

    // QuickBlox Client Delegate Methods below
    func didReceiveNewSession(session: QBRTCSession!, userInfo: [NSObject : AnyObject]!) {
        currentSession = session
        currentSession.acceptCall(nil)
    }

    func session(session: QBRTCSession!, acceptedByUser userID: NSNumber!, userInfo: [NSObject : AnyObject]!) {
        print("user has accepted the call")
    }

    func session(session: QBRTCSession!, rejectedByUser userID: NSNumber!, userInfo: [NSObject : AnyObject]!) {
        print("user has rejected your call")
    }

    func session(session: QBRTCSession!, hungUpByUser userID: NSNumber!, userInfo: [NSObject : AnyObject]!) {
        endChat()
    }

    func session(session: QBRTCSession!, disconnectedFromUser userID: NSNumber!) {
        endChat()
    }

    func session(session: QBRTCSession!, userDidNotRespond userID: NSNumber!) {
        endChat()
    }

    func session(session: QBRTCSession!, connectionFailedForUser userID: NSNumber!) {
        endChat()
    }

    func session(session: QBRTCSession!, initializedLocalMediaStream mediaStream: QBRTCMediaStream!) {
        mediaStream.videoTrack.videoCapture = self.localVideoCameraCapture
        mediaStream.videoTrack.enabled = true
    }

    func session(session: QBRTCSession!, receivedRemoteVideoTrack videoTrack: QBRTCVideoTrack!, fromUser userID: NSNumber!) {
        remoteVideoView.setVideoTrack(videoTrack)
        remoteVideoView.hidden = false
    }

    func endChatButtonPressed(sender: UIButton) {
        endChat()
    }

    private func endChat(forReporting: Bool = false) {
        currentSession.hangUp(nil)
        self.dismissViewControllerAnimated(true, completion: nil)
    }
}

Now for the initiator, this is where it is being called from:

func handleSegueToVideoChat(sender: UIButton) {

        let message = QBChatMessage()
        message.text = Constants.QuickBloxMessageText.kSpinFloorVideoChat
        message.senderID = 12213716       // hardcoded to Drew's QuickBlox ID right now
        message.recipientID = 12213838    // hardcoded to Daysi's QuickBlox ID right now
        message.customParameters = ["initiatorUserId" : UserService.sharedInstance.getCurrentUserId()]

        QBChat.instance().sendSystemMessage(message) { (error) in
            if error != nil {
                print(error)
            }
            else

                let storyboard = UIStoryboard(name: "Main", bundle: nil)
                let videoChatSwiftViewController = storyboard.instantiateViewControllerWithIdentifier("VideoChatViewController") as! VideoChatViewController

                videoChatSwiftViewController.yourQuickBloxId = message.senderID
                videoChatSwiftViewController.remoteQuickBloxId = message.recipientID
                videoChatSwiftViewController.didInitiateChat = true

                self.navigationController!.presentViewController(videoChatSwiftViewController, animated: true, completion: nil)
            }
        }
    }

And that system message that is sent is then received in the AppDelegate by the receiver, where they execute this code once they receive it, which will open the video chat on the receivers end:

private func handleVideoChatSystemMessage(userInfo: [NSObject: AnyObject]) {
        if let sessionInfo = userInfo["sessionInfo"] as? NSDictionary {

            let initiatorUserId = sessionInfo["initiatorUserId"] as? String
            let remoteQuickBloxId = sessionInfo["remoteQuickBloxId"] as! UInt
            let yourQuickBloxId = sessionInfo["yourQuickBloxId"] as! UInt

            let currentUser:PFUser = PFUser.currentUser()!;
            let currentUserObjId = currentUser.objectId!;

            if (currentUserObjId != initiatorUserId) {

                let storyboard = UIStoryboard(name: "Main", bundle: nil)
                let videoChatSwiftViewController = storyboard.instantiateViewControllerWithIdentifier("VideoChatViewController") as! VideoChatViewController

                videoChatSwiftViewController.didInitiateChat = false

                videoChatSwiftViewController.remoteQuickBloxId = remoteQuickBloxId
                videoChatSwiftViewController.yourQuickBloxId = yourQuickBloxId

                let navController = self.window!.rootViewController as! UINavigationController
                navController.presentViewController(videoChatSwiftViewController, animated: true, completion: nil)
            }
        }
    }

I also did a little bit of investigation into the crash that happens in the dealloc by enabling zombies and was able to come up with this bit of information as to the crash, although I'm not sure what it means or how to solve it

---- called deinit
2016-05-01 15:29:44.254 SpinTheBottle[2971:637444] *** -[FaceDetectorCamera videoDataOutput]: message sent to deallocated instance 0x1702ffe00
(lldb) command script import lldb.macosx.heap
(lldb) malloc_info --stack-history 0x1702ffe00
0x00000001702ffe00: malloc(   128) -> 0x1702ffe00 _NSZombie_FaceDetectorCamera

[ghost]

@astanton Please remove QBRTCClient.deinitializeRTC() from your code + [super dealloc] from FaceDetectorCamera.m and check again

[astanton]

@Pro100Andrey I just commented out the line of code and it's still crashing in the same spot in the dealloc method, on both devices.

[astanton]

I also just tried moving the initialization of the QBRTCClient into the AppDelegate since that is what the sample does, however from the docs it sounded like you wanted to initialize it only when you were using it, and deinitialize it right after you are done, which is why I had it in those spots.

If it's better to just keep it in the AppDelegate and never call deinitialize then I'll just do that.

[ghost]

@astanton please provide easy swift sample for investigate this issue.

[ghost]

@astanton please check

private func endChat(forReporting: Bool = false) {
        currentSession.hangUp(nil)
}

end dismiss View Controller Animated here:

#pragma mark -
#pragma mark QBRTCClientDelegate

- (void)sessionDidClose:(QBRTCSession *)session {

    // release session instance
    self.session = nil;
}

[astanton]

@Pro100Andrey I am not certain what you are asking me to check in this instance.

Also there is no self.session in my view controller, although there is currentSession.

I tried adding in the calls you mention above, and changing it to currentSession, and it's still not working out for me.

[astanton]

I'm trying to create a small sample app where this is reproducable so you can just download and compile, but when I try to log into QuickBlox from the code I get an error. I'm calling.

QBRequest.logInWithUserLogin(kUserName, password: kPassword, successBlock: { (response, user) in
            print("logged in")
            }) { (response) in
                print("error")
        }

And this is the error I get from it

2016-05-01 17:22:51.007 testQuickblox[3218:667528] *** Terminating app due to uncaught exception 'NSInvalidArgumentException', reason: '*** setObjectForKey: object cannot be nil (key: acct)'

kUserName and kPassword are just strings. It never gets into the success or error callback. This is in my app delegate, and I've also put this in a view controller to happen when I press a button - same thing. I get those errors when trying to call signUp as well.

This is using the frameworks via a manual install.

EDIT:

So turns out this is only crashing on my iPhone 6 which is on iOS 8.2. It's not crashing on my iPhone 5 that has iOS 9.2 on it.

[astanton]

So I wrote a test-app that would work on iOS 9 and after doing so, I realized that our app does not have ARC turned on. I realized this when I tried to run my code on my test-app and it complained about [super dealloc] and ARC didn't like it. Then I remembered I never saw that in my app.

So in the test app, with ARC turned on (and that line commented out), the problem with it crashing in dealloc did not happen with my face detector class. I then turned off ARC and uncommented that line out, and as expected, it crashed in the same spot.

So I tried turning on ARC in my app (in the state that the class was in up above), and sure as hell, both video's were displaying just fine, the deinit call was being made, and I could connect immediately after in a new chat.

So the problem the whole time has been that I don't have ARC turned on. On top of that though, I do also have to figure out why deinit is not being called in my class, which I should be able to track down by what is referencing it in the class.

So thanks once again for pointing me in the direction that lead me down this path.

[ghost]

@astanton any updates?

[astanton]

@Pro100Andrey as mentioned in my last post, the issue after all of this investigation was just that it was not in ARC. I changed that and everything seems to be working just fine now.

Thanks for the help, I'll close this.