Support for PIP mode in iOS

pooja816 commented 6 years ago

Support for PIP mode in iOS

Please add support for PIP mode in iOS

Expected Behavior

When any video call is in process, in between I also want to access other features of the app e.g. - WhatsApp video call

Actual Behavior

When I try to access other features of the app during call, I need to disconnect the call

Video iOS SDK

1.3.4 via CocoaPods

Xcode

9.2

iOS Version

10.0

iOS Device

iPhone 7

Derad6709 commented 3 years ago

If Twitch and Netflix can support Picture in Picture mode, but other video calling apps can't, is it because they're not allowed by Apple?　Or is it just a lack of resources?

You can check some Apple documentation on it here!

ceaglest commented 3 years ago

Hi,

Seeing if I can answer some questions since I attended the AVKit lab during WWDC.

If Twitch and Netflix can support Picture in Picture mode, but other video calling apps can't, is it because they're not allowed by Apple?　Or is it just a lack of resources?

You can check some Apple documentation on it here!

@Derad6709 is correct, you should take a look at the docs for iOS 15.0-beta1. In previous versions of iOS use of the camera in a multi-tasking environment was allowed but the entitlement was not documented anywhere. Apps like Netflix could use AVPlayer in PiP to play streaming video but even in the best possible scenario the delay was ~1s or more making it infeasible for live conferencing.

There are two points I'm interested in. 1.Is it possible to implement Picture in Picture mode in calling apps before iOS15?

It looks like the AVPictureInPictureVideoCallViewController API for PiP is only available in iOS 15 but multi-tasking with the camera has been available as an undocumented API / entitlement on iPadOS 14.

2.Is it likely to be possible to implement Picture in Picture mode in the calling app after iOS15 this fall?

Yes, but the approach you use will depend on if your app is using PiP for playback only (which won't require a new entitlement), or while running the camera.

3.Is it possible to mute my own voice or someone else's voice while the call screen is in Picture in Picture mode? With Facetime, you can't switch the mute in Picture in Picture mode. You need to jump to Facetime once.

Since you provide the UI with AVPictureInPictureVideoCallViewController it should be possible to have a mute button. I don't think it will work with the AVSampleBufferDisplayLayer version of the API which is for playback only.

Best, Chris

21-Hidetaka-Ko commented 3 years ago

@Derad6709 @ceaglest Thank you for the info.

Yes, but the approach you use will depend on if your app is using PiP for playback only (which won't require a new entitlement), or while running the camera.

It looks like the AVPictureInPictureVideoCallViewController API for PiP is only available in iOS 15 but multi-tasking with the camera has been available as an undocumented API / entitlement on iPadOS 14.

Do you think it would be difficult to implement on iOS15 or earlier (e.g. iOS14)?

Apps like Netflix could use AVPlayer in PiP to play streaming video but even in the best possible scenario the delay was ~1s or more making it infeasible for I'm sure you're right.

Is it the video that is delayed?　Does the audio itself have a delay? So that was it. Do you think that implementing it in Picture in Picture mode is not suitable for the nature of the call service (no lag is important)?

Yes, but the approach you use will depend on if your app is using PiP for playback only (which won't require a new entitlement), or while running the camera.

I would like to know what the approach will be in our case. Our case is not a video playback service, but a calling service. Since it is a call, the screen is basically never stopped. It is always live. In this case, which applies and what would be the approach? Is it the latter?

Derad6709 commented 3 years ago

Multitasking camera documentation (including usage of the camera in background while pip?) is here.

Hi,

Seeing if I can answer some questions since I attended the AVKit lab during WWDC.

If Twitch and Netflix can support Picture in Picture mode, but other video calling apps can't, is it because they're not allowed by Apple?　Or is it just a lack of resources?

You can check some Apple documentation on it here!

@Derad6709 is correct, you should take a look at the docs for iOS 15.0-beta1. In previous versions of iOS use of the camera in a multi-tasking environment was allowed but the entitlement was not documented anywhere. Apps like Netflix could use AVPlayer in PiP to play streaming video but even in the best possible scenario the delay was ~1s or more making it infeasible for live conferencing.

There are two points I'm interested in. 1.Is it possible to implement Picture in Picture mode in calling apps before iOS15?

It looks like the AVPictureInPictureVideoCallViewController API for PiP is only available in iOS 15 but multi-tasking with the camera has been available as an undocumented API / entitlement on iPadOS 14.

2.Is it likely to be possible to implement Picture in Picture mode in the calling app after iOS15 this fall?

Yes, but the approach you use will depend on if your app is using PiP for playback only (which won't require a new entitlement), or while running the camera.

3.Is it possible to mute my own voice or someone else's voice while the call screen is in Picture in Picture mode? With Facetime, you can't switch the mute in Picture in Picture mode. You need to jump to Facetime once.

Since you provide the UI with AVPictureInPictureVideoCallViewController it should be possible to have a mute button. I don't think it will work with the AVSampleBufferDisplayLayer version of the API which is for playback only.

Best, Chris

21-Hidetaka-Ko commented 3 years ago

@Derad6709 @ceaglest Thanks for the info. I've read here that third party apps cannot be implemented, but when implementing a video calling service using an SDK for video calling services like Twillo, is it possible to implement it in PIP mode?　 Then.is it possible to implement PIP in a non-video call instead of a video call?

21-Hidetaka-Ko commented 3 years ago

@ceaglest Picture-in-picture can't be implemented in a call without video, right?

fukemy commented 2 years ago

any new on this? https://developer.apple.com/documentation/avkit/adopting_picture_in_picture_for_video_calls?changes=_1

pawansharmaAccolite commented 2 years ago

@pooja816 Are you able to do that ?

piyushtank commented 1 year ago

Hi all,

Thanks for your patience, and all the suggestions.

We have just shipped TwilioVideo 5.3 with Picture-in-picture support. Here is the changelogs. Try it out and let us know if you have any questions.

Best, Piyush

4brunu commented 1 year ago

Hi @piyushtank, First of all, thanks for making PiP available in Twilio Video, this is a functionality that we have been excited about for some time 🙂

PiP for video calls was introduced in iOS 15 by using the entitlement com.apple.developer.avfoundation.multitasking-camera-access. The problem with PiP in iOS 15 is it requires asking Apple's approval to use it. That's not ideal, specially if you manage multiple apps in different organisations, which is my case.

Since iOS 16, PiP is generally available via the property AVCaptureSession.isMultitaskingCameraAccessEnabled.

This means that, starting with iOS 16, we no longer need Apple's approval to use PiP, so I would like to use PiP only on iOS 16 and above.

Would it be possible for Twilio Video to expose a boolean to enable/disable PiP by setting AVCaptureSession.isMultitaskingCameraAccessEnabled in the AVCaptureSession that is used internally?

Thanks

fukemy commented 1 year ago

Hi @piyushtank,

First of all, thanks for making PiP available in Twilio Video, this is a functionality that we have been excited about for some time 🙂

PiP for video calls was introduced in iOS 15 by using the entitlement com.apple.developer.avfoundation.multitasking-camera-access.

The problem with PiP in iOS 15 is it requires asking Apple's approval to use it.

That's not ideal, specially if you manage multiple apps in different organisations, which is my case.

Since iOS 16, PiP is generally available via the property AVCaptureSession.isMultitaskingCameraAccessEnabled.

This means that, starting with iOS 16, we no longer need Apple's approval to use PiP, so I would like to use PiP only on iOS 16 and above.

Would it be possible for Twilio Video to expose a boolean to enable/disable PiP by setting AVCaptureSession.isMultitaskingCameraAccessEnabled in the AVCaptureSession that is used internally?

Thanks

Hi, I tried this with my iphone ios 16, but ismultitaskingcamerasupport return false, does it only support ipad?

piyushtank commented 1 year ago

@4brunu Thanks! TwilioVideo SDK doesn't do anything with PiP and multi-tasking enablement, its apps responsibility. TwilioVideo provides a renderer which could be used to render video frames using PiP view controller. Our VideoApp demonstrates how to use PiP renderer in an app - https://github.com/twilio/twilio-video-app-ios. S, as of now, we don't see a need for adding an API for AVCaptureSession.isMultitaskingCameraAccessEnabled.

@fukemy iPhone 16 supports picture in picture.

4brunu commented 1 year ago

Hi @piyushtank, thanks for the fast response.

I understand what you are saying, but there are two ways of enabling PiP:

One is using the entitlement com.apple.developer.avfoundation.multitasking-camera-access, sending a request to Apple asking for authorisation to use this entitlement, and with luck they will aprouve it and let you use it. If they don't, PiP won't work at all.
The other option, is to use the new API available on iOS 16, by setting AVCaptureSession.isMultitaskingCameraAccessEnabled = true. This way it doesn't require approval from Apple, it works right away. The problem is that TwilioVideo uses AVCaptureSession internally and it doesn't expose the AVCaptureSession instance to outside of the SDK (and I understand why it's not exposed), therefore we can't activate PiP with the new official way. This is why I was suggesting to allow somehow to pass a configuration to the TwilioVideo SDK, that would internally be passed to the internal instance of AVCaptureSession.

Honestly having to ask Apple permission to use PiP is a bit cumbersome.

fukemy commented 1 year ago

hi @piyushtank , please tell me if my code is wrong, tested with Iphone 11 promax IOS 16

if (@available(iOS 16.0, *)) {
            [self.capturer.captureSession beginConfiguration];
            if([self.capturer.captureSession isMultitaskingCameraAccessSupported]){
    //            [self.capturer.captureSession isMultitaskingCameraAccessEnabled];
                [self.capturer.captureSession setMultitaskingCameraAccessEnabled:YES];
                NSLog(@"setMultitaskingCameraAccessEnabled success");
            }else{
                NSLog(@"isMultitaskingCameraAccessEnabled not supported");
            }
            [self.capturer.captureSession commitConfiguration];
        } else {
            NSLog(@"isMultitaskingCameraAccessEnabled not supported because IOS < 16");
        }

here is my log: 2022-11-10 01:51:54.060894+0700 ABCApp[70459:7233238] isMultitaskingCameraAccessEnabled not supported

piyushtank commented 1 year ago

@4brunu Thanks for the suggestions. We can open a new API for AVCaptureSession.isMultitaskingCameraAccessEnabled, its just that we can't commit on the timeline as we all are working on our media engine (webrtc) upgrade. If we get a chance we will can implement this sooner, I will keep you posted. Also, I believe, like face time, you should still be able to render remote participant using PiP View controller using our new renderer. Let me know if you have any questions.

@fukemy Sounds like you have a custom capturer implementation. I am not sure why isMultitaskingCameraAccessSupported would return fail on iOS16. Try out our video app.

Chandlerdea commented 1 year ago

I ran the video app on my iPhone 13 mini, and it seems like multitasking camera functionality is not supported. When I background the application during a video call, the local video track is paused and doesn't resume until the app is brought back into the foreground (I've uploaded a screen recording ). There is no AVCaptureSession to modify in the project so it seems like we cannot use AVCaptureSession.isMultitaskingCameraAccessEnabled until the Twilio SDK gives us a way to modify that property.

piyushtank commented 1 year ago

@Chandlerdea Thanks for reaching out. We are planning to add support in short term, mostly this quarter. We will update the ticket.

piyushtank commented 1 year ago

Update:

I wanted to reach out to to confirm that the suggested enhancement is under development. While developing the enhancement, we figured that, even though the AVCaptureSession.isMultitaskingCameraAccessSupported is available on iOS 16+ and iPadOS 16+, it is supported only on the iPad that supports Stage Manager with an extended display. We have posted a question to apple developer forum to get more clarification on this. Let us know if you have any questions.

Thanks!

jayPbriskstar commented 1 year ago

@piyushtank

I have implement the video call in PIP but Once i go to the background video is not showing so How can i continue the video call in background for now?

arunmbriskstar commented 1 year ago

@piyushtank

We have implemented twilio as a major feature for our app, But due to this feature we are stuck at a very crucial stage. Is there any support or something that we can do to over come this issue.

Please let us know

twilio / video-quickstart-ios