search

twilio / twilio-video-ios

Programmable Video SDK by Twilio
http://twilio.com/video
Other
65 stars 22 forks source link

Video is paused when app is in the background #38

Closed Moriquendi closed 2 years ago

Moriquendi commented 5 years ago

Description

On iOS, Video stream is paused when app is in the background.

Steps to Reproduce

  1. Use sample code from https://github.com/twilio/video-quickstart-swift
  2. Connect to room (somewhere where you can preview live the participants video)
  3. Background the app
  4. Video stream will be paused

Although I do see the app is working in the background (my own logs show up), video is paused on the moment I hit home button. I do have "Audio, AirPlay and Picture in Picture" capability enabled in project settings.

Expected Behavior

Participant on the other end see's my camera video stream even when the app is in the background.

Reproduces How Often

100%

Versions

Twilio 2.7.0

Xcode

Xcode 10.1

iOS Version

12.1.4

iOS Device

iPhone 7

ceaglest commented 5 years ago

Hi @Moriquendi,

iOS 12.0 doesn't allow you to use the camera in the background, even in a picture-in-picture view. Given this OS restriction, Twilio Video is behaving as expected. We use AVCaptureSession under the hood, and you can see more info about why interruptions occur with these APIs.

https://developer.apple.com/documentation/avfoundation/avcapturesession/interruptionreason

Best, Chris

Moriquendi commented 5 years ago

@ceaglest Thanks, didn’t know that. However, I also encounter the issue when I’m streaming screen frames from ReplayKit 2. Shouldn’t that be allowed?

Moriquendi commented 5 years ago

To explain futher: I pass ReplayKit frames from Upload extension to my app and then send them to the sink within the app.

ceaglest commented 5 years ago

Hi,

However, I also encounter the issue when I’m streaming screen frames from ReplayKit 2. Shouldn’t that be allowed?

If you have an extension encode the frames (see SampleHandler example) then it should work independently of your app.

To explain futher: I pass ReplayKit frames from Upload extension to my app and then send them to the sink within the app.

If you're sharing frames from your extension to your app, I'd be curious to know how you accomplished this in realtime (since others have asked). My guess would be that however you are doing XPC between the extension and the app doesn't work when the app is backgrounded.

Best, Chris

Moriquendi commented 5 years ago

I’m using this library for passing frames to the app: https://github.com/mutualmobile/MMWormhole I encode them into UIImages, pass into wormhole and then decode into pixel buffer on the app side.

Still have some issues with hooking things up but it looks promising and real-time.

I wanted to upload those frames straight from the extension but my video-call is started initiated in app and I don’t want the extension to connect to the same room as another participant (from what I found, there’s no way for the extension to add video tracks for already existing participant in the app)

ceaglest commented 5 years ago

Hi @Moriquendi,

Cool, thanks for sharing. Since we have had other questions on this subject, we are considering trying to integrate extension -> app communication in our example app.

One thing we've noticed about backgrounding is that unless audio is being played/recorded (AVAudioSession is active), then file descriptors will be forcefully closed upon the app entering a background state. This matters for us in terms of signaling and media connections but I wouldn't be surprised if this is what you are experiencing as well with IPC.

... I don’t want the extension to connect to the same room as another participant (from what I found, there’s no way for the extension to add video tracks for already existing participant in the app)

This is accurate.

Best, Chris

Moriquendi commented 5 years ago

Thanks, I’ll investigate tomorrow if the communication is terminated when backgrounderd and confirm it here.

Do you have any timeline for that extension->app communication - when could we expect that coming to the SDK?

ceaglest commented 5 years ago

Hi,

Do you have any timeline for that extension->app communication - when could we expect that coming to the SDK?

We will probably start with updated ReplayKit sample code, and then consider how to integrate the end result into our SDK. I'll share more info on timeline as I have it.

Best, Chris

Moriquendi commented 5 years ago

@ceaglest Turns out the wormhole communication does work in the background. Here's the screenshot I took ~30 sec after going to background. Imgur

Could it be possible that TVIVideoSink itself checks for app state and pauses when in the background?

Moriquendi commented 5 years ago

@ceaglest I did an experiment and swizzled UIApplication's state property to always return active. When I did it, the video stream wasn't paused.

ceaglest commented 5 years ago

Hi @Moriquendi,

@ceaglest I did an experiment and swizzled UIApplication's state property to always return active. When I did it, the video stream wasn't paused.

Are you using H.264? There are some restrictions to using VTCompressionSession while backgrounded. If you could share some debug logs I might be able to figure out why.

Chris

Moriquendi commented 5 years ago

@ceaglest Yup, I'm using H.264. builder.preferredVideoCodecs = [TVIH264Codec(), TVIVp8Codec()] Here's a log from this session: https://pastebin.com/kbHZgiKb

I then set only TVIVp8Codec as preferredVideoCodecs and indeed frames were streamed even when the app was in the background - nice :)

I still have one issue with framerate though. Not sure if Twilio is to blame but I couldn't find any other source of the problem yet. This happens only when I push frames through the flow extension->app->twilio. Doesn't seem to happen when I do extension->twilio.

So I do see in my logs that screen frames are received in rather real-time through the wormhole (definitely dozens a sec). I create a buffer and sent it straight to the video sink.

However, the video received on the participant end has a terrible framerate (sometimes it's as bad as 1 frame per 10 seconds). Sometimes the frames are entering some kind of loop and present-frames are interchanged with frames from the 30 secs ago. Is there any clue you might have on what's going on here?

Moriquendi commented 5 years ago

Nevermind my previous question. Because wormhole is passing data through files, we basically have two processes fighting for the same resource. When the extension is writing realtime to the file, the other process has no time to lock the resource for itself. I had to limit frame rate in which extension is passing frames through the wormhole so that the app has time to read them.

This is clearly not the best solution :< It feels like a hacky-workaround and I don't think it's a right direction to go. What we need is some support from Twilio SDK to add tracks to the same participant from app and extension.

ceaglest commented 5 years ago

Hi,

This is clearly not the best solution :< It feels like a hacky-workaround and I don't think it's a right direction to go. What we need is some support from Twilio SDK to add tracks to the same participant from app and extension.

We are unlikely to allow this for security reasons (see https://github.com/twilio/video-quickstart-swift/issues/357). I haven't had bandwidth to investigate sharing frames between the extension and app but I still believe that there is some promise in this approach. You should make sure that you are not actually expecting to write entire bitmaps to disk in order to share them with another process. This is not going to be very efficient, and you should share the IOSurfaces (which reference shared memory) instead.

Best, Chris