search

cordova-rtc / cordova-plugin-iosrtc

Cordova iOS plugin exposing the WebRTC W3C API
MIT License
690 stars 338 forks source link

iOS 17.5 background mode, app suspended now, call audio stops. #784

Closed bkervaski closed 3 months ago

bkervaski commented 4 months ago

YOU MUST read first!

Please use Community Forum for general technical discussions and questions.

Note: If the checkboxes above are not checked (which you do after the issue is posted), the issue will be closed, removing this checkbox will result in automatic closed issue.

Versions affected

Description

I'm not sure what iOS version this started, but with 17.5 our apps get "paused" when put in the background and at some point stops processing audio shutting down all network connections.

I have VoIP background mode enabled in XCode.

I can resolve the issue by completely removing RTCAudioController from the plugin, at which point there is no speakerphone/earpiece control, but it does keep the app running in the background like it used to.

I wrote my own audio controller for testing purposes, setting the correct category and mode, etc, to VoiceChat. Same results.

I even went as far as to integrate CallKit a bit to see if it would resolve the issue, no luck.

Basically, if I initialize any audio iOS 17.5 will pause the app in the background.

Any help or insight would be greatly appreciated.

RSATom commented 4 months ago

Do you have audio playing all the time while app is in background? if not - it's expected behavior. iOs tend to suspend app sooner or later and you should use push notifications to wake up it for audio output (VoIP or PTT pushs)

bkervaski commented 4 months ago

Do you have audio playing all the time while app is in background? if not - it's expected behavior. iOs tend to suspend app sooner or later and you should use push notifications to wake up it for audio output (VoIP or PTT pushs)

Yes, a VoIP call in progress.

Like I mentioned, if I completely remove RTCAudioController from the plugin calls will continue normally when the app is placed in the background just by selecting the "Voice Over IP" background mode in XCode.

RSATom commented 4 months ago

Are you sure have AVAudioSession activated? I had to disable audio session management completely in WebRTC library, but don't remember why exactly...

bkervaski commented 4 months ago

Are you sure have AVAudioSession activated? I had to disable audio session management completely in WebRTC library, but don't remember why exactly...

Did you retain speakerphone/earpiece toggle ability with the plugin?

RSATom commented 4 months ago

yes, but I have a little bit complicated configuration - I'm using iosrtc only for realtime video calls (and it's not supported to use it in backgrounded mode), and my own native implementation for audio messages (since I have to use PTT Framework). And only audio messages can be played/recorded in background in my case.

But the main idea is audio session have to be activated and some audio should play, or iOs suspends app. And it's not possible to workaround it. This answer I got some time ago directly from one of Apple support engineers.

bkervaski commented 4 months ago

Maybe I should ask this ... has anyone been able to use the plugin with iOS 17.5 with a call staying up when placing the app in the background and be able to toggle earpiece/speaker in the foreground?

bkervaski commented 3 months ago

Well, looks like there's no way to use the plugin any longer if you want voip calls to continue in the background. Used to work fine. Not sure when or how this broke, assuming an iOS 17.5 bug. If I remove the plugin it does work, but no way to send audio to earpiece, even with a custom plugin that just changes AVAudioSession's route, does nothing.

With the plugin, calls (and all websockets) shut down in 5-15 seconds. Without the plugin (and no earpiece/speakerphone control) calls and websockets will stay up indefinitely in the background.

RSATom commented 3 months ago

@bkervaski what do you mean by "If I remove the plugin it does work"? Do you mean you are switching to use WebRTC implementation from WebView?

bkervaski commented 3 months ago

@bkervaski what do you mean by "If I remove the plugin it does work"? Do you mean you are switching to use WebRTC implementation from WebView?

Yes, but no way to redirect audio to the earpiece using native WebRTC.

RSATom commented 3 months ago

The reason is simple. WebView works in separate process. And obviously that process has it's own reference to AVAudioSession with it's own settings. And your app just doesn't have access to that AVAudioSession and can't do any changes to it.

RSATom commented 3 months ago

So if you need extended AVAudioSession management - you have to use WebRTC implementation inside your app (with iosrtc or your own). It's also related to use PTT Framework and CallKit.

RSATom commented 3 months ago

Do you have any minimal example with reproducible issue? For me it still looks like as not activated Audio Session or missing background mode.

hthetiot commented 3 months ago

Hello @bkervaski, did you send 100 USD to me for this issue via Paypal ? If so please 🙏 provide email so I can give it back or I can give it to another good cause or to @RSATom that is really helping you instead.

I don't have the bandwidth to tackle this issue right now and @RSATom is more knowledgeable. That why I can't accept your donation.

Harold.

hthetiot commented 3 months ago

@bkervaski you need to provide sample javascript to reproduce.

You can provide in this issue a modified version of this file if you don't know where to start: https://github.com/cordova-rtc/cordova-plugin-iosrtc/blob/master/extra/renderer-and-libwebrtc-tests.js

bkervaski commented 3 months ago

Hello @bkervaski, did you send 100 USD to me for this issue via Paypal ? If so please 🙏 provide email so I can give it back or I can give it to another good cause or to @RSATom that is really helping you instead.

I don't have the bandwidth to tackle this issue right now and @RSATom is more knowledgeable. That why I can't accept your donation.

Harold.

I did. Please pay it forward to @RSATom if you like or I'll donate to him separately.

bkervaski commented 3 months ago

The issue appears to be that when using the plugin's WebRTC binaries, iOS will close WKWebView's JsSIP websocket after a few seconds, terminating the call.

Whereas, when using WKWebView's native WebRTC it will keep the websocket open, but no earpiece/speaker control.

Is there anyone pulling this off successfully?

@RSATom Thank you, I'm going to copy and strip the project down to just the plugin and JsSIP and post it for review. It's not a missing background mode.

RSATom commented 3 months ago

@bkervaski I have an idea what is going on in your case.

First of all, as I said before, WebView always works in separate process (read separate application). And if take this into account, when you use iosrtc, you have different parts of logic of your app spread across 2 separate processes (signalling part, and WebRTC Peer Connection part). And iOs manages state of that processes independently. And since WebView's process doesn't play any sound (and as consequence doesn't have activated Audio Session) - iOs have all reasons to to suspend that process on backgrounding. And no, it's not possible to have 2 separate processes with activated Audio Session at the same time (except mixWithOthers AVAudioSession option but it's totally different story).

So the only solution I see here - implement Cordova plugin with signalling part inside (I have this in my app). Then all parts will be in the same process and suspending WebView's process will not affect your logic.

Also, you should keep in mind, if app loses activation of AudioSession in backgrounded state - it's not possible to reactivate it again without foregrounding app by user. The only exception is - VoIP push notifications (+ CallKit) or PTT push notifications.

hthetiot commented 3 months ago

I did. Please pay it forward to @RSATom if you like or I'll donate to him separately.

@bkervaski Done, @RSATom you should have received a 100 USD paypal payment on your github public email. Thank you @RSATom for helping the community as always.

RSATom commented 3 months ago

Oh... It was bad idea. I have some troubles with using PayPal. I'll try to workaround or reject.

hthetiot commented 3 months ago

Oh... It was bad idea. I have some troubles with using PayPal. I'll try to workaround or reject.

Ah shit, sorry i had my own issue with paypal account, feel free to reject and i will send elsewhere, you can reach me here hthetiot at sylaps.com if you want to share privatly the paypal account to use.

bkervaski commented 3 months ago

Reached out to Apple through the dev program, basically they said "tough luck" on earpiece to speaker toggle for WKWebView.

So, we're having to work around it by suspending calls while the app is in the background and re-establish when the users brings it back.

Not ideal, really takes away from the experience.

RSATom commented 3 months ago

Yep, WKWebView is in separate process -> it has separate Audio Session -> there is no any control from your app over Audio Session in separate process...

bkervaski commented 3 months ago

Yep, WKWebView is in separate process -> it has separate Audio Session -> there is no any control from your app over Audio Session in separate process...

Right, and ios rtc gets suspended now, so no more phone calls in the background.

hthetiot commented 3 months ago

To make iosrtc works in background you need to add this plugin cordova-plugin-background-mode@0.7.3 and add the folllowing code:

document.addEventListener('deviceready', function () {
    if (cordova.plugins.backgroundMode) {
         cordova.plugins.backgroundMode.enable();
    }
}, false);

This can be tested using Sylaps on iOS https://apps.apple.com/us/app/sylaps/id954016392

Tested on iPhone and iPAd on, 17.5.1, notice that video from the device will go black while application in background, but audio input and ouput keep working.

bkervaski commented 3 months ago

To make iosrtc works in background you need to add this plugin cordova-plugin-background-mode@0.7.3 and add the folllowing code:

document.addEventListener('deviceready', function () {
    if (cordova.plugins.backgroundMode) {
         cordova.plugins.backgroundMode.enable();
    }
}, false);

This can be tested using Sylaps on iOS https://apps.apple.com/us/app/sylaps/id954016392

Tested on iPhone and iPAd on, 17.5.1, notice that video from the device will go black while application in background, but audio input and ouput keep working.

Thank you very much for posting, but I've tried this. When I use cordova-plugin-background-mode@0.7.3 event "activate" to call isActive(), it reports "true". Just before this, I get the cordova "pause" event. I would think this even wouldn't fire if background mode is active.

No matter what I do the WKWebView won't keep the JsSIP websocket registration alive. My SIP server (Asterisk) just reports the connection closed a few seconds with a "Connection reset by peer" error.

If a call is active, I'll get the websocket closed error in the javascript console. If not, when I resume the app, the javascript console reports the websocket errors.

The only thing the debugger reports is a two "0x109024d40 - ProcessThrottlerTimedActivity::activityTimedOut" just before and after the app is paused and then a series of "Invalidating grant <invalid NS/CF object> failed" which may or may not be related.

Edit: Oddly, and to reiterate, if I don't use ios rtc, the JsSIP websocket does stay active in the background while on a call. Of course, no speaker/earpiece toggle without the plugin.

hthetiot commented 3 months ago

I'm sorry but I disagree with your test result, Websocket stay alive using JSSip and freeswitch as server or P2P in my tests via Sylaps app (I'm the owner of it).

If not using iosrtc work for you then I guess you have your solution, iosrtc is becoming less relevant this days with built-in webrtc from WkWebview anyway.

hthetiot commented 3 months ago

Note: iosrtc does not modify websocket behavior.

bkervaski commented 3 months ago

I'm not suggestion that ios-rtc changes websocket behavior, I'm suggestion WKWebView will keep the sockets open when there is an active call initiated with native WebRTC.

I agree that the plugin is less relevant, however, I can find no way to toggle earpiece/speaker with WKWebView so I've been holding on to the plugin for this purpose only.

Thank you for your help.

bkervaski commented 3 months ago

Websocket stay alive using JSSip and freeswitch as server or P2P in my tests via Sylaps app (I'm the owner of it).

While using ios-rtc plugin or just WKWebView's webrtc?

hthetiot commented 2 months ago

While using ios-rtc plugin or just WKWebView's webrtc?

Both we test both mode and on ios <= 14.5 we use iosrtc when missing in webview.