No audio output on iOS when app is in the background

[amilcar-uptech]

Describe the bug I have a Flutter app that handles chats and voice calls (through the SIP plugin of a Janus gateway, using the janus_client package) and handling incoming/outgoing calls through the callkeep package. It works mostly fine on Android, but I've run into a particular issue when running the app on iOS:

Basically, I have the app running on the background and when the device is locked so that it's able to receive calls (using the flutter_background_service for iOS). However, while I am able to receive and answer these calls, as well as send audio to my peer, I seem to be unable to receive audio from my peer; this only happens when the app was in the background state (and when the device is locked), as the calls work just fine when the app is on the foreground. I have tried adding the background audio capability to the app on the Info.plist file, among others, but it didn't seem to make much of a difference.

I also tried using a suggestion from #816 , but I just get the following in the console:

Failed to set audio session category: Error Domain=NSOSStatusErrorDomain Code=2003329396 "(null)"

I've also tried using the audio_session package to attempt to replicate the previously mentioned suggestion through Dart code, but that also didn't seem to make much of a difference. Is there some sort of workaround for an issue like this?

To Reproduce

Expected behavior When answering an incoming call while the app is in the background or when the device is locked, the device should receive and send audio.

Platform information

• Flutter version:

  
  [✓] Flutter (Channel master, 3.1.0-0.0.pre.1596, on macOS 11.6.5 20G527
  darwin-x64, locale en-MX)
  • Flutter version 3.1.0-0.0.pre.1596 on channel master at
    /Users/user/flutter/flutter
  • Upstream repository https://github.com/flutter/flutter.git
  • Framework revision 0d209f2e4c (4 days ago), 2022-07-11 11:30:09 -0400
  • Engine revision 14fbf746bf
  • Dart version 2.18.0 (build 2.18.0-271.0.dev)
  • DevTools version 2.15.0

[✓] Android toolchain - develop for Android devices (Android SDK version 32.1.0-rc1) • Android SDK at /Users/user/Library/Android/sdk • Platform android-32, build-tools 32.1.0-rc1 • Java binary at: /Applications/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6222593) • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 13.2.1) • Xcode at /Applications/Xcode.app/Contents/Developer • Build 13C100 • CocoaPods version 1.11.3

[✓] Chrome - develop for the web • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 4.1) • Android Studio at /Applications/Android Studio.app/Contents • Flutter plugin can be installed from: 🔨 https://plugins.jetbrains.com/plugin/9212-flutter • Dart plugin can be installed from: 🔨 https://plugins.jetbrains.com/plugin/6351-dart • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6222593)

[✓] VS Code (version 1.68.1) • VS Code at /Applications/Visual Studio Code.app/Contents • Flutter extension version 3.44.0

[✓] VS Code (version 1.67.0) • VS Code at /Users/user/Downloads/Visual Studio Code.app/Contents • Flutter extension version 3.44.0

[✓] Connected device (3 available) • iPhone SE • ios • iOS 14.4.2 18D70 • macOS (desktop) • macos • darwin-x64 • macOS 11.6.5 20G527 darwin-x64 • Chrome (web) • chrome • web-javascript • Google Chrome 103.0.5060.114

[✓] HTTP Host Availability • All required HTTP hosts are available

* **Plugin version**: ^0.8.7 (as per listed on `janus_client: ^2.2.3` )
* **OS**:  iOS
* **OS version**: 14.4.2 18D70

[ycherniavskyi]

I am investigating this issue for my project. As I see for now the cause of this issue is not respecting provider:didActivateAudioSession: and provider:didDeactivateAudioSession: handlers by callkeep and by WebRTC.

It seems that to resolve such issues we need to implement the possibility of next integration with WebRTC library for cases when CallKit used:

CallKit integration Disclaimer: I am not sure if this is the best way doing it but this has worked for me so far:

1. Configure WebRTC audio session to use manual audio and disable audio:
   1. RTCAudioSession.sharedInstance().useManualAudio = true
   2. RTCAudioSession.sharedInstance().isAudioEnabled = false
2. On your CXProvider delegate's provider(CXProvider, didActivate: AVAudioSession) method:
   1. Call RTCAudioSession.sharedInstance().audioSessionDidActivate with the AVAudioSession from the CXProvider
   2. Enable audio: RTCAudioSession.sharedInstance().isAudioEnabled = true
3. On your CXProvider delegate's provider(CXProvider, didDeactivate: AVAudioSession) call RTCAudioSession.sharedInstance().audioSessionDidDeactivate with the AVAudioSession from the CXProvider

WebRTC and CallKit talk from 2016: https://youtu.be/JB2MdcY1MKs?t=6m23s

credit to stasel from WebRTC-iOS

(Taken from Can we use Callkit for Apps with WebRTC)

[ycherniavskyi]

It seems that mentioned mechanism is worked! 🎉

But now, I need to find/invent a proper API to introduce this functionality in flutter_webtrc.

[amilcar-uptech]

Those are wonderful news! For the time being, is there a way to replicate it on my app? Or should I wait for an upcoming versionof webrtc? I'm not too well-versed in Swift/Objective-C.

[ycherniavskyi]

It will be additional API In flutter_webtrc, webrtc already has all messers functionality - 1, 2 and 3.

[ycherniavskyi]

Just to collect all useful links related to such issues:

• CallKit:No sound when I use WebRTC
• No sound after connecting to WebRTC when app is launched in background using PushKit and CallKit on the lock screen
• Apple CallKit using VOIP push notifications

[amilcar-uptech]

Oh, I meant flutter_webrtc, my bad. I'll try looking into some of the links, see if I can understand a bit of it. 😅

[CHA7R1K]

I am investigating this issue for my project. As I see for now the cause of this issue is not respecting provider:didActivateAudioSession: and provider:didDeactivateAudioSession: handlers by callkeep and by WebRTC.

It seems that to resolve such issues we need to implement the possibility of next integration with WebRTC library for cases when CallKit used:

CallKit integration Disclaimer: I am not sure if this is the best way doing it but this has worked for me so far:

1. Configure WebRTC audio session to use manual audio and disable audio:

   1. RTCAudioSession.sharedInstance().useManualAudio = true
   2. RTCAudioSession.sharedInstance().isAudioEnabled = false

2. On your CXProvider delegate's provider(CXProvider, didActivate: AVAudioSession) method:

   1. Call RTCAudioSession.sharedInstance().audioSessionDidActivate with the AVAudioSession from the CXProvider
   2. Enable audio: RTCAudioSession.sharedInstance().isAudioEnabled = true
3. On your CXProvider delegate's provider(CXProvider, didDeactivate: AVAudioSession) call RTCAudioSession.sharedInstance().audioSessionDidDeactivate with the AVAudioSession from the CXProvider

WebRTC and CallKit talk from 2016: https://youtu.be/JB2MdcY1MKs?t=6m23s credit to stasel from WebRTC-iOS

(Taken from Can we use Callkit for Apps with WebRTC)

Congratulations! What a great news.

However, I'm still struggle on this problem even following your solution. I notice that you using CallKeep to handle incoming call which mine using FlutterCallkitIncoming. I'm not quite believe that the different of call handler plugin is related to the problem but I'll check out Callkeep to make sure of that.

[ycherniavskyi]

Actually, I don't use Callkeep. I use the in-house Callkeep analog, which will be open-sourced and published to https://pub.dev as soon as the Android version is finished.

As for callkeep and its inspirer React Native CallKeep they init AVAudioSession in different places on different events and in such way not respecting CallKit interface. So in my version Callkeep, I completely remove this AVAudioSession inits, then apply "CallKit integration" recommendation, and everything seems to work as expected 🤷🏻‍♂️.

[amilcar-uptech]

Is there an example code and/or repo that could be used for reference to this solution? I'm trying to figure this out on my own but some guidance would be appreciated.

[CHA7R1K]

Actually, I don't use Callkeep. I use the in-house Callkeep analog, which will be open-sourced and published to https://pub.dev as soon as the Android version is finished.

As for callkeep and its inspirer React Native CallKeep they init AVAudioSession in different places on different events and in such way not respecting CallKit interface. So in my version Callkeep, I completely remove this AVAudioSession inits, then apply "CallKit integration" recommendation, and everything seems to work as expected 🤷🏻‍♂️.

Thank you for the information. So, in this case, I need to consult with plugin publisher to work it out since editing CXProvider delegate in plugin's source code by myself doesn't seem to make it work.

Is there an example code and/or repo that could be used for reference to this solution? I'm trying to figure this out on my own but some guidance would be appreciated.

This issue is also new to me so I can't offer much help. But if you are using call handler plugin (like Callkeep or FlutterCallkitIncoming) there are CXProviderDelegate in their source code which have both provider(CXProvider, didDeactivate: AVAudioSession) and provider(CXProvider, didActivate: AVAudioSession) you can start from there. Unfortunately, I'm still can't make it through even after implement ycherniavskyi's solution either because I messed up the plugin source code or I have no idea how plugin really works.

[amilcar-uptech]

Yes, I'm using Callkeep in this case. I tried to change the code as per suggested, but given I'm not too familiar with Objective-C, I don't know how I can make use of RTCAudioSession, as it is something external of the Callkeep plugin itself.

[ycherniavskyi]

@amilcar-uptech, no, you can't (at last easily) access RTCAudioSession from Callkeep. That is why I will extend flutter_webrtc API and introduce the necessary methods to control RTCAudioSession. So, in the end, my callkeep implementation only calls didActivate & didDeactivate handles in App/Flutter, and then App/Flutter calls new APIs to fulfill "CallKit integration" recommendation.

[amilcar-uptech]

I understand. For now, I just wanted to check if there were any updates regarding this? Cheers!

[magdabudzinskagoodsoft]

Do you know when the updates can be expected?

[magdabudzinskagoodsoft]

I try implement code as suggested. I edited CXProvider delegate in callkit plugin source code. @ycherniavskyi you said, that you completely remove AVAudioSession inits in Callkeep. Can you provide details of what exactly are you removing?

[TatankaConCube]

@ycherniavskyi could you please share your solution with us, cause we applied this solution in our plugin but it still doesn't work if the call started from the killed app state

[magbdev]

@TatankaConCube look at my code. Audio session configuration in the right places fix audio in background/terminated state.

[TatankaConCube]

@magbdev as I can see, you use it only there, we have the similar solution on the master branch, but customers still report the issue if accept a call from the lock screen in the killed state

[TatankaConCube]

I use the in-house Callkeep analog, which will be open-sourced and published to https://pub.dev/ as soon as the Android version is finished.

@ycherniavskyi do you have any success with it? could you share your solution if it is possible?

[amilcar-uptech]

@ycherniavskyi Are there any updates regarding this issue?

[behzodfaiziev]

Eventually, I found a solution of this issue. 1) Enable background mode from XCode. 2) Enable Audio, Airplay..... (which is first one)

3)That's all.

We need to update documentation to enable background mode. I can open PR in readme.md

@amilcar-uptech @ycherniavskyi @CHA7R1K @TatankaConCube @magbdev

[amilcar-uptech]

@behzodfaiziev I believe I already had this activated beforehand, but I'll need to double check since I am just coming back to testing our app on iOS, and I need to do some fixes before getting to the calls proper. Hopefully it does work on my end.

[TatankaConCube]

@behzodfaiziev unfortunately we already used this parameter in our project and the issue still reproduces. check one more time on your side the starting the call from the killed app state. in our case, the issue reproduces only with the killed state, it works correctly when the app is in the background even when the app is inactive for a long time, but the issue reproduces after removing the app from the recents and receiving a call and immediately accepting it.

[behzodfaiziev]

There were three issues regarding soundOutput in the background. @TatankaConCube Mistakenly, I shared one of the solutions in this issue. I'm sorry. I mentioned others isssues so they can see it.

[TatankaConCube]

@behzodfaiziev understood, no problem)))

[TatankaConCube]

Looks like we found the solution for our plugin. We take a peek at the solution from the flutter_callkit_incoming. That is this part of the code:

        var userInfo : [AnyHashable : Any] = [:]
        let intrepEndeRaw = AVAudioSession.InterruptionType.ended.rawValue
        userInfo[AVAudioSessionInterruptionTypeKey] = intrepEndeRaw
        userInfo[AVAudioSessionInterruptionOptionKey] = AVAudioSession.InterruptionOptions.shouldResume.rawValue
        NotificationCenter.default.post(name: AVAudioSession.interruptionNotification, object: self, userInfo: userInfo)

which needs to be called in the callback didActivateAudioSession before the configureAudioSession method.

The plugin react-native-callkeep uses the same logic but it is written on the Objective-C and you can use their code on the Objective-C:

    NSDictionary *userInfo
    = @{
        AVAudioSessionInterruptionTypeKey: [NSNumber numberWithInt:AVAudioSessionInterruptionTypeEnded],
        AVAudioSessionInterruptionOptionKey: [NSNumber numberWithInt:AVAudioSessionInterruptionOptionShouldResume]
    };
    [[NSNotificationCenter defaultCenter] postNotificationName:AVAudioSessionInterruptionNotification object:nil userInfo:userInfo];

[1cue-jrupp]

@TatankaConCube We're using flutter-webrtc together with flutter_callkit_incoming and experiencing the same issue. On the first incoming call with closed app there is no audio output when answering from lockscreen. Mic works fine. On a second call everything works. Did you find a solution? Where do we need to implement it? Is it flutter_callkit_incoming specific or flutter-webrtc specific?

[TatankaConCube]

@1cue-jrupp this solution already implemented in flutter_callkit_incoming, we have applyed it in our plugin and the problem stopped reproducing

[1cue-jrupp]

@TatankaConCube We're still experiencing this issue with the latest versions, we're additionaly using sip-ua-helper. So is a new issue with further details better be placed in flutter_callkit_incoming repo?

[fullflash]

I am investigating this issue for my project. As I see for now the cause of this issue is not respecting provider:didActivateAudioSession: and provider:didDeactivateAudioSession: handlers by callkeep and by WebRTC.

It seems that to resolve such issues we need to implement the possibility of next integration with WebRTC library for cases when CallKit used:

CallKit integration Disclaimer: I am not sure if this is the best way doing it but this has worked for me so far:

1. Configure WebRTC audio session to use manual audio and disable audio:

   1. RTCAudioSession.sharedInstance().useManualAudio = true
   2. RTCAudioSession.sharedInstance().isAudioEnabled = false

2. On your CXProvider delegate's provider(CXProvider, didActivate: AVAudioSession) method:

   1. Call RTCAudioSession.sharedInstance().audioSessionDidActivate with the AVAudioSession from the CXProvider
   2. Enable audio: RTCAudioSession.sharedInstance().isAudioEnabled = true
3. On your CXProvider delegate's provider(CXProvider, didDeactivate: AVAudioSession) call RTCAudioSession.sharedInstance().audioSessionDidDeactivate with the AVAudioSession from the CXProvider

WebRTC and CallKit talk from 2016: https://youtu.be/JB2MdcY1MKs?t=6m23s credit to stasel from WebRTC-iOS

(Taken from Can we use Callkit for Apps with WebRTC)

after didActivate event you still should setCategory to .playAndRecord if you want to record during audio transmission for our experiment with IOS Native PushToTalk framework audioSession category that comes with didActivate method is .playback

[sanekyy]

Same issue, waiting for fix in plugin 🙏

[petroniuchacz]

Same issue for me.