[iOS] Callkit & WebRTC Randomly no audio

juskuc commented 1 year ago

First and foremost - thanks for the library! Now on to the question, I'm only interested in iOS atm, so given that we accept the call from CallKit UI buttons, it immediately starts the call timer, however as our application is based on WebRTC, the action of accepting does not necessarily mean a successful connection between the two peers has been established, which is not a good UX as the connection part can take ~3-4s or more. How can I manage the actual start of the CallKit's call from dart/flutter code so that I would only call it once I know both peers have established a connection? I've tried saving the incoming call action in a variable in swift code and creating a method channel to fulfill it, but doing that made my call sound disappear on the peer that was accepting the call. Any help is greatly appreciated!

hiennguyen92 commented 1 year ago

I'm not sure, but I think you can try using this function.

await FlutterCallkitIncoming.setCallConnected(this._currentUuid);

juskuc commented 1 year ago

Sadly, I've tried it, but it does not do the job, the timer is started instantly, as soon as I press accept on CallKit UI (as the CXAnswerCallAction is fulfilled). Just after writing my question I found an interesting PR which was what I was looking for https://github.com/hiennguyen92/flutter_callkit_incoming/pull/246, but I just tested it out - experiencing the same issue of having no audio on receiver end of the call 🤔

juskuc commented 1 year ago

I completely understand if you do not have the time to work on this, but this question is something that is very important to me at the moment. I'd be more than happy to fix it for myself if you could at least guide me in the right direction? @hiennguyen92

anhquangmobile commented 10 months ago

@juskuc same issue. I have read the code of this package and discovered that the count timer code is calculated as follows. Do you have any solution yet?

anhquangtech commented 10 months ago

Sadly, I've tried it, but it does not do the job, the timer is started instantly, as soon as I press accept on CallKit UI (as the CXAnswerCallAction is fulfilled). Just after writing my question I found an interesting PR which was what I was looking for #246, but I just tested it out - experiencing the same issue of having no audio on receiver end of the call 🤔

@juskuc I just tried this solution and it works. You just need to call the function " FlutterCallkitIncoming.startIncomingCall();" as soon as your Webrtc application connects -> Now the timer starts working & audio working.

I think "but I just tested it out - experiencing the same issue of having no audio on receiver end of the call" because You don't call function "FlutterCallkitIncoming.startIncomingCall()" anywhere.

anh @hiennguyen92 chỗ support active bộ đếm thời gian này anh check pull request này #246 em thấy chưa được merge á anh.

hiennguyen92 commented 10 months ago

@anhquangtech ok e. mấy nay bận quá nên chưa check được.

anhquangtech commented 10 months ago

@anhquangtech ok e. mấy nay bận quá nên chưa check được.

Dạ, em cảm ơn anh. À, có 1 lỗi bên version 2.0.0+2 á anh.

Bản 2.0.0+2 em thấy có fix vụ endCall (ở 1.0.3+3) để tắt màn hình full screen notification bên Android (em đang test trên Samsung android 14). Tuy nhiên, gây ra lỗi không mong muốn là cứ test vài lần việc mở màn hình notification full screen bên android thì khi ở màn hình khoá -> cuộc gọi đến nó không có hiển thị notification full screen nữa ạ.

Hiện tại em đang back về bản 1.0.3+3 thì case này bình thường ạ.

juskuc commented 10 months ago

@anhquangmobile @anhquangtech

Hey! I do not have a solution yet, could you please elaborate? I've taken a look at your fork - I see You've done something similar to me where you save CXAnswerCall action in a variable and then fulfill it from Flutter with FlutterCallkitIncoming.startIncomingCall();. I did call this function, the timer would start, but there would be no sound - does it work for you in first and every subsequent calls after that? Does the audio still work if app is woken up with VoIP from terminated app state? What about accepting an audio call from a locked screen? Do all of these cases work? 😄

Also: I noticed you've removed setCallConnected() method - how do you then tell the callkit that a call has successfully connected from the caller's POV?

Sorry for a lot of questions - this topic is important to me, I'd be happy to discuss it over a google meet if You have a free minute or two :)

hiennguyen92 commented 10 months ago

please check version 2.0.1 and changelog

juskuc commented 10 months ago

@hiennguyen92 taking a look, will let you know how it goes

anhquangtech commented 10 months ago

@anhquangmobile @anhquangtech

Hey! I do not have a solution yet, could you please elaborate? I've taken a look at your fork - I see You've done something similar to me where you save CXAnswerCall action in a variable and then fulfill it from Flutter with FlutterCallkitIncoming.startIncomingCall();. I did call this function, the timer would start, but there would be no sound - does it work for you in first and every subsequent calls after that? Does the audio still work if app is woken up with VoIP from terminated app state? What about accepting an audio call from a locked screen? Do all of these cases work? 😄

Also: I noticed you've removed setCallConnected() method - how do you then tell the callkit that a call has successfully connected from the caller's POV?

Sorry for a lot of questions - this topic is important to me, I'd be happy to discuss it over a google meet if You have a free minute or two :)

@juskuc @hiennguyen92 Sorry everyone I just checked again. Sound only works on the first call. The next time you call, the microphone will not be activated for the call. So the device receiving the call can only listen and not speak.

juskuc commented 10 months ago

@anhquangtech yeah, this is similar to what I am experiencing with the exception. Updating to ^2.0.1 did not seem to fix the issue for me personally, now it's just random what happens. Sometimes it connects and there's sound on both, sometimes it connects and sound only works on one of the peers, with no consistency, seems like a race condition is happening somewhere in Swift. @hiennguyen92 would you be so kind to look into this? Or maybe you have an idea what could be wrong so I could try to debug the code myself? Everything works as intended on the first call, after that, in every subsequent call something goes wrong. Any ideas? 🤷‍♂️

anhquangtech commented 10 months ago

Hi magbdev. Sorry for bothering you. Have you encountered this situation? Many thanks for your support.

anhquangtech commented 10 months ago

Anh @hiennguyen92 em check version 2.0.1 có update code chỗ này. Chỗ này nếu em muốn dùng thì cần viết thêm logic để hàm performRequest để khi cuộc gọi webrtc sẵn sàng thì bộ đếm giờ của hàm onAccept mới hoạt động đúng ko á anh. Nếu ko thì callkit sẽ hiển thị "Đang kết nối âm thanh".

Với chỗ này hàm action.fulfill sẽ chạy sau khi hàm onAccept được kích hoạt thì trạng thái callkit sẽ là: Đang kết nối âm thanh -> vài giây sau thì onAccept success -> Bộ đếm time hoạt động

Mong anh hỗ trợ giúp em với ạ.

anhquangtech commented 10 months ago

Anh @hiennguyen92 em check version 2.0.1 có update code chỗ này. Chỗ này nếu em muốn dùng thì cần viết thêm logic để hàm performRequest để khi cuộc gọi webrtc sẵn sàng thì bộ đếm giờ của hàm onAccept mới hoạt động đúng ko á anh. Nếu ko thì callkit sẽ hiển thị "Đang kết nối âm thanh".

Với chỗ này hàm action.fulfill sẽ chạy sau khi hàm onAccept được kích hoạt thì trạng thái callkit sẽ là: Đang kết nối âm thanh -> vài giây sau thì onAccept success -> Bộ đếm time hoạt động

Mong anh giải đáp thắc mắc giúp em với ạ.

hiennguyen92 commented 10 months ago

mấy chỗ này a nghĩ nên dùng để call API báo hiệu server biết là state của call thôi. chứ còn kết nối, hiện video thì nên set ở flutter. e có thể check https://github.com/hiennguyen92/flutter_callkit_incoming/blob/master/example/lib/calling_page.dart

hiennguyen92 commented 10 months ago

cũng có thể phải sử dụng nó để kết nối webRTC ở đây(trong trường hợp nhận cuộc gọi khi app đang tắt- Flutter Engine chưa đc attach)

anhquangtech commented 10 months ago

mấy chỗ này a nghĩ nên dùng để call API báo hiệu server biết là state của call thôi. chứ còn kết nối, hiện video thì nên set ở flutter. e có thể check https://github.com/hiennguyen92/flutter_callkit_incoming/blob/master/example/lib/calling_page.dart

Ở Flutter thì em có 1 màn hình callscreen để hiển thị bộ đếm giờ. Tuy nhiên, trong trường hợp app ở lockscreen (dùng callkit của IOS) thì khi nhấp accept call thì bộ đếm giờ chạy luôn. Em đang cần set time start delay lại ở callkit á anh, kiểu như thế này ạ

Ban đầu sẽ như này:
Khi webrtc sẵn sàng thì sẽ như này:

Em có thấy giải pháp này đúng nhu cầu em đang muốn (khi webrtc bắt đầu đàm thoại thì gọi hàm "startCallIncoming" để run action.fulfill() ) như nó gây ra lỗi là chỉ work ở cuộc accept call đầu tiên (hoặc tuỳ lúc, không ổn định). Những lần accept call sau thì không ép được micro vào đàm thoại đúng với webrtc ạ.

hiennguyen92 commented 10 months ago

https://pub.dev/packages/flutter_callkit_incoming/versions/2.0.1-dev.2 e thử version này

đồng thời e add thêm cái này nếu dùng webRTC, để audio không bị mất tiếng

anhquangtech commented 10 months ago

Em cũng phải viết logic mới cho hàm này để biết lúc nào webrtc active đúng không á anh. Ví dụ, hàm này em phải sửa lại thành:

Khi webrtc sẵn sàng thì onAccept sẽ được gọi (giống như hàm "startIncomingCall" ở trên FlutterCallkitIncoming.startIncomingCall();).

juskuc commented 10 months ago

@hiennguyen92 @anhquangtech hey, have you guys found anything yet?

juskuc commented 10 months ago

One thing we have noticed is that during the calls on which audio is missing, in iOS control center Mic Mode is shown as Off.

anhquangtech commented 10 months ago

@juskuc @hiennguyen92 After debug & self-test. I found a solution as follows: Apply solution #246. You must call function FlutterCallkitIncoming.startIncomingCall(); before webrtc active.

Then (not work): I call startIncomingCall() after webrtc connected and in ACCEPT state: Incoming callkit -> click accept button in callkit IOS -> register candidate (webrtc) -> ACCEPT state of webrtc (I call startIncomingCall() in here) -> Not work.
Now (work): Incoming callkit start -> click accept button -> register candidate of webrtc (I call startIncomingCall() in here) -> ACCEPT state of webrtc ->Work.

I think you must call function startIncomingCall(); before webrtc connected. Because, audio & micro must ready before the webrtc call is successfully connected. I tested when app foreground, background, terminate and lock screen.

juskuc commented 10 months ago

@anhquangtech thanks, will go try this now. Just to clarify - so now your calls work 100% of the time? Calling from user A to user B and reverse? and from all states - correct?

Also, could you elaborate, what do you mean by 'register' candidate of webrtc? Do you mean save candidate locally, or create answer peer connection and add candidates to it?

Also, which version of flutter_callkit_incoming are you using? 2.0.1-dev.2 or prod version? And how does your AppDelegate.swift look like? Do you use this code

RTCAudioSession.sharedInstance().useManualAudio = true
RTCAudioSession.sharedInstance().isAudioEnabled = false

etc?

anhquangtech commented 10 months ago

@juskuc I forked version 1.0.3+3 and edited according to #246. Register mean websocket connected success. You know, to connect A to B peer-to-peer. Webrtc must have 1 signaling server, this is websocket. So, I call function startIncomingCall() when B connected websocket success.

anhquangtech commented 10 months ago

@juskuc In my case, I only use Callit of IOS for incoming call. With case out going call, I make out going call with webrtc out going call.

anhquangtech commented 10 months ago

@anhquangtech thanks, will go try this now. Just to clarify - so now your calls work 100% of the time? Calling from user A to user B and reverse? and from all states - correct?

Also, could you elaborate, what do you mean by 'register' candidate of webrtc? Do you mean save candidate locally, or create answer peer connection and add candidates to it?

Also, which version of flutter_callkit_incoming are you using? 2.0.1-dev.2 or prod version? And how does your AppDelegate.swift look like? Do you use this code
RTCAudioSession.sharedInstance().useManualAudio = true
RTCAudioSession.sharedInstance().isAudioEnabled = false
etc?

I use version 1.0.3+3 and don't change anything in file AppDelegate.swift

rytisder commented 10 months ago

Finally, I think we have fixed the problem 🙇

Issue Summary:

Upon closer inspection, the iOS notification center displayed "Mic Mode Off," indicating that the microphone wasn't actively capturing audio from iOS side.

Investigation:

To address this, we conducted an extensive investigation into various components and systems. The following resources provided invaluable information and guidance:

Apple's Audio Session Programming Guide Apple Documentation
AVAudioSession Article Medium Article
AVAudioSession Documentation Apple Developer
WebRTC and CallKit talk from 2016 YouTube Video
CallKit + WebRTC integration practical example - https://github.com/stasel/WebRTC-iOS/#callkit-integration
There is a specific issue with the audio I/O and CallKit - https://stackoverflow.com/a/55393873/7092969

Resolution

https://github.com/hiennguyen92/flutter_callkit_incoming/commit/b2fce5b77d19529afc320105eb1d0d434f3812bf - MVP version

image (6)

image (5)

After thoroughly reviewing the documentation and resources, we identified the core issue. It became clear that the audio session needed manual configuration to ensure that it was properly activated during the RTC session. Specifically, we implemented the following steps:

Manual Audio Configuration: Prior to initiating the RTC session, we manually configured the audio settings to ensure that the device was prepared to capture and transmit audio. Activation Post CXProvider didActive: We discovered that the optimal time to activate the WebRTC audio was right after the CXProvider's didActive event. This ensured that the audio session was in the correct state and ready to handle the audio stream.

Conclusion

By setting the audio session manually and timing the activation of WebRTC's audio correctly, we successfully resolved the no-audio issue.

InvalidReferenceException commented 8 months ago

For anyone else I was able to fix the issue by setting audioSessionActive: false and configureAudioSession:false in the CallKitEntities.IosParams. That by itself solved the problem for me without having to change anything in the WebRTC side 👍

     ios: const CallKitEntities.IOSParams(
        iconName: 'LaunchImage',
        handleType: '',
        supportsVideo: true,
        maximumCallGroups: 1,
        maximumCallsPerCallGroup: 1,
        audioSessionMode: 'videoChat',
        // these two are FALSE because they interfere with WEBRTC audio stream and session if they are true.
        audioSessionActive: false,
        configureAudioSession: false,
        audioSessionPreferredSampleRate: 44100.0,
        audioSessionPreferredIOBufferDuration: 0.005,
        supportsDTMF: true,
        supportsHolding: false,
        supportsGrouping: false,
        supportsUngrouping: false,
        ringtonePath: 'ringtoneshort.aiff',
      ),

datpt11 commented 8 months ago

Maybe the same situation I'm having on Android

dpatel-jac commented 7 months ago

@juskuc , I face the same issues in my case I am using agora for video call. and I notice One thing we have noticed is that during the calls on which audio is missing, in iOS control center Mic Mode is shown as Off. Have found any solution on it?

hiennguyen92 / flutter_callkit_incoming