Known audio issues in iOS Safari browser

anna-vasilko commented 4 years ago

This ticket describes a couple of known audio issues in the iOS Safari browser and provides most recent status and Twilio recommendations on how to handle them: Note:

Both described issues are rarely reproducible. If you hit an audio problem all the time, make sure to go through this guide to avoid most common pitfalls
Both issues are not Twilio SDK bugs, but iOS Safari browser issues.

1. Remote audio does not play occasionally in iOS Safari browser

Webkit Bug: https://bugs.webkit.org/show_bug.cgi?id=198545 Status: This issue was fixed by Apple, the fix is released in iOS version 13.6. Reproducibility of this issue before the fix

Group rooms - occurs in 0-2 rooms out of 50
Peer to peer rooms - occurs in 6-9 rooms out of 50

2. RemoteParticipants can't hear audio from iOS Safari after using another app (ex. Siri, Youtube etc)

Webkit Bug: https://bugs.webkit.org/show_bug.cgi?id=208516 Status: A workaround for this issue is shipped in SDK version 2.6.0. The workaround is expected to work on iOS versions 13.4.X and 13.5.x, and based on our testing it solves the problem in most of the interruption use cases. We will continue adding iterative improvements in the future for some more rare/edgy use cases. Reproducibility This is 100% reproducible in particular scenarios when other applications like Siri or Youtube takes over the mic. Example: Join a video room, share audio/video -> Open Siri and talk to it -> come back to your video application -> now your mic will be sending silence to the room, remote party will not be able to hear you. Recommendation:

Update to the latest SDK version which contains a workaround 2.6.0.
Add an audio indicator UI in your app so that your users can see if they are sending silent audio and react to it. More here.

Follow this guide with the best practices from our dev team to avoid media issues across browsers in your applications:

Other important bugs in iOS Safari:

bjm88 commented 4 years ago

+1. We are eagerly awaiting a fix for this from Twilio or Safari. We are able to reproduce #1 remote audio not playing on iOS Safari iPhone setups a lot, like 50% of the time. This is after doing the suggested work arounds of pause/play and avoiding rerendering of

manjeshbhargav commented 4 years ago

@bjm88

We are in touch with Safari WebKit engineers and will keep you posted on any future updates from them regarding these issues.

Thanks,

Manjesh Malavalli JSDK Team

XuanyuHu commented 4 years ago

@manjeshbhargav Could you help to check for the iPadOS as well? We have clients reported that the iPad Safari is losing audio as well. Around 90% time, refreshing the Safari Browser will resolve the issue (audio will work).

bjm88 commented 4 years ago

thanks @manjeshbhargav - ya fyi We escalated with Apple and put in developer ticket with them, 734340602, if you need to reference.

We had both the "option #2" pause/play and the "avoid rerender by adding directly to dom" audio track attachment workarounds in and were still able to reproduce the issue about 30-50% fo the time. We actually toook out the pause/play and just left the attachment change and are seeing better results, seems like more like 1 out of 10, but still early in the testing.

manjeshbhargav commented 4 years ago

@XuanyuHu , @bjm88 ,

In our best practices guide, we recommend that you provide an audio level indicator for your local and remote audio tracks, so that the application can recommend users to reload the app or restart their browsers. Please take a look at the recommendations and apply them where appropriate.

Thanks,

Manjesh Malavalli JSDK Team

bjm88 commented 4 years ago

thx @manjeshbhargav we are planning to do that. We did notice the issue happens more on older iPhones 6/7 and less on X. Also our QA eng found she could reproduce a LOT on normal iPhone settings with Siri enabled, but when disabling Siri could not reproduce at all. Perhaps something to rely to apple.

anna-vasilko commented 4 years ago

@bjm88 Thank you so much for sharing your observations, I relayed it to the Safari team at Apple

elebumm commented 4 years ago

I have the same issue where the entire calls works apart from audio being sent from the iOS device to the participant (Chrome usually but anything really.)

I was able to create this issue on my iPhone 11 Pro Max doing the following:

Go to your web application on safari
Go back to the home screen (while having safari in the background)
Use an audio input of some sort. (I just said hi to Siri)
End that audio input
Go through the typical flow of how your application starts (accept permissions etc)
So far, 100% of the time it detects audio device but doesn't capture audio (or send)

I can fix this by exiting safari and opening again.

I feel like this happens when Safari is open the entire time. I had it happen when safari was in the background for at least 5 minutes. This could be depending on what "type" of person you are:

Someone who has all of their applications open in the background constantly (my dad)
Someone who exits as soon as their done (me).

I am on iOS 13. Let me know if anyone finds issue running into it this way as well.

I have temporarily fixed this by adding tests mentioned by @anna-vasilko before video and audio is enabled and gives instructions as to how to fix it.

Let me know if this was helpful for anyone.

chr4ss1 commented 4 years ago

I can produce audio issues in almost 99% of time in 1v1 direct rooms between ios devices, here is how:

1) I force everyone to go through mic / video check.

2) After they have gone through the mic check together, their browsers will automatically refresh by using location.reload(), (I thought I am clever: that would ensure there is no old garbage left behind, such as some streams floating around, basically start from completely clean sheet)

3) They now press Join room button and will be joined using Twilio.connect()

4) Neither party can hear each other. If they both refresh their browsers manually, they start hearing each other.

Basically this approach backfired. It looks like using location.reload() will almost guarantee that there is no audio anymore. Perhaps this method can be used by other people to get into the state of "there is no audio" very easily.

For now I have removed the window.location.reload() and added a button of "I can't hear the other person" which will show popup which tells both people to refresh their browser. I've also use only one single audio element throughout the app which is decoupled from react:


audioService.setRef(document.getElementById('audio-element'));

render(
  <App />,
  document.getElementById('root'),
);

hmheng commented 4 years ago

im trying to use the proposed workaround in #922 with pause() and play(). however, Mac Safari tell me it is NotAllowedError, similar to the issue : https://github.com/mediaelement/mediaelement/issues/2410

rajeyecarelive commented 4 years ago

Workaround doesn't works at all. iPad Pro and iPhone devices - none produced any audio at all.

Orbit-Akhil-Puri commented 4 years ago

This bug is majorly coming on devices with ios version < 13.0.

everbslab commented 4 years ago

Same thing with iOS/Safari up 13.4 - no remote audio available when participiant is on iPhone

elebumm commented 4 years ago

Have you guys tried seeing if my comment is what is causing this issue?

Since this bug is in the hands of Apple who have identified it as part of their issue, perhaps the best option is to create a little dialog box for users to restart safari and try to run the application again. I know it's not the best answer but I found that solution to work best and have reduced no audio signal connections by almost 95%.

Until Apple fixes this issue on the next iOS update, it might be the best solution for now. Sadly, since the fix to safari will require a software update, the problem might still persist throughout iOS 13.x

hmheng commented 4 years ago

@elebumm thanks for your response. Understand that bug could be coming from iOS Safari team for iOS devices. May I know if it is the same issue for Safari on Mac devices? Or i should raise separate ticket?

anna-vasilko commented 4 years ago

@hmheng Two audio issues described in this ticket are specific to mobile Safari. They are bugs in the mobile Safari browser itself. If you encounter an issue with desktop Safari participants (not involving mobile Safari at all), open a separate ticket. Before this, please make sure to go through this best practices guide to avoid common pitfalls - https://www.twilio.com/docs/video/build-js-video-application-recommendations-and-best-practices

anna-vasilko commented 4 years ago

All, quick update on state of these two issues: 1. Remote audio does not play occasionally in iOS Safari browser Status: Safari team at Apple is working on a fix, they let us know it should be testable within a few weeks Reproducibility of this issue is quite low

Group rooms - occurs in 0-2 rooms out of 50
Peer to peer rooms - occurs in 6-9 rooms out of 50
Note: The issue is less reproducible with iOS/Safari versions 13.4 and up

2. RemoteParticipants can't hear audio from iOS Safari after using another app (ex. Siri, Youtube etc)

Status: We found a workaround for this Safari bug and plan to ship it in the next release of our SDK. The workaround is expected to work on iOS Safari versions 13.4+. Reproducibility: This is 100% reproducible in particular scenarios when other applications like Siri or Youtube takes over the mic. Example: Join a video room, share audio/video -> Open Siri and talk to it -> come back to your video application -> now your mic will be sending silence to the room, remote party will not be able to hear you. Recommendation: We recommend that you add an audio indicator UI in your app so that your users case see when they are sending silent audio and react to it. More here.

Follow this guide with the best practices from our dev team to avoid media issues in your applications https://www.twilio.com/docs/video/build-js-video-application-recommendations-and-best-practices

romanalyoshkin commented 4 years ago

@anna-vasilko Hi Anna, we are looking at a possible solution for the video/audio calls and faced that issue with our existing solution. I'm curious how reliable is your workaround and when do you plan to make it public?

makarandp0 commented 4 years ago

Hello @romanalyoshkin - The safari audio workaround is in test, and we are hoping to release this week. If you are interested you can try it out, and let us know how it works out for your application.

Thanks, Makarand

bjm88 commented 4 years ago

hi @makarandp0 are you talking about work around for issue #2 above right, not #1 ?

makarandp0 commented 4 years ago

@bjm88 yes, the workaround is only for Webkit Bug: https://bugs.webkit.org/show_bug.cgi?id=208516

RusseII commented 4 years ago

Group rooms - occurs in 0-2 rooms out of 50

Is that for iOS/Safari versions 13.4 and up, or for the older versions? @anna-vasilko

everbslab commented 4 years ago

Please be aware that using twilio-video https://github.com/twilio/twilio-video.js/tree/2.6.0-rc1 requires update with https://raw.githubusercontent.com/twilio/twilio-webrtc.js/4.3.0-rc1/lib/util/index.js.

Otherwise there will be an error with

var _require = require('@twilio/webrtc/lib/util'), ... isWebRTCSupported = _require.support;

everbslab commented 4 years ago

@makarandp0 We just had a test with iPhone/Safari workaround version which comes from 2.6.0.-rc1. No luck. Still iPhone user unable to unmute himself using iPhone.

anna-vasilko commented 4 years ago

@everbslab Yes, the 2.6.0.-rc1 went through a testing cycle and we found that the workaround is not working on some of the iOS versions, the team is investigating.

anna-vasilko commented 4 years ago

Hi all, Here is an update on the state of things:

1. Remote audio does not play occasionally in iOS Safari browser Safari team from Apple communicated that it got fixed in the latest iOS beta version 13.5.5. Our internal testing confirmed the fix.

2. RemoteParticipants can't hear audio from iOS Safari after using another app (ex. Siri, Youtube etc) We continue working on a workaround in the SDK, the first RC did not pass testing. Second one is baking.

Note that both described issues are rarely reproducible. If you hit an audio problem all the time, make sure to go through this guide to avoid most common pitfalls

RusseII commented 4 years ago

@anna-vasilko are you able to share exactly what the apple team said or where they posted that they fixed the issue?

We have a client who is waiting for these issues to be fixed to move past the pilot period, and having that to share with them would be very useful.

anna-vasilko commented 4 years ago

@RusseII You can see public update from Apple on the WebKit ticket representing this particular issue https://bugs.webkit.org/show_bug.cgi?id=198545

hameshiv commented 4 years ago

@anna-vasilko I'm joining the thread because of a new and different issue.

I work for a company in the field of tele-audiology (remote hearing tests), using Twilio for video-conferencing on our web solution. We currently face an issue with our solution on iOS: during a conference call, our signal generated using webaudio is played in both channels as mono. Whenever we stop the video (using the red camera in url toolbar), the sound is played on the right channel, as intended. Back to the video, it's on both channels again.

Any help would greatly be appreciated. Thanks!

RusseII commented 4 years ago

RemoteParticipants can't hear audio from iOS Safari after using another app (ex. Siri, Youtube etc) We continue working on a workaround in the SDK, the first RC did not pass testing. Second one is baking.

@anna-vasilko is there any estimates on when this will be rolled out?

anna-vasilko commented 4 years ago

@hameshiv Thanks for writing in! The issue you describe sounds different and new to us. We will be happy to look at it, but could you pls open a separate github ticket for it.

anna-vasilko commented 4 years ago

@RusseII We are still working on that issue (#2 from the ticket description). It is on top of our priorities. So far the workaround in the SDK seemed to improve the behavior in some cases/ iOS versions, but stumbled upon another iOS Safari browser issue. We are investigating, talking to Apple team and doing everything to improve it. We are aiming for a release within a week if no more surprises. I will keep you posted.

ryango commented 4 years ago

I still see #1 on 13.6 beta. Fixes itself when I background Safari. Audio element reports playing. Reproduces very frequently when remote participant is Chrome on Mac. Pause/play seems to play one buffer then it gets back into the bad state.

ryango commented 4 years ago

Zooming the page also fixed it? Audio element emits no events when it fixes itself. Previous play() call reports success

RusseII commented 4 years ago

@ryango that's not good. :(

How commonly is "very frequently" ? Is it happening more commonly than the reported 0-2 rooms out of 50?

ryango commented 4 years ago

30-50% on iPhone SE 13.6. Seems to not repro in 14 though.

ryango commented 4 years ago

When it happens, it's always the second audio track added to the DOM. Using the JS quickstart code.

ryango commented 4 years ago

We do have a modification that gets local media at the same time as connecting - delaying recording start seems to help

RusseII commented 4 years ago

@anna-vasilko any more updates on the timeline? 13 days ago you said you were aiming for a release within the week. I'm assuming some unexpected things came up. Do you have an updated timeline?

Asking again because this is a problem for a significant percentage of our user base and we have a few pilots paused while we wait for a fix.

manjeshbhargav commented 4 years ago

@RusseII ,

We just released v2.6.0, which contains workarounds for a couple of these issues. Please try it out and see if it improves the user experience on iOS Safari.

Thanks,

Manjesh Malavalli JSDK Team

anna-vasilko commented 4 years ago

@RusseII Yes, as was mentioned above, the latest SDK release v2.6.0 contains the workaround for the issue #2 (audio can't be heard after interruptions by other apps like Siri, Youtube, Facetime etc). The workaround is expected to work on iOS versions 13.4.X and 13.5.x, and based on our testing it solves the problem in most of the interruption use cases. We will continue adding iterative improvements in the future for some more rare/edgy use cases.

XuanyuHu commented 4 years ago

@anna-vasilko , could you confirm if the iOS/iPadOS 13.6 (13.5.5) has all the fix for issue #1 Remote audio does not play occasionally in iOS Safari browser?

I believe Apple renamed 13.5.5 to 13.6 and just want to get an offical confirmation from Twilio team about the fix is deployed with iOS 13.6

XuanyuHu commented 4 years ago

@anna-vasilko , after upgrading the iOS and iPadOS to 13.6, with the latest SDK: https://github.com/twilio/twilio-video.js/releases/tag/2.7.0 We still see audio issues on iPhone and iPad devices. Could you confirm if the Safari fix is patched in the 13.6 iOS/iPad release?

anna-vasilko commented 4 years ago

@XuanyuHu Hi! Thanks for the info. Issue 1 (Remote audio does not play occasionally) - is fixed by Apple since iOS 13.5.5 and the fix seems to be included in iOS 13.6. In our testing we can't reproduce this with iOS 13.6.

Issue 2 (Can't hear audio from iOS Safari after using another app) - This is still reproducible in some of the interruption use cases with iOS version 13.6 and 14 Beta. Our workaround works well in 13.4 and 13.5 but not as well in those newer versions and we are actively working on improving there. It is hard to say either you are hitting this interruption issue or something different. Could you please share a room sid and reproduction steps so we can investigate further? Opening a separate ticket would be preferable.

XuanyuHu commented 4 years ago

Thanks, @anna-vasilko . We will keep testing on the 1st issue and open a support ticket if we have any issues.

webjay commented 4 years ago

Sorry if this is the wrong place to ask, but how can I detect if a user can't hear or see a peer?

manjeshbhargav commented 4 years ago

Hi everyone,

We have just released version 2.7.1 which has workarounds for several iOS Safari audio and video bugs. Our internal testing showed that most of the common iOS Safari issues have been worked around in this release. Please try it out and share any feedback you may have for us.

Thanks,

Manjesh Malavalli JSDK Team

TravBradfield commented 4 years ago

@manjeshbhargav

Thanks for this update! Do you still suggest implementing a "test mic" check before entering a call?

I'm still having the same issues when calling between mac and iPadOS Safari. The way to solve the issue seems to be simple minimise safari and then reopen it then the audio works fine.

Should I be doing the mic test before entering the call?

manjeshbhargav commented 4 years ago

@TravBradfield ,

Yes, I would recommend checking the microphone (and camera) before joining the Room. Our best practices guide shows how you can build an audio level indicator. Hope it helps.

Thanks,

Manjesh Malavalli JSDK Team

TravBradfield commented 4 years ago

@manjeshbhargav Thanks.

Do you have any recommendations for auto playing audio policy on mobile browsers? I'm able to get the microphone and videos fine. Just the iPad/iPhone is not playing the sound from the video... I'm making sure that the video is only joined after the user interacts with the screen Still no audio playing on the Safari mobile device.

twilio / twilio-video.js