search

Hubs-Foundation / hubs

Duck-themed multi-user virtual spaces in WebVR. Built with A-Frame.
https://hubsfoundation.org
Mozilla Public License 2.0
2.13k stars 1.41k forks source link

[Safari] The audio is broken on certain MacOS/Safari versions #4411

Closed Softvision-RemusDranca closed 2 years ago

Softvision-RemusDranca commented 3 years ago

[Notes]:

[Affected Versions]:

[Affected Platforms]:

[Prerequisites]:

[Steps to reproduce]:

  1. Spawn a video inside the room.
  2. Play the video for a small period of time.
  3. Observe the behavior.

[Expected results]:

[Actual results]:

[Additional Notes]:

┆Issue is synchronized with this Jira Task

Softvision-RemusDranca commented 3 years ago

I've encountered an additional behavior while testing. If for example a YouTube video is playing in a different tab, when focusing the Hubs tab the video's audio from the Youtube tab is broken.

Softvision-RemusDranca commented 3 years ago
takahirox commented 3 years ago

On my MacBook Pro, macOS Big Sur 11.4, Safari Version 14.1.1, I can reproduce the audio issue.

But I can't reproduce the performance issue. According to the console logs you shared, the shader compile fails and I speculate it's the cause. But I don't see shader compile error on my end. Could it be scene or object specific issue? Cab you create a scene and pin objects where you encounter the compile error and the performance issue?

brianpeiris commented 3 years ago

I'd suggest we open a new issue for the performance problem.

brianpeiris commented 3 years ago

I've reproduced the audio distortion issue consistently in Safari 14.1.1 on macOS (11.4 MacBook Air M1 2020) and iOS (14.6 iPhone 11). The issue reproduces at different rates in different environments, but it does happen eventually in all cases. The issue does not occur on Chrome or Firefox on macOS. I've reproduced it with a simplified test case, using only WebAudio. https://brianpeiris.github.io/spatial-audio-test/ The key factor is a PannerNode that updates its transform continuously. I've also posted my details in an existing webkit bug report: https://bugs.webkit.org/show_bug.cgi?id=227199 I think our only option for now is to disable spatial audio in Safari until this bug is fixed in the browser. We might emulate attenuated audio instead.

takahirox commented 3 years ago

Nice investigation!

I think our only option for now is to disable spatial audio in Safari until this bug is fixed in the browser. We might emulate attenuated audio instead.

I agree with it. Disabling spatial audio should be much better than broken audio.

Regarding the performance issue, I confirmed that CPU usage on Mac is very high even on a simple Three.js positional audio example so I speculate the performance problem may be very relevant to the audio issue as @Softvision-RemusDranca mentioned.

@brianpeiris Did you already start to work on the workaround? If not and you don't have time, I can take it. I hope we add the workaround soon and test to check if the performance issue can also be resolved.

brianpeiris commented 3 years ago

4442 Fixes avatar audio issues for the next Hubs Cloud release, but I'll have to deal with the current version of the code separately for HMC, since the new audio code is quite different.

takahirox commented 3 years ago

I would like to try to apply the same change to master, just in case after an audio related PR #4487 will be merged, unless @brianpeiris wants to work on it because hubs.mozilla.com is very useless on MacOS/iOS Safari and very hard to even test on them now.

brianpeiris commented 3 years ago

Go ahead @takahirox I'm going to continue focusing on Hubs Cloud.

takahirox commented 3 years ago

OK, made a PR #4503

takahirox commented 3 years ago

Thanks to #4594, the distorted audio on MacOS Safari seems to be resolved by sacrificing PositionalAudio.

The iOS Safari one seems much better than before, but I realized it stills has some audio distortion issues. After being a room for like 5-10mins, the audios are distorted and broken.

How to reproduce.

  1. Enter a room on MacOS Safari and iOS Safari
  2. Speak and move around like for 5-10 mins (It seems the moving around is necessary to cause the problem, or it makes the problem happens earlier)
  3. Confirm on iOS Safari that audios are distorted, broken, and laggy.
  4. Confirm on MacOS Safari that audio "from the iOS Safari" is distorted and broken. After this problem happened, I entered the same room on MacOS Firefox and the audio from/to MacOS Firefox is fine on MacOS Safari. So the problem seems to be on iOS safari end.

@Softvision-RemusDranca reported the same problem.

Also, I can reproduce the problem with just a video. I spawned a video on iOS Safari, move around for like 5-10 mins, and then the audio started to be distorted and broken.

I couldn't reproduce the problem on MacOS Safari so far.

When the problem happens, my iPhone is very hot. I have a impression that the problem happens when CPU is busy (or it can be Audio buffer problem). So potentially the problem could be related to #4423.

takahirox commented 3 years ago

Regarding the Positional (Panner) audio bug fixed in #4594, we thought it was a Safari specific bug but it seems that I can reproduce the problem even on iOS Chrome/Firefox. (I speculate iOS Safari/Chrome/Firefox use the same core webkit library?)

Currently we disable Positional audio for Safari but we may need to disable it for Safari or iOS.

Can anyone here check if you can reproduce the problem on iOS Chrome/Firefox on your end, too?

johnshaughnessy commented 3 years ago

@takahirox If we can't diagnose and fix the root cause, I wonder if there's a workaround we can do where we disconnect from dialog, tear down all the voice connections, and then reconnect/recreate them every 5 minutes or so.

takahirox commented 3 years ago

@johnshaughnessy It might be worth to try, although I'm not really sure if it can resolve the problem. (And I'm really not sure yet if we have a workaround)

takahirox commented 3 years ago

I made a simple test case.

https://takahirox.github.io/ios-webaudio-test/

It plays music with or without WebAudio.

On my iOS 14.8, the audio with WebAudio is broken after about two minutes regardless of browsers (Safari/Chrome/Firefox) while the one without WebAudio isn't broken.

I'm going to report this problem to webkit bugzilla. (I guess all the major iOS browsers use the same webkit engine for WebAudio.)

WebAudio on iOS doesn't seem to be stable. I googled and found that WebAudio on iOS has repeatedly had bugs, fixed them, and had regression bugs again.

I'm thinking of avoiding to use WebAudio until iOS WebAudio will be matured enough.

A workaround in my mind is to add a new (Three.js) audio class which plays audio just with audioElement.play(), doesn't use WebAudio API, and is compatible with Three.js Audio API. It doesn't have panner, but has audio attenuation by controlling audioElement.volume.

takahirox commented 3 years ago

TLDR: I would like to suggest to force to disable echoChancellation for iOS to reduce the crackle audio noise.

MacOS BigSur 11.6 and iOS 15 have been released and I tested some audio on them.

I first thought we had a single root issue causing the broken audio problem even without panner node, but It seems we had two (or more) audio issues in iOS 14. One of them seems to have been resolved in iOS 15. But another one seems to still exist.

  1. Distorted and broken audio problem with WebAudio createMediaElementSource()

This is the problem which seems to have been resolved in iOS 15.

WebAudio createMediaElementSource() seemed to cause audio noise in iOS 14. But it seems to have been resolved iOS 15. I confirmed it with simple test case https://takahirox.github.io/ios-webaudio-test/

  1. Crackled audio with echoCancellation

With fixed createMediaElementSource(), the audio in iOS 15 sounds better. But the audio sounds still crackled. It seems another issue still exists in iOS 15.

The root issue seems echoCancellation of getUserMedia(). Somehow user media (mic) seems to cause a audio noise.

(Finally!) I made a simple test case reproducing the crackle noise problem on iOS

https://takahirox.github.io/ios-webaudio-test/video_user_media.html

If you click "run" button on your iOS, you would hear crackled audio from the video.

A workaround seems to be disabling echoCancellation when creating user media for mic. If you uncheck echoCancellation in the test, you would notice that the crackle noise is reduced. (Somehow the noise is not perfectly removed on my iOS, but much better.)

I also tried to disable echoCancellation in the preference on Hubs and confirmed that the audio becomes much better.

So, I would like to suggest to force to disable echoCancellation on iOS.

  1. getUserMedia() can make the audio quieter

While doing the echoCancellation test, I noticed that if you call getUserMedia() the audio from the video can be quieter.

You can test it in the test by unchecking "Use getUserMedia() for mic".

I haven't come up with a solution yet. We might be able to try other getUserMedia() options.

  1. Panner node still seems problematic in iOS 15

I also checked if we still need disabling panner node workaround for Safari. It seems yes, we still need it. I tested panner node on Safari. But the audio is broken after being in a room after some minutes. It sounds even worse then before. The audio problem can be reproducible even in the Three.js audio orientation example.

https://threejs.org/examples/#webaudio_orientation

takahirox commented 3 years ago

I discussed the crackle audio noise problem and we found that it might be old iPhone specific problem.

We don't have many enough iOS devices, so iOS users, can I ask for volunteers?

I'm happy if you run the test to check whether you can reproduce the problem I encountered, and report to me here whether you can reproduce or not.

  1. Go to https://takahirox.github.io/ios-webaudio-test/video_user_media.html on your iOS device with Safari (and other browsers)
  2. Every item is checked and press run button
  3. You hear crackle noise from video audio
  4. Reload the page, uncheck "echoCancellation", and press run button
  5. You hear less noise
jure commented 3 years ago

Data point for your latest test @takahirox.

On iOS 15 on an iPhone XS Max I hear no crackling with echoCancellation on or off - it does not make a difference.

takahirox commented 3 years ago

@jure Thanks for the test. Your model is newer and more powerful than mine (iPhoneX) so I guess you didn't see the issue.

Softvision-RemusDranca commented 3 years ago

@takahirox I have the same result as @jure , no crackling noise with or without echoCancellation. I used Iphone SE2 with Safari 15.

takahirox commented 3 years ago

Thanks all.

I also asked for the test at different places and we have tested five iPhone devices in total.

The models on which the audio noise problem is reproducible

The models on which the audio noise problem isn't reproducible

Sounds like the problem is reproducible only on old (2017 or before?) models (maybe meaning less powerful models). We couldn't find global iPhone model market share numbers, but I guess iPhone users generally use newer models?

So I don't think we should force to disable echo cancellation. (#4679)

We can share the tips with users instead

  1. we suggest to upgrade iOS to iOS 15
  2. if you still hear audio noise, we suggest to disable echo cancellation in the preference
takahirox commented 3 years ago

@Softvision-RemusDranca

Sounds like the broken audio issue shouldn't happen in iPhone SE 2 + iOS 15 and with the patches we recently applied. Do you mind if testing again to check if the problem is still reproducible?

vincentfretin commented 3 years ago

No crackling with echoCancellation on or off with iPad 2018 6th gen iOS 14.8.

Softvision-RemusDranca commented 3 years ago

@takahirox Unfortunately, I was able to reproduce the issue using Iphone SE2 after approximately ten minutes but it was only for a very short moment, after that the audio was working again. The issue reproduced while I was playing a video in the background and also having a conversation between two users.

takahirox commented 3 years ago

@vincentfretin

Thanks for testing. The problem doesn't seem to be occurred on iPad.

@Softvision-RemusDranca

It's good to know that at least it sounds better than before. But you couldn't reproduce the audio noise problem on https://takahirox.github.io/ios-webaudio-test/video_user_media.html but can still reproduce the noise problem on Hubs. Hm, there might be another issue or high resource usage may affect the audio quality.

I think it's hard to further investigate. I will report the problem to Apple devs so far.

takahirox commented 3 years ago

Reported to https://bugs.webkit.org/show_bug.cgi?id=227199 Please post there if you have additional context.

brianpeiris commented 2 years ago

Just to add more data points, I tested this today with my iPhone 11, running iOS 15.4 and Safari 15.4. I used two bots with movement and audio, and ran the test for 30 minutes. Audio did not distort or crackle. I guess this is primarily due to #4594. The Safari team may have also fixed some underlying issues, but we'd have to test with positional audio re-enabled to verify that.

I'd suggest we can close this bug, since it is not an issue for current models and the latest iOS/Safari. If it is still an issue on older iPhone models, and we care about fixing that issue, we can open a new ticket with up-to-date, summarized information about the issue.

pigubaoza commented 2 years ago

hi i would like to check if positional audio is re-enabled? it looks like it's still disabled here https://github.com/mozilla/hubs/blob/fe4eb6ffbc94fb411f8d98754376aed7e0c2b01a/src/systems/sound-effects-system.js#L141 I have also filed an issue that I think might be related: https://github.com/mozilla/hubs/issues/5458

takahirox commented 2 years ago

Thanks for the report.

OK, just in case let's test again and if the problem is not reproducible we can re-enable it.

takahirox commented 2 years ago

Made a PR #5460

takahirox commented 2 years ago

5460 has been merged. Closing this issue. Please file a new issue if anyone encounters broken audio problem on Safari again.

Audio/Video noise problem seems to still exist. Let's track that issue in #4322