Autoplay behavior needs improvement

[n0samu]

Issue The behavior of web builds of Ruffle with "autoplay": "on" is less than ideal in several ways:

• An unmute overlay is always shown, even if the SWF doesn't play any audio before the first user interaction. Here is an example.
• If animation with synced audio starts playing before the audio is unmuted, the animation will pause completely, rather than playing without audio. Here is an example.
• When the audio is unmuted, all of the sound effects that were supposed to play while the audio was muted will play at once. Notice what happens if you let Alien Hominid play through the intro, then click the unmute overlay. Here is the link.

Improvement

• With "autoplay": "on", an unmute overlay should only appear if the SWF tries to play audio before the first user interaction.
• SWFs should play as normal even if muted; the only difference should be that the user hears no audio.
• When the audio is unmuted, only the sound that's supposed to be currently playing should play.

[ActionWavele]

I thought browsers break autoplay?

[Herschel]

The animation stops because the audio syncing code is trying to sync the animation to the non-playing sounds. The easiest fix is for the web audio backend to simply bail out of playing any sounds if it the audio context isn't playing. This would fix the all-sounds-playing-at-once issue on resume, too. The one downside is that any background music won't start playing once resumed.

[n0samu]

The one downside is that any background music won't start playing once resumed.

I feel like that's a pretty significant downside because, for example, when you unmute an autoplaying HTML5 video it wouldn't be missing its background music. Users and webmasters wouldn't understand why this should be any different. But still, it would be an improvement from how things are now!

I wonder if there's any way to play audio into some kind of "null backend" if the audio context isn't available? And switch to the web audio backend once the audio context is available? Sorry if that doesn't make sense.

[n0samu]

I just noticed that part of this issue was already described in #6800.

[n0samu]

The animation stops because the audio syncing code is trying to sync the animation to the non-playing sounds. The easiest fix is for the web audio backend to simply bail out of playing any sounds if it the audio context isn't playing. This would fix the all-sounds-playing-at-once issue on resume, too. The one downside is that any background music won't start playing once resumed.

Actually I now think this is a good suggestion. To address the background music issue, I think we should check if the sound has looping enabled, and if so, we should store a reference to it for later when bailing. And then when the audio context resumes, we can call a method on the web audio backend to tell it to check if those looping sounds would still be playing, and play them now if so.

Edit: That last part may not even be necessary, we might be able to get away with just pretending nothing is wrong and playing the sound as usual if it's a looping sound, as we're currently doing for all sounds

[n0samu]

I tried this approach and it was very easy to implement, but it didn't work for games like Home Sheep Home or Volt Connect 2 because they expect to receive events from the sounds after they play. For example, Volt Connect 2 plays its music in non-looping segments, and adds a completion listener to each segment that makes the next segment start. If we don't add the sound, events don't fire. Both games end up never playing any music.

After discussion with @torokati44 on Discord, it seems like the proper approach would be to keep pumping our audio buffers even if the browser isn't playing any sound from them. That would mean adding an interval in JS that periodically calls play() on our buffer, just like the onended handler does. But this means we're passing the buffer in 2 separate places, and we can't just clone the buffer because pumping the clone won't affect the original. So @torokati44 suggested that we probably need to use an Arc to manage access to the buffer. This feels quite a bit beyond my skill level, and it seems like no one else is going to pick up this issue :(