Audio fails to start in Safari 14.0.2

[egaudrain]

I've been battling with Safari and audio problems for a while, but that seems to be a new issue.

I understand that modern browsers only allow audio to play after there has been some interaction with the user (e.g. a click on a button). In a new experiment, the participants go through the instructions by hitting the spacebar or clicking on a piece of text. Then come some trials using the audio-keyboard-response plugin. In Firefox and Chrome, the sound plays just fine (whether the user clicks or presses the spacebar), but in Safari, nothing happens, and it just hangs silently.

To unlock it, I had to add an async call-function trial that displays a button with a callback that calls jsPsych.pluginAPI.audioContext():

    const is_Safari = /Version\/.*Safari\//.test(navigator.userAgent) && !window.MSStream;
    if(is_Safari){
        timeline.push({
            type: 'call-function',
            async: true,
            func: function(done){

                var display_element = document.getElementById('jspsych-content');

                display_element.innerHTML = "<p>Click on the screen to start...</p>";
                //var init_button = display_element.querySelector('#safari_audio_init');

                function init_audio_files(){
                    jsPsych.pluginAPI.audioContext();
                    done();
                }

                document.addEventListener('touchstart', init_audio_files, false);
                document.addEventListener('click', init_audio_files, false);
            }
        });
    }

It is a pain... but it works... I tried to have just a button without the jsPsych.pluginAPI.audioContext() callback, and it did not work. This is the only combination that worked, and with or without webAudio made no difference.

Honestly, I don't think there is anything wrong with jsPsych. I don't know why Safari makes it extra hard and quirky to get audio started... I'm going to make a plugin to deal with this automatically, but if there's some more insight out there, that'd be great!

[egaudrain]

To clarify, I am guessing that the difference between Safari and Chrome is that Chrome accepts any user interaction since context initialisation to unlock the audio context, while Safari only accepts user interactions that are directly linked to the audio context? As a result, I haven't found a way to make a feature-based case discrimination... and am thus relying on ugly User Agent parsing...

Meanwhile, here's the plugin, and note that this only works with __use_webaudio=true__:

/**
 * jspsych-audio-safari-init
 * Etienne Gaudrain
 *
 * Safari is the new Internet Explorer and does everything differently from others
 * for better, and mostly for worse. Here is a plugin to display a screen for the user to click on
 * before starting the experiment to unlock the audio context, if we are dealing with Safari.
 *
 **/

jsPsych.plugins["audio-safari-init"] = (function() {

    var plugin = {};

    //jsPsych.pluginAPI.registerPreload('audio-safari-init', 'stimulus', 'audio');

    plugin.info = {
        name: 'audio-safari-init',
        description: '',
        parameters: {
            prompt: {
                type: jsPsych.plugins.parameterType.STRING,
                pretty_name: 'Prompt',
                default: "Click on the screen to start the experiment",
                description: 'The prompt asking the user to click on the screen.'
            }
        }
    }

    plugin.trial = function(display_element, trial) {

        // Ideally, we would want to be able to detect this on feature basis rather than using userAgents,
        // but Safari just doesn't count clicks not directly aimed at starting sounds, while other browsers do.
        const is_Safari = /Version\/.*Safari\//.test(navigator.userAgent) && !window.MSStream;
        if(is_Safari){
            display_element.innerHTML = trial.prompt;
            document.addEventListener('touchstart', init_audio);
            document.addEventListener('click', init_audio);
        } else {
            jsPsych.finishTrial();
        }

        function init_audio(){
            jsPsych.pluginAPI.audioContext();
            end_trial();
        }

        // function to end trial when it is time
        function end_trial() {

            document.removeEventListener('touchstart', init_audio);
            document.removeEventListener('click', init_audio);

            // kill any remaining setTimeout handlers
            jsPsych.pluginAPI.clearAllTimeouts();

            // kill keyboard listeners
            jsPsych.pluginAPI.cancelAllKeyboardResponses();

            // clear the display
            display_element.innerHTML = '';

            // move on to the next trial
            jsPsych.finishTrial();
        }

    };

    return plugin;
})();

[egaudrain]

For __use_webaudio=false__, one has to use the play/pause trick on one sound... To do that, I exposed the list of ids of preloaded sounds in jsPsych.js by adding, before return module:

module.preloaded_audio_IDs = function(){ return Object.keys(audio_buffers) };

Then in the plugin, the init_audio function looks like this:

function init_audio(){
    var context = jsPsych.pluginAPI.audioContext();
    if(context==null){
        jsPsych.pluginAPI.preloaded_audio_IDs().slice(0,1).forEach(function(a){
            var b = jsPsych.pluginAPI.getAudioBuffer(a);
            b.play();
            b.pause();
            b.currentTime = 0;
        });
    }
    end_trial();
}

This seems to cover all cases. Note that I think that it could be in general useful to expose some access to preload buffers.

[egaudrain]

I made a minimalistic case trial here: https://dbsplab.fun/test/jspsych/test_jspsych-audio-safari-init.html.

Doing so I think it is a bit clearer which cases this concerns, but I am still not 100% certain.

It looks like the issue arises only when there has been no click on elements of the DOM that were not generated by javascript. If I create HTML buttons on the page that trigger jsPsych.init, the issue does not occur. But if the first click occurs on elements that were javascript generated, they need to be connected to the audio in some form to unlock the playback...

[jodeleeuw]

Thanks so much for all of this info @egaudrain.

I'm certainly open to different ways of solving this. It sounds like a generic solution is going to be tough to generate right now, but perhaps we can add some of these approaches to the preload plugin that we're currently working on. Or maybe we should have an init-audio plugin like the one you've created and recommend that people use it when running audio experiments. Obviously if there is a way to modify how we are handling audio files that fixes this behind the scenes I'm all for that.

I don't have easy access to a mac right now so I can't do much testing of possible fixes.

[becky-gilbert]

Thanks very much @egaudrain! Do you happen to know whether this problem will replicate on an iPad (iOS / Safari 14). If so then I can help test.

If I understand correctly, it sounds like this isn't something that we can address with changes to the preload plugin or any of the audio-* plugins, because this is all about initiating the audio context via a user interaction, and the form of that interaction isn't something that we know about or have access to inside the preload or audio plugins.

I guess we could add a parameter to audio-* plugins for whether or not to do this interaction/initialization at the start of the trial. But that would mean using a different parameter value for the first vs subsequent audio trials. And it would require adding more parameters to audio plugins related to the interaction (text on the screen, button vs keyboard response, etc.), all of which replicate behavior that is already covered by other plugins.

Maybe another option is to have a universal plugin parameter that flags a particular trial as an 'audio interaction' trial. This way the init_audio function could be stored in jspsych.js and added as a click or keyboard event handler in the plugin. But this might be a little tricky and time-consuming to set up.

So my view is that your standalone init-audio plugin makes the most sense, and as Josh says, we could just recommend using it in audio experiments (at least when the researcher wants to allow Safari).

[egaudrain]

Thanks for your feedback, @jodeleeuw and @becky-gilbert! I had a look at the preload plugin. Is there a discussion item somewhere that outlines the rationale?

But I also agree with your analysis, @becky-gilbert. Integrating the Safari odd case would mean looking though the timeline, checking if there are clickable elements (if that's possible) before the first sound is played, locate the first one, and add a 'click' callback that starts an audio context. And if there isn't any clickable element, add a button at the beginning that activates the audio context.

Another workaround would be to always recommend triggering jsPsych.init() through the click of an HTML button (hard-coded, not JS generated). Then, I think it would always work... That feels a bit cumbersome, but on the other hand, it is perhaps not a bad practice in keeping with the rationale behind all these autoplay restrictions. This is now added to the test case.

I haven't found much documentation on all these yet. I don't know if it is a Safari oddity, or if it is in AppleWebKit. If so, likely Chrome will be hit by the same issue when it catches up in version numbers. To be safe, I had been looking for ways to detect it using feature-based detection (i.e. trying to create a context and see if it is created suspended or not), but the different between browsers seems to be in what user interaction they consider when allowing the audio context to resume... and I have no clue how to test that.

Regarding iPad, unfortunately I do not have direct access to one to try, so if you do, perhaps you can try the test case above. If the sound plays without the init-audio plugin (i.e. first button), then we're good.

[becky-gilbert]

I had a look at the preload plugin. Is there a discussion item somewhere that outlines the rationale?

Yep the preload plugin issue is #1351. The first comment in that issue thread references pull request #1234, which is also relevant. Please feel free to share your thoughts.

Regarding iPad, unfortunately I do not have direct access to one to try, so if you do, perhaps you can try the test case above.

Sorry, I missed your test link before. Thanks for putting this together! I tried on my iPad but couldn't get to the audio trial because it requires a space bar press, and I'm on touchscreen only. Just let me know if you have a chance to change from a keypress to button to start the audio trial, and I'd be happy to try testing again.

[egaudrain]

Ahah, of course! I modified it such that you can just wait 2s before the sound is played (I had put the spacebar trial just to check that pressing a key wasn't enough to count as user gesture).

Also I tried it with Safari 14.0.3, the behaviour is identical. And I tried it in an XCode simulated iPad with iOS 13 (which should be Safari 13) and it is the same as well (although I am not entirely sure to what extent the simulation is faithful).

Another side note, perhaps worth mentioning: the safari-init plugin is invisible because the event listener on document that is created in the trial function of the plugin catches the click on the button in the previous trial, I assume, as the event is bubbling up...? This is the same in Chrome, but it took me a while to understand, and I am still scratching my head as to how an event listener created after the event was triggered can still capture the event...

[charles2910]

Hi. Today I've experienced the same problem while testing an experiment (which uses the image-audio plugin from @becky-gilbert - thanks BTW) in different browsers. It works in firefox and chrome, but in Chromium (the free software version of chrome) it fails as @egaudrain experienced in safari. Actually it fails jsPsych initialization, I'll post the logs from Chromium below.

The AudioContext was not allowed to start. It must be resumed (or created) after a user gesture on the page. https://goo.gl/7K7WLu
(anonymous) @ jspsych.js:46
(anonymous) @ jspsych.js:1067

Note that I'm still using version 6.2 of jsPsych and line 46 corresponds to:

core.webaudio_context = (typeof window !== 'undefined' && typeof window.AudioContext !== 'undefined') ? new AudioContext() : null;

The link from the error points to autoplay policy changes from the browser and it contains some workarounds for the problem.

[becky-gilbert]

Aha, thanks @charles2910, really helpful to know that Chromium causes the same problem. Are you able to use @egaudrain's safari-init plugin as a temporary workaround? Or trigger jsPsych.init from a button click?

BTW I think this same console message:

The AudioContext was not allowed to start. It must be resumed (or created) after a user gesture on the page.

shows up a lot when audio is used in jsPsych experiments, but doesn't always cause the experiment to break.

And @egaudrain, I forgot to say thanks for updating your test link to work with touchscreens. I did try this on my iPad and am getting exactly the same behavior you described.

[charles2910]

Hi, I have some news. I tried the safari init plugin and it doesn't work. Now to the interesting part. I use Debian (GNU/Linux distribution) and the audio only works on firefox, today I tried on Chrome too and it doesn't work, the webaudio_context is initialized but suspended before any interaction with the window. After the interaction with the safari plugin or the welcome screen that I have, it changes to running, but the image-audio-response shows "Audio recording not possible.". The interesting is that on windows, this problem doesn't happen on Chrome, so I think it may be related with the OS and how the browser was compiled.

[egaudrain]

As far as I understand it, as @becky-gilbert noted, the AudioContext is always created suspended. On macOS, hat happens with Firefox, Chrome and Safari. However, it is only suspended until there is a user interaction. That's where Safari differed from other browsers: it looks like, in Safari, a click on a JS generated button does not resume the AudioContext, but other browsers do allow that. One workaround I've found that should work in all browser is to have a button hard-coded in your HTML, and call jsPsych.init() in the callback of that button... That seems to work on all browsers. Do you want to try this? Actually you can perhaps try the test-case I had prepared and see what works and what doesn't: https://dbsplab.fun/test/jspsych/test_jspsych-audio-safari-init.php.

And @egaudrain, I forgot to say thanks for updating your test link to work with touchscreens. I did try this on my iPad and am getting exactly the same behavior you described.

@becky-gilbert, thanks for trying on an iPad! At least there is consistent behaviour :)

I tracked down some people responsible for Safari on Twitter, but they didn't know anything specific about it. I need to prepare a pure JS test-case that's not jsPsych-dependent to submit to them. The online documentation is extremely vague on the matter... to the point that I wondered if there wasn't voluntary obfuscation since this is a feature that's supposed to prevent exploitation from unscrupulous advertisers... but maybe I'm being paranoid.

[charles2910]

Thanks for providing the tests @egaudrain. Feedback:

• All the options worked with Chrome and Chromium, I could hear the audio;
• Only when using "with HTML button" option the error is absent.

Note that the trial I'm using is to record audio after a image is displayed. This plugin is from @becky-gilbert.

[becky-gilbert]

Hi @charles2910, I think the "Audio recording not possible" error that you're getting with the image-audio-response plugin is separate to this issue, which is about getting audio to play and the browser's user interaction requirement. The audio-response plugins use the MediaRecorder API, which is not supported in Safari - I've noted this in the plugin docs here. I'm not sure if the MediaRecorder API is supported in Chromium.

Also, the "Audio recording not possible" message that you're seeing is related to the getUserMedia function, which tries to access the computer's mic. So in addition to the browser compatibility issues, there are a few other reasons why you might see this error, e.g. another program on the computer is already accessing the mic. This blog post has a list of reasons why getUserMedia can fail.

Apologies if I misunderstood the problem you're having. But if you think you're running into a separate problem with the image-audio-response plugin, then feel free to follow-up on this thread about audio recording: #494.

[charles2910]

Oh boy. @becky-gilbert you're right, they are two different issues. Turns out that firefox is a little dumber and let me record audio even without having a microphone, but Chrome(ium) does not.

[jamesalvarez]

The issue I faced was that jsPsych (use_webaudio: true) is calling the audio after a setTimeout (in TimeoutAPI) and not from the click. The solution I used is to play an empty sound file directly from a click early in the task e.g.:

            const soundEffect = new Audio();
            soundEffect.autoplay = true;
            soundEffect.src = "data:audio/mpeg;base64,SUQzBAAAAAABEVRYWFgAAAAtAAADY29tbWVudABCaWdTb3VuZEJhbmsuY29tIC8gTGFTb25vdGhlcXVlLm9yZwBURU5DAAAAHQAAA1N3aXRjaCBQbHVzIMKpIE5DSCBTb2Z0d2FyZQBUSVQyAAAABgAAAzIyMzUAVFNTRQAAAA8AAANMYXZmNTcuODMuMTAwAAAAAAAAAAAAAAD/80DEAAAAA0gAAAAATEFNRTMuMTAwVVVVVVVVVVVVVUxBTUUzLjEwMFVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVf/zQsRbAAADSAAAAABVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVf/zQMSkAAADSAAAAABVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVV";

[qbunt]

This is nothing to do with the framework, it is iOS (and other browsers) policy to block auto-playing media.

to respond to @egaudrain:

I don't know why Safari makes it extra hard and quirky to get audio started

History of the policy was because media autoplay was abused by advertisers, and does burn bandwidth on behalf of users before they can stop it. Documentation on the blockage is not great, but it is clear enough to understand the rules.

[egaudrain]

Thanks @qbunt, after some research, we had indeed figured out that much. The issue boiled down to the fact that there is no clear documented definition of what Safari considers a "user gesture" and some cases were defying our intuition.

But this issue is a bit old, I haven't had much time to look at it more in details (and jsPsych switching to TypeScript has spooked me a bit), and everything could be different by now. At the time, it seemed that, sometimes, clicks on JS generated buttons were not good enough for Safari to unlock playback (see above). It worked to use an HTML button as a workaround for cases where we wanted the test to be ran on iOS or iPad, despite the fact that the HTML button is there before the AudioContext is created... so definitely some unexpected behaviour. If you get to the bottom of this, it would be great to have some workarounds documented somewhere.

[qbunt]

@egaudrain all good, 2 years is a long time. Definitely no easy answers. We haven't seen JS or static html make any difference, but if buttons are added/removed from the tree, that will trip Safari up.

The thing that PsychJS could do is require an element first for interaction for media with audio (only on iOS, super problematic) OR use all three required attributes to play as expected without interaction if it can play without full screening the video (not full screening the UI)

• autoplay
• muted
• playsinline

Obviously can't work for anything with audio, so that would have to force the interaction first. Simplest way to handle would be an overlay on the video element but it's real intrusive in a clinical setting.