`stereo.play` hangs in Safari

[jagthedrummer]

I have a custom action in my component that's calling stereo.play, and that call seem to hang indefinitely.

The relevant bits of my component look like this:

import Component from '@glimmer/component';
import { inject as service } from '@ember/service';
import { action } from '@ember/object';
import { tracked } from '@glimmer/tracking';
import { task } from 'ember-concurrency';

export default class BouncePlayerComponent extends Component {
  @service stereo;
  @service store;

  togglePlaySoundTask = task(async () => {
    console.log('togglePlaySoundTask');

    let bounceDownloadRequest = this.store.createRecord(
      'bounce-download-request',
      { bounce: this.args.bounce }
    );

    console.log('saving');
    await bounceDownloadRequest.save();
    console.log('saved');

    let identifier = bounceDownloadRequest.downloadUrl;

    console.log('playing');
    let { sound } = await this.stereo.play(identifier).catch(error => {
      console.log('caught an error trying to play a sound', error);
    });
    console.log('played');
  });
}

In Safari when togglePlaySoundTask is performed the console shows:

togglePlaySoundTask
saving
saved
playing

And it just never progresses from there, and it never throws an error.

But mobile Safari does something different than on the desktop.

On mobile if I click the "play" button again it doesn't start the task over from the top, but instead that seems to make the already running task resume execution somehow and then I almost instantly see 'played' show up in the console and then the sound begins playing. Tapping anywhere else on the screen also makes it progress past whatever is hung just the same as tapping the play button a second time.

On the desktop clicking anywhere else on the app does not trigger the task to continue and it just remains hung. And clicking the play button again makes the togglePlaySoundTask start over from the beginning and then when it gets to stereo.play it hangs again, and also triggers the catch with this error:

TaskCancelation: TaskInstance 'playTask' was canceled because it belongs to a 'restartable' Task that was .perform()ed again. For more information, see: http://ember-concurrency.com/docs/task-cancelation-help

If I remove the catch then no errors are reported.

[jkeen]

Sounds like an safari-autoplay blocking issue, since the tap anywhere on the screen clears it up. Safari is particularly finicky when it comes to what it thinks is autoplay, and generally the play call and the click need to happen as close as possible together. So instead of waiting for your handler to return the identifier which you then give to play, you might want to try passing a promise into that play handler that fetches and returns the url.

Timing is crucial for the autoplay situation, and ember-stereo can handle promises that resolve to identifiers for that reason.

Give that a shot and if it doesn't work we can dig in further

[jagthedrummer]

Thanks for the tip! I'll give that a try and report back.

[jagthedrummer]

Thanks for the pointer @jkeen, I think we're on the right track.

Wrapping the URL retrieval code in a promise and handing that to stereo.play did the trick for mobile Safari.

But on the desktop it still does the same thing where it just hangs, and I can't find any way to make it play via user interaction (like by clicking the play button multiple times, or trying to play several clips one after the other). It just completely refuses to play.

In case it'll help anyone later, here's what the handler looks like using a Promise:

import Component from '@glimmer/component';
import { inject as service } from '@ember/service';
import { action } from '@ember/object';
import { tracked } from '@glimmer/tracking';
import { task } from 'ember-concurrency';

export default class BouncePlayerComponent extends Component {
  @service stereo;
  @service store;

  togglePlaySoundTask = task(async () => {
    let bouncePromise = new Promise((resolve, reject) => {
      (async () => {
        let bounceDownloadRequest = this.store.createRecord(
          'bounce-download-request',
          { bounce: this.args.bounce }
        );
        await bounceDownloadRequest.save();
        let identifier = bounceDownloadRequest.downloadUrl;
        resolve(identifier);
      })()
    });

    let { sound } = await this.stereo.play(bouncePromise);
  });
}

[jkeen]

On desktop is anything revealed if you enable debugging and sift through the log messages? It'd be useful to know which connection is handling that request on safari desktop.

In the console: require('debug').enable('ember-stereo*')

Also, are you on the latest 4.x version or the 5.x beta?

[jagthedrummer]

I'm currently on version 4.2.0.

Here's what I'm seeing in the console after I enable debug messages:

[Log] togglePlaySoundTask – "bounce_152"
[Log] saving
[Log] playing
[Debug] ember-stereo:shared-audio-access creating new audio element +0ms
[Log] saved
[Log] elapsed milliseconds =  – 140
[Log] resolving – "https://seshy-dev.s3.amazonaws.com/..."
"https://seshy-dev.s3.amazonaws.com/..."
[Debug] ember-stereo given urls: https://seshy-dev.s3.amazonaws.com/... +0ms
[Debug] ember-stereo:sound-cache cache miss for +0ms – ["https://seshy-dev.s3.amazonaws.com/..."] (1)
[Debug] ember-stereo TRYING: [Native Audio] -> https://seshy-dev.s3.amazonaws.com/... +0ms
[Debug] ember-stereo:Native Audio (anothercolor: inheritmol.m4a) Handling 'emptied' event from audio element%6c +0ms
[Debug] ember-stereo:Native Audio (anothercolor: inheritmol.m4a) audio-paused%6c +0ms
[Debug] ember-stereo:Native Audio (anothercolor: inheritmol.m4a) Handling 'loadstart' event from audio element%6c +0ms
[Debug] ember-stereo audio-paused +0ms – {sound: NativeAudio}
{sound: NativeAudio}Object
[Debug] ember-stereo:Native Audio (anothercolor: inheritmol.m4a) Handling 'durationchange' event from audio element%6c +0ms
[Debug] ember-stereo:Native Audio (anothercolor: inheritmol.m4a) Handling 'loadedmetadata' event from audio element%6c +0ms
[Debug] ember-stereo audio-duration-changed +0ms – {sound: NativeAudio, duration: 256069.65986394556}
{sound: NativeAudio, duration: 256069.65986394556}Object
[Debug] ember-stereo:Native Audio (anothercolor: inheritmol.m4a) Handling 'loadeddata' event from audio element%6c +0ms
[Debug] ember-stereo:Native Audio (anothercolor: inheritmol.m4a) triggering audio ready%6c +0ms
[Debug] ember-stereo:Native Audio (anothercolor: inheritmol.m4a) audio-ready%6c +0ms
[Debug] ember-stereo:Native Audio (anothercolor: inheritmol.m4a) audio-loaded%6c +0ms
[Debug] ember-stereo SUCCESS: [Native Audio] -> (https://seshy-dev.s3.amazonaws.com/... +0ms
[Debug] ember-stereo firing sound-ready for https://seshy-dev.s3.amazonaws.com/... +0ms
[Debug] ember-stereo:sound-cache cache miss for +0ms – ["https://seshy-dev.s3.amazonaws.com/..."] (1)
[Debug] ember-stereo:sound-cache caching sound with url: https://seshy-dev.s3.amazonaws.com/project-74/bounces/152/anothercolor: #CCCC00mol.m4a%6c +0ms
[Debug] ember-stereo:Native Audio (anothercolor: inheritmol.m4a) telling audio to play%6c +0ms
[Debug] ember-stereo:Native Audio (anothercolor: inheritmol.m4a) #stop%6c +0ms
[Debug] ember-stereo:Native Audio (anothercolor: inheritmol.m4a) Handling 'emptied' event from audio element%6c +0ms
[Debug] ember-stereo:Native Audio (anothercolor: inheritmol.m4a) audio-paused%6c +0ms
[Debug] ember-stereo audio-loaded +0ms – {sound: NativeAudio}
{sound: NativeAudio}Object
[Debug] ember-stereo audio-blocked +0ms
Object

error: "The request is not allowed by the user agent or the platform in the current context, possibly because the user denied permission."

event: NotAllowedError: The request is not allowed by the user agent or the platform in the current context, possibly because the user denied permission.

sound: NativeAudio {eventManager: Class, connectionKey: "NativeAudio", options: {}, timeout: 30000, retryCount: 0, …}

Object Prototype
[Debug] ember-stereo audio-paused +0ms – {sound: NativeAudio}
{sound: NativeAudio}Object
[Debug] ember-stereo Looks like the mobile browser blocked an autoplay trying to play sound with url: https://seshy-dev.s3.amazonaws.com/... +0ms
[Debug] ember-stereo audio-blocked +0ms
Object

sound: NativeAudio {eventManager: Class, connectionKey: "NativeAudio", options: {}, timeout: 30000, retryCount: 0, …}

Object Prototype

[jkeen]

And that's on safari desktop for mac? What version of safari?

[jagthedrummer]

My Safari version is: Version 16.6 (18615.3.12.11.2)

Also just wanted to document some things that I've been trying in attempts to debug this.

I tried using a "preloaded" URL via an aciton, which works.

  @action
  async playPreloadedSound(){
    let identifier = "https://...";
    let { sound } = await this.stereo.play(identifier);
  }

I tried using a preloaded URL which is passed to Ember Stereo via a promise, which also works:

  @action
  async playPreloadedSoundViaPromise(){
    let bouncePromise = new Promise((resolve, reject) => {
      (async () => {
        let identifier = "https://...";
        resolve(identifier);
      })()
    });
    let { sound } = await this.stereo.play(bouncePromise);
  }

Then I tried introducing a delay in that promise and if the delay is long enough then it stops working:

  @action
  async playPreloadedSoundViaPromiseWithDelay(){
    let bouncePromise = new Promise((resolve, reject) => {
      (async () => {
        let identifier = "https://...";
        await timeout(1000);
        resolve(identifier);
      })()
    });
    let { sound } = await this.stereo.play(bouncePromise);
  }

If the timeout is 1 second or more it always shows the blocked message. If it's set extremely close to 1 second, at like 995 it sometimes works, and sometimes doesn't. And if it's sufficiently less than 1 second, like 980 and below, it always works.

I've done a very rough benchmark on my real code which is retrieving the URL over the network and that usually runs in 200ms or less, so I don't think it's strictly a timing issue. It's almost like Safari is measuring the amount of work done between the click and the attempt to play the sound.

I've also tried all of these examples as an @action and as a task and that change doesn't seem to make any difference for any of them.

[jagthedrummer]

I went digging into Safari settings and found an option that I can change which will get things working. (Though it would be less than ideal to have to ask every desktop Safari user to make the same settings change.)

In the Safari settings I saw that my site was kind of "auto-blocked for auto-play" on this screen:

If I change "Stop Media with Sound" to "Allow All Auto-Play" then I can play clips from my app.

I never set that to "Stop Media with Sound" in the first place, so I'm guessing that Safari automatically adds any sites that try to play sounds to that list.

[jkeen]

Excellent context, thanks for that!

And yeah, Safari definitely does some extra calculations to detect any funny business with regard to autoplay that isn't strictly timing related. Good on them for fighting annoying autoplay video ads, but in legit cases like this it can be a little frustrating.

I know Howler does some clever stuff to get around these things, for a quick fix, you might try using the Howler connection instead of NativeAudio? For the 5.x version I was exploring removing NativeAudio entirely, so it might be worth a shot.

You can either do that in your env setup by removing the NativeAudio reference in the list of defaults, or by passing a useConnections argument to play or load

let { sound } = await this.stereo.play(identifier, { useConnections: ['Howler'] );

The setting you found just tells the browser to allow autoplay, which will remove all these checks you're running into. But it's up to the end-user to enable that, which in cases like this I find to be kind of a non-starter.

There are some helpers in ember-stereo that can help you combat autoplay issues by displaying dialogs and things if autoplay is enabled, or if it detects that a sound was blocked via browser autoplay prevention measures. That might be worth looking into? https://ember-stereo.com/docs/autoplay

[jagthedrummer]

Unfortunately forcing it to use Howler doesn't seem to make a difference. Here's the debug log when I do that:

[Log] togglePlaySoundTask – "bounce_148"
[Log] playing
[Debug] ember-stereo:shared-audio-access creating new audio element +0ms
[Log] elapsed milliseconds =  – 175
[Debug] ember-stereo given urls: https://seshy-dev.s3.amazonaws.com/... +0ms
[Debug] ember-stereo:sound-cache cache miss for +0ms – ["https://seshy-dev.s3.amazonaws.com/project-74/bounces/148/smol42.mp3"] (1)
[Debug] ember-stereo TRYING: [Howler] -> https://seshy-dev.s3.amazonaws.com/... +0ms
[Warning] HTML5 Audio pool exhausted, returning potentially locked audio object.
[Debug] ember-stereo:Howler (smol42.mp3) audio-loaded +0ms
[Debug] ember-stereo:Howler (smol42.mp3) audio-ready +0ms
[Debug] ember-stereo SUCCESS: [Howler] -> (https://seshy-dev.s3.amazonaws.com/...) +0ms
[Debug] ember-stereo firing sound-ready for https://seshy-dev.s3.amazonaws.com/... +0ms
[Debug] ember-stereo:sound-cache cache miss for +0ms – ["https://seshy-dev.s3.amazonaws.com/project-74/bounces/148/smol42.mp3"] (1)
[Debug] ember-stereo:sound-cache caching sound with url: https://seshy-dev.s3.amazonaws.com/project-74/bounces/148/smol42.mp3 +0ms
[Debug] ember-stereo:Howler (smol42.mp3) #play +0ms
[Debug] ember-stereo audio-loaded +0ms – {sound: Howler}
{sound: Howler}Object
[Debug] ember-stereo Looks like the mobile browser blocked an autoplay trying to play sound with url: https://seshy-dev.s3.amazonaws.com/... +0ms
[Debug] ember-stereo audio-blocked +0ms
Object

sound: Howler {eventManager: Class, connectionKey: "Howler", options: {}, timeout: 30000, howl: Object, …}

Object Prototype

[jkeen]

Dang. What about the 5.x release? Any change there?

It's strange that the promise strategy works with mobile safari but not on desktop, from what I recall mobile was always more of a pain with autoplay than desktop.

Having the url already before initiating the click really does seem like the way to go to avoid hassles. Not sure how your app is structured but is it possible to load that url before the user clicks the button? Or maybe giving the play function a server url that redirects to your media url? I seem to recall running into a case where that was necessary and it worked for me… but that may have been HLS-specific. Might be worth a shot?

[jagthedrummer]

No change that I can see with 5.0.0-beta.11.

It was also surprising to me that the promise method fixed mobile but not the desktop.

I'm not sure it's possible for me to get the URL ahead of time. I have to get a signed URL from S3 and will have lots of play button on screen all at once which would be lots of signed URLs to generate, most of which will never be used.

I'll see if I can make some kind of redirect shenanigans work.

[jagthedrummer]

@jkeen wanted to also note that passing useConnections: ['Howler'] to play seems to break the promise method in mobile Safari.

When I pass that option in it does the same thing where it hangs at first, and I see this in the console:

[Log] togglePlaySoundTask – "bounce_152"
[Log] playing
[Log] elapsed milliseconds =  – 181
[Warning] HTML5 Audio pool exhausted, returning potentially locked audio object.

And then when I tap anywhere on the screen the sounds starts and the console shows:

[Log] played

When I remove that option the promise method works to play things on the first tap in mobile Safari (but still not on desktop so far).

[jagthedrummer]

OK, I got something working for all platforms (as far as I can tell) by doing some redirects and I just wanted to document some things I ran into in case it helps anyone in the future. (Including possibly future-me.)

At a high level, previously I was doing this:

• someone clicks a play button
• player component makes a call to the API server to grab a signed URL for the sound file that's stored on S3
• pass that signed S3 URL to stereo.play
• Ember Stereo (or Howler or Native Audio?) retrieves the sound file
• sound begins to play

Now I'm doing this:

• someone clicks a play button
• immediately pass a redirect URL to stereo.play
• Ember Stereo (or Howler and/or Native Audio?) tries to retrieve a sound file from the redirect URL which hits my API server
• API server generates a signed S3 URL and redirects to that URL
• Ember Stereo (or whatever) follows the redirect to S3 and retrieves the sound file
• sound begins to play

My first attempt was to generate a redirect URL that looked like this:

https://api.seshy.me/bounces/42/download

When I tried to pass that to stero.play it didn't work and sound-error-details showed an error about an unsupported format. It was complaining before even attempting to fire off a network request, so I guessed that it might be trying to infer the format from the URL.

So I tried appending the file name to the URL like this:

https://api.seshy.me/bounces/42/download/some_sound.mp3

That seemed to work great, but then I noticed that some files that had previously been playing fine would fail with an error saying Failed to open media.

After looking into it I realized that all of the failing files had more than one . in the file name.

This would work:

https://api.seshy.me/bounces/42/download/some_sound.mp3

But uploading the exact same file with a . instead of a _ in the name would cause it to fail:

https://api.seshy.me/bounces/42/download/some.sound.mp3

So to be sure that user generated file names can't break things I ended up making a fake filename that gets handed to Ember Stereo:

https://api.seshy.me/bounces/42/download/bounce_42.mp3

A separate issue was how to authorize the call to the redirect URL on the API server.

I tried passing an xhr hash to stereo.play with an Authorization header:

let { sound } = await this.stereo.play(this.identifier,{
  xhr: {
    headers: {
      'Authorization': `Bearer ${this.session.data.authenticated.access_token}`
    }
  }
});

That would raise an error in the console saying xhr options are not supported in NativeAudio.

So I tried to force it to use Howler:

let { sound } = await this.stereo.play(this.identifier,{
  useConnections: ['Howler'],
  xhr: {
    headers: {
      'Authorization': `Bearer ${this.session.data.authenticated.access_token}`
    }
  }
});

When I did that I couldn't see the header being added in the network call to my API server, and so it was returning 401 Unauthorized, which is to be expected when the header is missing. As far as I can tell the Howler connection only accepts the xhr option, but doesn't actually use it.

So finally I ended up tacking the token onto the redirect URL as a request parameter. That's not exactly ideal, but it does work:

https://api.seshy.me/bounces/42/download/bounce_42.mp3?Authorization=Bearer%20xyz

After doing all of this sounds will play on the first click/tap on every platform that I've tested.

[jagthedrummer]

@jkeen I accidentally click "Close with comment" instead of just "Comment". Not sure if there's anything else on this that you want to address...

Either way, thanks for the help, and the awesome library!