Troubleshooting iOS background audio playback

[drew-512]

• Unity 2021.3.7
• AVPro Core 2.5.7 (1282)
• iOS 15.4

Hi AVPro friends,

We got AVPro with the intent to use it as an audio playback engine for a Unity based audio player.

Currently, AVPro seems to auto-pause when the iOS Unity app goes to background. I've double-checked the Unity background settings, and get Xcode runtime output when I move the app to background:

-> applicationWillResignActive()
-> applicationDidEnterBackground()
2022-08-02 16:23:03.538976-0500 PlanetPlayer[22052:11704175] [Player] player rate changed to 0.000000
-> applicationWillEnterForeground()
-> applicationDidBecomeActive()
2022-08-02 16:23:13.056482-0500 PlanetPlayer[22052:11704175] [Player] player rate changed to 1.000000

...which seems to suggest that AVPro is pausing the audio, whereas we want to have full control over that (and not pause when going to background). My attempts to find where this auto-pausing is controlled have been unsuccessful, so they are presumably in the AVPro binaries and outside our control.

Also, best practices and recommendations to produce a stable, happy audio player and kindly invited. 💙

Thanks and keep up the great work!

Drew

[MorrisRH]

Hey @drew-512, pausing playback is standard iOS behaviour when you move an app to the background. In order to have playback continue you will need to do the following:

1. Add the background audio capability to your application in the Xcode project, details can be found here
2. Change the AVAudioSession category to "playback", details here
3. Make sure the MediaPlayer component has "Audio Mode" set to "System Direct" in the iOS platform settings

The only awkward part is point 2 as it will require native code. You can do this directly in the generated Xcode project by editing the method - (void)startUnity:(UIApplication*)application in UnityAppController.mm (approx. line 117) and replacing:

[audioSession setCategory: AVAudioSessionCategoryAmbient error: nil];

with:

[audioSession setCategory: AVAudioSessionCategoryPlayback error: nil];

[drew-512]

Lovely, thank you sir.

Any other best practices that come to mind would be appreciated. We've been authoring realtime AV software on native iOS, OS X, and Windows for 10 years now, so doing things anew in Unity is a little scary -- glad to have AVPro at our backs!

Drew

[drew-512]

Hi @MorrisRH,

I made these changes and a UnityAppController.mm patch for AVAudioSessionCategoryPlayback and had success.

However, since System Direct bypasses Unity's AudioSource, I'm not seeing any way to get access to FFT data in Unity. This is unfortunately a major issue for us since having fancy audio visualizers is a core value-add of our product.

Is there any way to get PCM audio out of AVPro? We have our own FFT analyzers and filters, but for this approach, I'd need a live PCM tap so I can do the processing and then hand that off to our visualization subsystem. Perhaps I could intercept where AVPro passed audio to iOS and use that? That of course resides within AVPro's iOS glue bundle, and so I have no way to access that. Perhaps there is another solution?

We've made music visuals for almost two decades, so it'd be a tough setback if we can't navigate this and would have to reevaluate our audio engine options (check out SoundSpectrum and Tunr if you're curious). Fun fact is I made some of the original Winamp visualizers that made their way to be built into iTunes back in the day, so I'd love to figure this out with you and happy to take this to email.

Anyway, I would be open to checking out the code for AVPro's iOS 'system direct' glue if that is available. Perhaps I could fork it and could gift it to AVPro?

The good news is that the audio data for visualization is only needed when the app is in the foreground, so maybe an audio tap is available on the AVPro side before it's passed off to iOS?

Thanks, Drew

[MorrisRH]

Hey @drew-512 The Unity audio output mode will deliver what you need to be able to run your FFT analysers, however you will lose support for background audio playback. There's currently no way to switch between the two modes at runtime although this may be possible but will require some investigation from our end.

[drew-512]

Ok, I figured as much.

Well, I've decades of work on the native iOS and desktop realtime audio and graphics (see previous links), so I'm open minded on how I can contribute or work with you.

Drew

[drew-512]

Hi @MorrisRH,

Sorry if this is a dumb question but what is AVPro's GrabAudio(float[] buffer, int sampleCount, int channelCount)? Couldn't I use that as an audio tap to do necessary audio visulization?

Thanks, Drew

[MorrisRH]

We have a mini spectrum visualiser built into our media player demo scene. If you look at MediaPlayerUI.cs and search for the method named "UpdateAudioSpectrum" you'll see how we're doing it.

[drew-512]

Hi, yes I saw that. However, that seems to just pull FFT data from Unity's AudioSource each frame and pump that into AVPro's player UI shader. Of course that updating stops once AVPro is set to use the audio engine.

GrabAudio() on the other hand appears to be exactly what we're looking for (live PCM data that can we use in our visual engine).

I suspect GrabAudio() was once intended to be used within UpdateAudioSpectrum() when not using AudioSource but for whatever reason that code is not present. It would be lovely that GrabAudio() does the trick but I didn't want to hang our hat on that it's dead or on it way out.

[MorrisRH]

I think reading data from the AudioSource using GetOutputData() is your best bet then. GrabAudio is our route for pulling audio data back from the native side of the plugin and feeding it into Unity. Were you to use it you would break how our audio playback components work (of course you could reimplement these yourselves).

[MorrisRH]

I've had a play with things and I think I can give you the best of both worlds; being able to get you the audio data via GrabAudio and allowing background playback.

If you can send an email to unitysupport@renderheads.com with the following subject "#1243 Troubleshooting iOS background audio playback", I'll send you a build with the necessary changes.

[Chris-RH]

Thanks @drew-512, email received :)

[MorrisRH]

Closing this issue as now resolved with version 2.6.0.