Audio system is broken on iOS with AIR 50.2.2.5

[sjabberwocky]

I recently updated our game to the latest AIR 50.2.2.5 and checked everything except the sound, then released the new version to everyone.

On desktop and Android the sound works fine. On iOS it looks like the sound is working at 200-300% of the volume scale with some audio artifacts.

This fix for the audio system would be appreciated ASAP, cause new version of the game available to all our users. And I'm not sure which stable version of AIR I should use to downgrade our build.

@ajwfrost Andrew please take a look.

iOS build is here - https://apps.apple.com/us/app/ancient-planet-tower-defense/id869341551

[sjabberwocky]

Just a quick note on what players think about the latest update and new issues with sound. Perhaps it will add some context to this issue.

This is a quote from our support system:

About your last update. Think the original version of the game was better. The new version sounds like someone took this game and shitted it up. The sound effects make it sound like you are trying to tell someone to go jerk off, sounds like you keep calling someone a jerk off. You. Say ass wipe. Have you thought of maybe going back to the original version or maybe changing this game so that you don't sound like you're trying to offend anyone. Would appreciate this. Especially if you want people to spend money on this game, which I no longer do. Too bad you shitted this game up, or it sure sounds like it to me. Thank you. An offended user.

[ajwfrost]

Hi

I was just looking at this app, it sounds fine on an iPhone 11 .. but then I tried on an iPhone 14 and yes I see what you mean - the sound is significantly louder but also interrupted quite a lot with glitches/pauses. We already have a known issue (well, we don't know the root cause) on iPhone 14 with audio that's glitching like this, but that's typically when the display adaptive brightness is on, and when the phone is in bright light.

Can I check, what version of the AIR SDK were you using previously? We haven't really changed anything about the sound in iOS, there was one sound-related problem that was solved just by changing how we handled background/foreground notifications but you're not affected by that one.

We have been looking to just implement simpler sound outputs so can I check how you're playing your music and triggering the sound effects? Is it just Sound.play() using local asset files? Do you do any sound transforms either globally or on a single channel?

I'm curious as to how we can't reproduce this with simple test cases .. if you are able to create a simple swf that exhibits the problem, we can try to take a look; otherwise I'm thinking:

• short term, if it was working with an earlier AIR SDK version, then we can check for differences and/or push out a version with that runtime but with the updated build values for Xcode/iPhoneOS SDK
• then slightly longer term we can provide the alternative mechanisms for playing sound via simple platform APIs..

That "offended user" has a bit of an attitude problem :-( do you get to see what handset they were using?

[sjabberwocky]

I was just looking at this app, it sounds fine on an iPhone 11

On my iPhone 12 mini with iOS 16, the audio issue is here.

Can I check, what version of the AIR SDK were you using previously?

It's AIR 33.1.1.935.

We have been looking to just implement simpler sound outputs so can I check how you're playing your music and triggering the sound effects?

I have been playing sounds this way for about 10 years and maybe some of it is outdated now. There was a problem with looping mp3 files, so there is loopOffset for that. And there was something about initialising loop sounds.

I also transform each channel individually, but only with volume.

public function play() : void
{
    busy = true;
    channel = sound.play(loop ? loopOffset : 0, loop ? 999 : 0, new SoundTransform(volume));

    if (!channell)
    {
        busy = false;
        return;
    }
    channel.addEventListener(Event.SOUND_COMPLETE, loop ? repeat : clear);
}

private function repeat(event : Event = null) : void
{
    clear();
    play();
}

private function clear(event : Event = null) : void
{
    if (channel != null)
    {
        channel.stop();
        channel.removeEventListener(Event.SOUND_COMPLETE, clear);
        channel = null;
    }
    busy = false;
}

public function pause() : void
{
    if (channel != null)
    {
        position = channel.position;
        clear();
        busy = true;
    }
}

public function unpause() : void
{
    if (!channel)
    {
        channel = sound.play(position, 0, new SoundTransform(volume));
        if (channel)
        {
            channel.addEventListener(Event.SOUND_COMPLETE, loop ? repeat : clear);
        }
        else
        {
            clear();
        }
    }
}

short term, if it was working with an earlier AIR SDK version, then we can check for differences and/or push out a version with that runtime but with the updated build values for Xcode/iPhoneOS SDK

As latest version was built with AIR 33.1.1.935 and we use many Distriqt's ANE migrated to 50.2 I'm not sure we'll downgrade easily.

then slightly longer term we can provide the alternative mechanisms for playing sound via simple platform APIs

That would be nice.

That "offended user" has a bit of an attitude problem :-( do you get to see what handset they were using?

No, he does not send the report correctly. We do not have automatically collected technical data.

[teotigraphix]

That "offended user" has a bit of an attitude problem :-(

That is verbal abuse, plain and simple.

[sjabberwocky]

That is verbal abuse, plain and simple.

But he find and point on this issue.

[teotigraphix]

But he find and point on this issue.

Of course, I am just stating something for the wind.

[ajwfrost]

As latest version was built with AIR 33.1.1.935 and we use many Distriqt's ANE migrated to 50.2 I'm not sure we'll downgrade easily.

One thing you may be able to try out .. given how a lot of this works, you may be able to take the files in the "lib/aot/lib" folder from 33.1.1.935 and copy them over the top of the files in 50.2.2.5. I'm not 100% sure but think that the differences may be isolated enough that the build tools and the runtimes (on iOS anyway) can be separated.

Your audio code looks straightforward enough! I'm not sure of the logic on 'unpause', I think there's a typo there (presumably not in your actual code) where you call sound.play() and then if that results in a null channel, you add an event listener to it ... presumably the ! shouldn't be there.

We will see if we can reproduce this with a lengthy MP3 file and see what happens - the MP3 presumably loaded via Sound.load() with a URLRequest with an app URI? or do you load it from a byte array?

I've just checked back the history of the audio code for iOS, there is only one change that was made to the iOS-specific audio handling code, on September 8 2020, which was to fix an issue that happened when audio was being interrupted. And of course, that change is also in 33.1.1.935. So this might mean that the difference is more about the OS compatibility level or similar, or just with iOS - the APIs that are used are somewhat outdated now, Apple created new ways of handling audio output which is what we've been looking at more..

Will let you know what we find from trying to reproduce it and whether we can trace that to some routing or configuration setting in the OS..

thanks

[ajwfrost]

@sjabberwocky we've tried a variety of different options and combinations of building/packaging with the AIR versions 33.1.1.935 and 50.2.2.5, but aren't able to do anything that makes a difference... consistently, we find that with an iPhone 14, there are occasional glitches/microsecond drops in audio, but the volume levels are all as expected. If you do pass in a volume above 1, then yes there's a massive distortion and it's quite loud, but that's the same with the earlier AIR version too.

Have you managed to check whether it works to build against the 33.1 runtime but using the 50.2 build tools? and whether that made a difference?

If you're able to provide any SWF that demonstrates the issue i.e. where we'd hear a difference packaging with 33.1 vs 50.2, we can try to investigate more, otherwise we may be better off just trying to hook up the APIs to play back sound using a more straightforward/up-to-date iOS mechanism..

thanks

[sjabberwocky]

@ajwfrost If it helps I can create a simple app with one button with click sound and my sound system from the game and I could email sources to you.

[ajwfrost]

@sjabberwocky yes that would be ideal! thanks

[neutronized]

@ajwfrost hi, is there any update on this issue? I feel this audio problem should be prioritized as experiencing such audio glitches on iPhone14 might negatively affect the apps on many levels. Thanks

[ajwfrost]

Hi @neutronized - yes, we had been looking at this: currently though we've not got a way to reproduce this in terms of hearing a difference between an application built and packaged using AIR 33.1 vs AIR 50.2; when we create an app using the audio code provided by @sjabberwocky it's working the same for both builds.

From playing the game mentioned in the original post, we hear a difference between iPhone 11 vs iPhone 14, both running the latest iOS, and presumably both built with AIR 50.2. So I guess this is partly iPhone model specific, but also it would be good if we can confirm that it used to be okay on iPhone 14 with the older AIR runtime..

We did have a change go in around September last year around MP3 decoding which was caused by changes to Apple's compiler but (a) that code looks fine and shouldn't actually change the behaviour, and (b) it would be the same also on iPhone 11. So I don't expect that this could be the cause.

Currently what we're looking at is to see whether it helps to ensure the new AIR media classes work for simple sound output on iOS as this would then be an alternative and OS-native way of doing things... but if we are able to get some SWF files where there's a difference in output between AIR 33.1 and AIR 50.2, we can look at providing more of a proper "fix" here.

(As an aside, I'm also wondering whether we should peg the volume level to the expected range of 0..1 since it seemed more natural to us to pass in "100" for full volume - but that's definitely not what should be done!!)

thanks

[sjabberwocky]

@ajwfrost I just sent you a test app.

While testing this issue, I've noticed that with the latest iOS 16.4.1 (a) update, this issue seems to be gone. Need confirmation from other developers.

[ajwfrost]

Thanks, we'll take a look.

Just playing with your app with a decibel meter (not massively scientific! checking the ranges during sound effects):

• iPhone 11, iOS 16.1.1 -> around 68-70 dB
• iPhone 14, iOS 16.0.3 -> around 75-77 dB (it sounds a lot louder, I guess 7dB is actually fairly significant...)
• iPhone 14, iOS 16.4.1 -> still around 75-77 dB and with the choppy-ness..
• iPhone 14, iOS 16.4.1a -> same

So I'm not entirely sure about those reports...? though with all of this, it's just slightly loud (and not great audio quality with the glitchy bits) but turning it down from full volume and it's fine. Makes me wonder whether I'm hearing the same thing..

[neuronix]

The sound glitches are absolutely horrible on our background music.. it sounds like the iPhone has a speaker issue.

Tested with 50.2.2.6 on iPhone 14 Pro 16.5.1

[cleverbeapps]

Guys, @sjabberwocky @neuronix, are you playing sounds by distriqt MediaPlayer ANE, or by AIR native Sound?

[sjabberwocky]

Guys, @sjabberwocky @neuronix, are you playing sounds by distriqt MediaPlayer ANE, or by AIR native Sound?

We use native AIR sound.

[neuronix]

Same here, native sound

[ajwfrost]

If you can download AIR SDK 50.2.3.1, there's code in there which means you can play back audio files on iOS (and Windows) using a new mechanism. Not yet fully featured (e.g. no 'completed' type event, and no volume I just realised!) but it may help...

Example code:

package
{
    import air.media.AudioOutput;
    import air.media.FileSource;
    import air.media.MediaElement;
    import air.media.Pipeline;
    import air.media.States;
    import flash.events.Event;
    import flash.display.Sprite;
    import flash.events.MouseEvent;
    import flash.filesystem.File;
    import flash.ui.Multitouch;
    import flash.ui.MultitouchInputMode;

    public class Main extends Sprite 
    {

        public function Main() 
        {
            Multitouch.inputMode = MultitouchInputMode.TOUCH_POINT;
            trace("Application starting");

            // add a play button...
            var s : Sprite = new Sprite();
            s.graphics.beginFill(0x00ff00);
            s.graphics.moveTo(0, 0);
            s.graphics.lineTo(200, 100);
            s.graphics.lineTo(0, 200);
            s.graphics.lineTo(0, 0);
            s.graphics.endFill();
            s.x = s.y = 100;
            s.addEventListener(MouseEvent.CLICK, startAudio);
            addChild(s);
        }

        private function startAudio(e : Event) : void
        {
            trace("Starting audio");
            var f : File = File.applicationDirectory.resolvePath("mp3Aud128kbps.mp3");
            if (f.exists) trace("Found MP3 file");
            else
            {
                trace("Could not find mp3Aud128kbps.mp3");
                return;
            }
            var fileSource : FileSource = new FileSource();
            fileSource.file = f;
            // check we have some audio routes..
            var audioRoutes : Vector.<String> = AudioOutput.getAudioDescriptors();
            if (audioRoutes.length == 0)
            {
                trace("No audio routes found");
                return;
            }
            trace("Selecting audio route = " + audioRoutes[0]);
            var audioOut : AudioOutput = new AudioOutput(audioRoutes[0]);

            var pipeline : Pipeline = Pipeline.createPipeline(fileSource, audioOut);
            pipeline.addEventListener(MediaElement.STATE_CHANGE, onPipelineStateChange);
        }

        private function onPipelineStateChange(e : Event) : void
        {
            var pipeline : Pipeline = e.target as Pipeline;
            trace("Pipeline state change -> " + pipeline.readyState);
            if (pipeline.readyState == States.READY) pipeline.play();       
        }

    }

}

Feedback/suggestions very welcome..

[neuronix]

Has there been any progress on fixing these regressions ?

The whole point of using AIR for us, is having a single code base to reliably work on multiple platforms so using a specific new API for sound on iOS is going to require major refactoring just for testing.

Furthermore, the bug concerns long and repeating sounds like music, where the complete event and volume management are needed.

[FliplineStudios]

Does anyone know if these issues are still present with iOS 17 on the iPhone 14 lineup, and/or on the iPhone 15 lineup?

[ged-mc]

Unfortunately, I can confirm that this serious audio issue is still present on IOS 17 / iPhone 15 ( AIR SDK 50.2.2.4 ) Audio is a major part of our app and we use sound complete events extensively so a workaround isn't an option.

[ged-mc]

Could this be related to: AudioUnit V2 app with RenderCallback glitchy audio on iOS 17 with iPhone 14/15

[ged-mc]

Also maybe related to this thread - which has a Solution!

AudioUnitRender error kAudioUnitErr_CannotDoInCurrentContext on iPhone 14 only

[FliplineStudios]

@ged-mc When this is happening for you, does turning off the iPhone's Auto-Brightness stop it from happening? (Settings > Accessibility > Display & Text Size > Auto-Brightness) I just picked up an iPhone 15 and I'm getting the audio glitches in all of our existing apps, though it seems like it's when in a well-lit room and Auto-Brightness on (which I think it is by default). If I'm in a dimly-lit room, or turn off Auto-Brightness, it seems to go away in my brief tests. Though not super helpful for users, if Auto-Brightness is enabled by default on iPhones, with the toggle hidden away in the accessibility settings.

And for reference, I'm hearing the same auto-brightness-related glitches in anything published all the way back to AIR 32, so it doesn't seem to be something that was introduced in the 50.2 builds specifically.

@ajwfrost Should there be a separate issue here for tracking audio glitches related to Auto-Brightness, or are they interconnected?

[ajwfrost]

Hi @FliplineStudios -> our current belief is that these are all interconnected and related to how the audio threads/priorities are being handled. Although @ged-mc that looks to be an interesting article, thanks -> we're not using those exact settings but it could perhaps be interrelated. I'll pass that on to the developer who was looking at the audio issues to see if any of that rings true...

thanks

[ged-mc]

Hi @FliplineStudios , @ajwfrost, When I tried to reproduce the 'choppy sound' problem yesterday on the iPhone 15 pro the app audio played perfectly with auto-brightness on or off.

Today I tried again and it was okay at first ( auto-brightness on ). However dimming the screen seemed to trigger the choppy sound. Switching to ( auto-brightness off ) restored the audio back to perfect. Toggle auto-brightness while looping one mp3 even starts and stops the issue.

So there certainly seems to be a connection to screen auto-brightness.

I will check all this again on the iPhone 13 pro, but I'm pretty sure that phone is okay.

[FliplineStudios]

@ajwfrost I just tested out the "Pipeline" code above, and can confirm that at least doesn't have the crackling/glitches that the internal Sound/SoundChannel APIs do on iPhone 15. Using this setup was fairly convoluted though, and not really usable in live apps as-is with the missing features like looping, real-time volume/pan, and using internal sound files in the SWF...

It would be great to get an idea of whether work is being done to correct these glitches in the usual Sound/SoundChannel code on iOS, or if we're expected to eventually write completely different code on iOS using these media pipelines for handling any sort of sound. And if changes are being worked on for Sound/SoundChannel glitches, if they would be coming in 50.2.3.x or 50.2.4.x, or not until an eventual 51.x?

Our games use multiple layered music tracks that need to play simultaneously and stay in sync, and which will each fade volume up and down during gameplay, and all sound/music tracks are currently all compiled into the SWF itself. Concerned about how well we could still accomplish this if we need to switch over to a different audio system for iOS like this Pipeline, and especially if we'll need to be handling sound completely differently on Android...

[ajwfrost]

Hi @FliplineStudios - we are going to try to get these glitches sorted out using the existing audio pipeline within AIR. It's a little challenging because of the complexity of what's there (and of course we didn't write it, so the risk of regression is a little higher than normal). But with some of these hints, we may be able to get to a workaround scenario for it. These would be put into whatever version we've got at the time - 50.2.4 most likely and we'll probably support that for a little while, as when 51.0 first comes out there will be a period of updates to that before we think it's stable/complete enough for production distribution..

The longer term goal though is to implement the necessary features using the newer API mechanisms: these would be cross-platform so it's not like you would have different APIs to use for different platforms. And we'd either ensure all these sorts of features/capabilities are built-in, or could be easily done in a SWC file by a third party...

thanks

[FliplineStudios]

@ajwfrost Great, thanks for the update! If there's anything we can do to help with testing builds/patches/etc. as you're working through the glitches we're happy to help.

[ged-mc]

@ajwfrost That's great thanks. Likewise happy to help.

[FliplineStudios]

@ajwfrost Just a quick note: I'm noticing that Auto-Brightness on iPhone 14/15 is also having an impact on app performance with framerate hiccups, not just on the audio glitches. Not sure if you're also investigating framerate/rendering hiccups due to Auto-Brightness on these devices, I can put together a separate issue here for that if it's not already being worked on...

[ajwfrost]

Hi @FliplineStudios -> thanks for adding that extra entry, but can I check from your description above:

Auto-Brightness on iPhone 14/15 is also having an impact on app performance with framerate hiccups, not just on the audio glitches

Do you see an impact on performance on an app that doesn't have sound? Or are these framerate hiccups likely to just be caused by the sound 'complete' notification per that new issue? (I'm wondering whether it's the same problem ..)

@neuronix would you be able to hook your app up to Scout and see whether you get a similar pattern to that shown in the above linked issue please?

thanks

[FliplineStudios]

@ajwfrost I just commented on #2917 with more details, but the framerate/performance impact seems directly related to "Dispatching Sound Complete" from what I can tell. In an app with no sound, there doesn't seem to be any framerate issues. Additionally, no actual Event.SOUND_COMPLETE events are being fired despite Scout labeling it that, and the lag/glitch seems to occur twice per second regardless of the framerate being used in the app...

[ajwfrost]

Hi

So we've delved down further with this and are finding that the delays are happening within an Apple API: AudioQueueEnqueueBufferWithParameters. For some reason this is taking longer if there's variable brightness and the sensor has a lot of light input...

We've got this thread set at the highest possible priority, but it's still blocking. Currently trying to see if there's any way we can disassociate this so that it doesn't block the main runtime loop; whether this would stop the audio choppiness is tbc but it should stop the animation problems.

The other Apple support links above are interesting but don't seem to be exactly the same problem, at least from an API usage perspective.. but perhaps indicative of wider issues that Apple have introduced with this. We'll see whether we can work around it..

thanks

[ajwfrost]

... and a quick update - we can eliminate those delays in the 'sound complete callback' i.e. the main rendering thread can be unblocked here and we get smooth animations, no delays in Scout, etc. But the audio is still choppy. Really seems to be something fundamentally wrong in the audio Apple subsystem... we can do some more investigations but I'm not massively hopeful now...

[FliplineStudios]

That's so strange... I'm surprised that I haven't seen anything from non-AIR developers having similar issues with sound or with AudioQueueEnqueueBufferWithParameters, I haven't turned up anyone else talking about this issue on the Apple Developer forums, stackoverflow, etc. Is this just an uncommon API usage where others aren't seeing the same issue? I've at least checked out some Unity-built apps and they don't seem to have the auto-brightness audio problem.

If there's any more progress, or anything else we can do to help or test just let me know!

[ajwfrost]

Hi

We've completely reworked this code now, adding a couple of threads and more decoupling between the decode buffers that AIR is using for sound composition vs the buffers that are sent off to the OS, or rather, how the samples are copied between these and queued up in the system.

And it seems to have worked.. A test version of the library is available from the below link, I'd be interested in your feedback to see whether we've managed to hit all cases or whether there are still some issues.. https://transfer.harman.com/message/OEPrHpLqL9M74AgomYnhib

thanks

[ged-mc]

Brilliant. Thank you! I will check it.

ᐧ

On Tue, Dec 12, 2023 at 12:22 PM Andrew Frost @.***> wrote:

Hi

We've completely reworked this code now, adding a couple of threads and more decoupling between the decode buffers that AIR is using for sound composition vs the buffers that are sent off to the OS, or rather, how the samples are copied between these and queued up in the system.

And it seems to have worked.. A test version of the library is available from the below link, I'd be interested in your feedback to see whether we've managed to hit all cases or whether there are still some issues.. https://transfer.harman.com/message/OEPrHpLqL9M74AgomYnhib

thanks

— Reply to this email directly, view it on GitHub https://github.com/airsdk/Adobe-Runtime-Support/issues/2642#issuecomment-1851932932, or unsubscribe https://github.com/notifications/unsubscribe-auth/AM5K7JPBJYKCKFBTTWKYGL3YJBD7PAVCNFSM6AAAAAAXV3Y7EGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNJRHEZTEOJTGI . You are receiving this because you were mentioned.Message ID: @.***>

[MalacTheLittle]

Regarding #2933, I was thinking this patch would help, but unfortunately did not.

[ajwfrost]

@MalacTheLittle soon I hope we can update all the iOS video support stuff! there seem to be quite a few fairly fundamental issues with it :-(

[FliplineStudios]

@ajwfrost Thanks so much, I've done some tests with this in one of our apps on an iPhone 15 and iPhone 15 Pro Max, and it seems like the audio is playing perfectly again in a bright room -- amazing job with this rework!

[ajwfrost]

That's good to know, thanks for the quick feedback! We'll kick off a new release shortly to include this.. (will just check if there are any other fixes almost ready for inclusion..)

[ged-mc]

@ajwfrost Sound appears to be playing perfectly in our app on iPhone 15 pro. Thanks again. Great work.