Audio stuttering under CPU stress

[Guy-kun]

My previous issue was closed before I managed to reply and I don't think my follow-up was seen...

Previous issue: https://github.com/superpoweredSDK/Low-Latency-Android-Audio-iOS-Audio-Engine/issues/113

To follow up, I am not doing any DSP or additional overhead but am running a game entirely in native code using the cocos2dx library.

My audio processing callback is

if (player->process(stereoBuffer, false, numberOfSamples, playerVolume, 0.0f, -1.0 ))
        {
            SuperpoweredFloatToShortInt(stereoBuffer, audioIO, numberOfSamples);
            return true;
        }

and buffers initialized like so, as per the Android Cross example

player = new SuperpoweredAdvancedAudioPlayer(&player ,playerEventCallback ,samplerate, 0);
audioSystem = new SuperpoweredAndroidAudioIO(samplerate, buffersize, false, true, audioProcessing, 0, 0);
stereoBuffer = (float *)memalign(16, (buffersize + 16) * sizeof(float) * 2);

where the buffer size and sample rate are passed in as values from

audioManager.getProperty(AudioManager.PROPERTY_OUTPUT_SAMPLE_RATE);
audioManager.getProperty(AudioManager.PROPERTY_OUTPUT_FRAMES_PER_BUFFER);

However in-game and exasperated by running a screen recorder audio performance is shoddy and lots of popcorn/stuttering is audible, (less frequent but present without strain)

From my understanding the audio processing callback should be called at a high priority and supposedly not blocked by other native threads? But I'm not sure what's going on. Additionally, it feels like the buffersize isn't big enough but even multiplying it by 8 I'm seeing the same effect. To compare, another game I've tested that uses FMOD for playback has zero performance issues when screen recording.

All in all, the problem can't lie inside the actual processing callback but something external, timing related or buffer size related I feel.

[svenoaks]

To interject, I don't think this an issue with Superpowered, it's because OpenSL is crappy on some devices with some versions of Android. Have you tried on different device? There is literally no way to increase the buffer size in any way to prevent it either, you may think you are increasing but it is really not doing anything. Please read the following article:

http://heatvst.com/wp/2013/11/30/high-performance-low-latency-audio-on-android-why-it-still-doesnt-work/

Thankfully this is only a problem on handful of devices nowdays, mostly they run with the native buffer size with no problems even under heavy CPU load.

[superpoweredSDK]

Yes, please try it on another device as well. It seems like FMOD is not doing any processing in the audio processing callback, but processes audio in another thread, and just doing copy from a fifo buffer (of course this increases latency). Perhaps you can try that approach as well. SuperpoweredAndroidAudioIO supports this, just submit a negative latencySamples when you create it, so if it was 128, then submit -128.

[Guy-kun]

Thanks very much for the replies, I've read the link too and I myself did some digging around in systrace finding nothing useful really except callbacks taking a long time to complete. I see about not being able to increase the buffer size too, thanks for that. I did wonder why it seemed to have no effect.

I've only tried on two devices, but my main device is a Nexus 6P running 6.0.1 (secondary device was an old Sony tablet and that had much worse issues in performance overall). I'm pretty disappointed if this comes down to purely a device thing as the 6P is pretty much a flagship phone.

@superpoweredSDK Sure, this would be ideal for me actually and if this is working well for FMOD and the article example it seems perfect. My main reasons for using Superpowered are a lower overhead and a more reliable latency compared to Android high-level APIs, I'm willing to accept a higher latency as this can be compensated for in-app (I'm not doing any real-time processing after all).

I've tried sending this parameter to SuperpoweredAndroidAudioIO's constructor: specifically '-512' on the Nexus 6P but after running I don't seem to see any improvement (my gut reaction was that it sounded worse but this could have been by chance)

I noticed that the docs say @param latencySamples How many samples to have in the internal fifo buffer minimum. Works only when both input and output are enabled. Might help if you have many dropouts. so could this be the reason this isn't working? Is there any additional setup I need to do to get this to work like enabling input? (I don't actually need input whatsoever in-app)

[superpoweredSDK]

You can try enabling input, it doesn't hurt. I have a Nexus 6P as well and it runs without any audio glitches, but i didn't try to record the screen.

[Guy-kun]

Okay, I'll try out enabling input too. I actually get the odd sound pop even when not recording the screen, I just found it as a way of making the problem more apparent since it happens during high CPU utilization.

Otherwise, is there any way to know when the thread-based processing and copying of buffer data is working properly once I've tried to enable it together with input?

[superpoweredSDK]

Yes - if it still plays the audio you send, and you submit a negative latencySamples value, then you have it.

[Guy-kun]

That's not very clear though, the documentation says

Works only when both input and output are enabled.

However when passing a negative latencySamples to the constructor as such without input enabled

new SuperpoweredAndroidAudioIO(samplerate, buffersize, false, true, audioProcessing, 0, 0,-buffersize);

Audio is still working. Is the documentation incorrect? or is the copy buffer silently not being used?

[Guy-kun]

I also can't enable input due to not including the 'Record Audio' permission in my application. Is there any reason this is needed for this buffer copying behaviour?

I unfortunately can't include the permissions as it would stick out like a sore thumb in a game that requires no access to the microphone

[superpoweredSDK]

1. Just enable input please and add record audio permission for your test, to check the hypothesis FMOD processes on a different thread or not. Audio goes through the "regular way" otherwise.

2. Please check what you do in the player callback, that's called from the audio processing thread as well.

3. Please check if FMOD starts audio silently, simultaneously to Superpowered. Perhaps they conflict when screen recording starts.

[Guy-kun]

1) okay, so sending neagive latency samples when input is disabled does indeed do nothing then as I understand. I'll try enabling input.

2) by player callback do you mean the event callback? Surely this isn't called often and should have an impact? Regardless, the only code in that event callback is
if (event == SuperpoweredAdvancedAudioPlayerEvent_EOF) { if (!looping){ player->pause(0,0); } };

3) I'm not sure what kind of conflict you are talking about. I am only using Superpowered in my application and the mentions of FMOD were in another app I used to compare performance on my device (which I did not create), not one I have full source access to. What should I be checking?

[superpoweredSDK]

1. That looks good.
2. I thought your game uses Unity perhaps, which launches FMOD automatically inside.

[svenoaks]

I still say try on another device too. It's not really about flagship device, I have a Nexus 9 that had this problem, and still does to some extent. Whenever I did anything besides the playing the audio in my app, such as run animations or even load an ad, there would be popping/static. I also noticed it happened in other apps, including YouTube. An OTA update cleaned it up somewhat, but I still can't run all the animations I would like in my app because it can cause static. I just played a game on it today by another developer, and noticed some popping still, which didn't happen when played on another device.

I also had an HTC One m7 which was a flagship device which had lots of it too in many apps when it had Android 4.4, then the OTA update to Lollipop fixed it completely.

On both devices, CPU activity in other threads/processes would exacerbate the problem.

On the other hand I have $10 Walmart phone with Android 4.4.2, which never has audio problems no matter what I do.

So, don't pull hair over this issue, I don't think there's much that can be done.

[Guy-kun]

I haven't had a chance to test enabling input yet, I'll post my findings here within the next few days, please don't close this issue yet! Would be good for anyone else encountering this.

I'm starting to realize the same thing @svenoaks. After you've mentioned it, I've heard bits of popping when doing heavy tasks in apps like youtube (minimizing video) and so forth... I guess it really is a device-specific issue. (As expected) iOS runs perfectly at all times for my app. I'll try to do a few more things to get popping down to a minimum on my device (like the buffer copying) but I'll keep the limitations in mind now... It's a real shame

[superpoweredSDK]

Actually, we are planning to remove that buffer-copy on a separate thread thing, as it doesn't seem to improve anything. Please let me know your findings.

[svenoaks]

To make a separate thread help anything the audio processing should be on the new thread feeding into a buffer, then the OpenSL callback just copies from that buffer. Not sure how current implementation does it.

[superpoweredSDK]

@svenoaks Yes, it's exactly what it does when you submit a negative latencySamples value. SuperpoweredAndroidAudioIO is open source, you can check.

[Guy-kun]

A few findings after some tests:

1) My initialization was wrong previously, I was missing the clientdata parameter and parameter order ended up wrong, this didn't have any impact though.

2) Enabling input while targeting android 23 results in the cryptic SIGEGV of error creating AudioRecord object; status -1 Which I guessed my way through to being because the user had not explicitly enabled the permission. This seems a bit difficult to deal with, especially if you enable recording at app start. In the end, I just built against 22 to circumvent this.

3) Enabling input and passing negative latencySamples has no notable difference (in my un-scientific opinion, it sounds worse), playback is still bad and in one of my tests even 100% cut out and never returned.

I guess if this feature for copying buffer data isn't even being pushed for support and there's nothing I can do I'll leave things as they are. I'll likely try out FMOD too to see how well it performs in my app since it seems to have no problems in other applications if there's nothing more I can do

[superpoweredSDK]

2) From Android 23, the permission system is very different. You don't specify permissions in the manifest file, but have to write your custom permission handler in Java.

Perhaps your issue may be related to the SuperpoweredAdvancedAudioPlayer. It has only a 2 seconds long buffer inside, and audio is decoded on some background threads. Maybe those threads get no CPU in time when you stress your system with a screen recorder. Currently there is no way to increase those buffers.

[svenoaks]

Is there anyway the decoding could be made to happen on the same thread (which would be the high priority audio one)?

[Guy-kun]

I think that's the default behaviour when positive latencySamples are passed? Despite having higher priority that thread is still not reliable due to getting interrupted and only having processing time within the callback time

On Wed, 27 Apr 2016, 8:33 p.m. Steve Myers, notifications@github.com wrote:

Is there anyway the decoding could be made to happen on the same thread (which would be the high priority audio one)?

— You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub https://github.com/superpoweredSDK/Low-Latency-Android-Audio-iOS-Audio-Engine/issues/115#issuecomment-215055896

[superpoweredSDK]

@Guy-kun In the "SuperpoweredAdvancedAudioPlayer" theory above, the audio processing thread finishes in time, but the player is "starving". Audio decoding happens on a background thread, not the audio processing thread. And when your screen recorder starts, that background thread may be called too late, and the player's internal 2 seconds long buffer may not be long enough. It has nothing to do with latencySamples.

@svenoaks Audio decoding in the audio processing thread is not a good idea, because audio decoding may read data from disk. Despite of we use mmap-ped files where we can, reading data from disk may be too slow to be done in the audio processing thread.

The only way to test this idea would be introducing some buffer size selection for the advanced audio player, however we don't have the resources (time) to extend the library for that at the moment.