Input dispatching timed out ANRs & ANR rate survey

[neuronix]

Hi guys,

I am mirorring my post from the Starling forum here with Daniel's permission.

We have ANRs on Android in all our apps and would like to know if it's maybe our apps particularly, or if it's AIR related.

Could you please check your Google Dev Console and go to "Android Vitals" > "Overview".

The first section is "Bad Behaviors".

Please share the following values (no need to mention the app, just pick the biggest app you have or the average of all your apps please):

• ANR Rate (around 2.6% on our apps)
• Crash Rate (1.1-1.8% on our apps)

The main ANR we are experiencing is "Input dispatching timed out" triggered by "com.adobe.air.customHandler.callTimeoutFunction". There are heaps of topics on it on the web, and they only proper response by Adobe is that we should avoid doing heavy calculations on user input.

But all our apps use Starling, and Starling defers all user inputs to next frame so this should not be an issue.

[cgascons]

Wow @NB13 that's awesome news! Will try that right away! Edit: @soccerob here's a useful link

[dcampano]

@chichilatte @cgascons I'm curious how you build your APKs. We use FlashDevelop and are currently on AIR 29 but we don't receive any of the extra information in the libCore.so lines:

lib/arm/libCore.so (???)

I'm wondering if there is some build parameter that allows you to get that extra information.

[cgascons]

One quick question though @NB13 , we are making some modifications to use MediaPlayer instead of the Flash's Sound Object and we realised that, in order to assign the resource to the AudioElement component of the MediaPlayer, we have to pass the full path to that element so:

• Should we remove all the Sounds loading code in the game? It looks like it can load anything right away, no matter what.
• In case we shouldn't, is there a way of knowing the url of the Sound object returned by AssetManager's 'getSound(name)' function? The attribute isURLInaccesible returns true and its url attribute is null which I suppose it's due to the strict mode set on.

Excited to try these out ASAP and check whether or not we get also rid of these ANR's!

Thanks!

[pnico]

@cgascons, I think @NB13 is talking about the Android native MediaPlayer using an ANE, there are several extensions available... one common reason for using them is to be able to play formats other than MP3 such as AAC. I'm curious to find out if just not using flash Sound at all solves the problem.. can anyone else confirm this?

[cgascons]

Oh I see! Well I thought he meant flash's Media player. I actually got to play some sounds already with it. Anyways, any extra info you can share will be appreciated!

[soccerob]

@NB13 how are you loading your sounds into the MediaPlayer.resource property? I was previously grabbing the sounds from Starling's AssetManager class which comes out as a Sound class but it's expecting a url to my sound (URLResource)...

Edit: nevermind, looks like it wants a path to the mp3, not a reference from AssetManager. I got it to work by using: var mediaElement:AudioElement = new AudioElement(new URLResource("music/song01.mp3")); musicPlayer.media = mediaElement;

Previously, I was preloading the sounds with AssetManager, seems like this method doesn't preload (which may not be an issue). Any thoughts on not having them preloaded?

[chichilatte]

@dcampano we're on macs so no FlashDevelop. IntelliJ publishes the swf then a custom build script runs the AIR 28 SDK adt command to pump out an apk. Something like this...

adt -package -target apk-captive-runtime -storetype jks -keystore 'YOUR_APP_KEY_FILE.jks' -storepass 'YOUR_APP_KEY_PASSWORD' 'YOUR_APP.apk' 'appDescriptor.xml' 'YOUR_APP.swf' -extdir 'extensions' 'android/icons'

I really don't know why our ANR reports suddenly have populated (???) lines. Maybe we released a debug version of the app by mistake?! That or Adobe / Google changed something.

[cgascons]

@soccerob Not sure how to manage the audio paths right now because I was also preloading all the audio files (or audio folders) when needed and don't really have the path for every audio file I need to use so I'm really not sure on how to proceed here.

I guess the MediaPlayer solution is useful for very small projects having all the audio files in the very same folder but at least from our point of view is not useful for very big projects with hundreds of audio files in different locations. We still need to think about how could we get the urls to every audio source we have, it would be really useful indeed if the url attribute in AssetManager's Sound object could be readable.

@dcampano Also on macs here, we use Flash Builder and AIR 29 (v122), anyways I'm with @chichilatte on this one, I think Google or Adobe changed something because we didn't change the way we create release builds.

[NB13]

Answers to your questions:

1. I'm using ANE - self-developed initialy based on https://github.com/DigitalStrawberry/ANE-Sounds

2. We've removed all sound loading code from as3, everything in java code

3. I don't know anything about AssetManager

4. We are loading sounds like

   MediaPlayer mediaPlayer = new MediaPlayer();
   if (uri.startsWith("app:/")) {
       AssetManager assets = ANESoundsContext.context.getAssets();
       AssetFileDescriptor assetFileDescriptor = assets.openFd(uri.substring("app:/".length()));
       mediaPlayer.setDataSource(assetFileDescriptor.getFileDescriptor(), 
       assetFileDescriptor.getStartOffset(), assetFileDescriptor.getLength());
       assetFileDescriptor.close();
   } else {
       mediaPlayer.setDataSource(uri);
   }
   mediaPlayer.prepare();
   mediaPlayer.setLooping(...);
   mediaPlayer.setOnCompletionListener(...);
   mediaPlayer.start();

5. In other project we use SoundPool, for short sound effects

6. Adobe AIR compress sounds in assets directory, native code works only with uncompressed. So, if you want to load sound from APK you have to - copy it to storage directory, or repack and resign apk with zipalign, aapt, jarsigner tools (as I remember).

[chichilatte]

I'm quite excited now

[cgascons]

I see, downside of that ANE is that is basically designed for small audio clips so it doesn't have any "ON_COMPLETE" callbacks, which is an issue on our end since we need to play the soundtrack in the game. I understand why you guys self developed an ANE based on that one. Anyways will take a look and make a few tests to see if it improves the ANRs rate.

[gosgame]

We tried moving sound playing codes to a worker thread but the AudioTrack ANR's are still coming.

[chichilatte]

We switched to playing sounds with ANESounds and did a release yesterday – lots of installs have updated and so far no ANRs!

[cgascons]

That's amazing news  @chichilatte ! We are currently implementing it as well, have it scheduled for a next Monday release, by the way if you don't mind me asking, do you have kind of a soundtrack system for your game? If so, would you mind sharing how are you managing the end of each tune to start the next one? We have like 7-8 different tracks for each game and don't quite know how to deal with it, if I had like 2 or 3 I could merge them into a unique track and play it looped but it's not our case. Anyways, glad it's sorted out for you!

[chichilatte]

:) Cheers @cgascons, hope it works for you too.

No, we only have short sounds, no long tunes. Sometimes we were listening for a rare sound to complete playing so we could unload it from memory, but now we're just unloading those sounds at a point in the game where we know they will have finished playing.

Definitely worth implementing ANESounds for your short sounds, I'm sure it will have some effect.

+Maybe it's not too hard to add onPlayComplete logic to ANESounds?

[cgascons]

Definitely shouldn't be hard, I actually asked for it but the developer said that due to the fact that Android uses SoundPool class (which supports only short sounds) it does not provide any callback for when a sound finishes playing. Not sure what all this is about to be honest, I will take a look again in case my ANEs keep rising. Anyhow if indeed this ANE makes these errors go away Adobe definitely has some clues to start with.

[pnico]

You could try https://github.com/freshplanet/ANE-AACPlayer, it provides events when sounds complete (it will play MP3s as well among others)

[cgascons]

Hey @pnico , thanks for the information, didn't know about this one, will definitely give it a try if I still get ANRs after using ANESounds. Cheers!

[NB13]

@gosgame can you share your working experience with Workers on mobile in general? I heard that it is very unstable feature. And we got problems with Android crashes on one project trying to improve loading time by parallel resource loading in workers.

[gsario]

Great news, guys! @chichilatte, @cgascons please keep all of us updated on results!

[mumeka]

I have implemented ANESounds yesterday and removed all Sound objects from AS3 code, I can confirm all AudioTrack related ANRs and Crashes disappeared.

[dcampano]

One thing that I have noticed now that I'm looking at sound related items is that almost every time a sound finishes playing using the native AS3 Sound class is I see the following line in my adb logs:

05-19 16:11:53.661 1884 2085 W AudioTrack: releaseBuffer() track 0xc9749b00 disabled due to previous underrun, restarting

I haven't gotten the sound library swapped out to use an ANE but just wanted to mention seeing this message.

[dcampano]

@Ankit-Adobe I just wanted to see if you had seen the updates to this thread and were aware of the recent findings regarding Sounds?

[gosgame]

@NB13 We use workers mainly for loading assets like textures and data files in android. (in IOS they are much slower) It decreases the loader overheads and also related ANRs (especially while recovering from a context loss). You can only share data as bytearrays between workers so you should split the loading and allocation tasks. Since each worker consumes fixed amound of memory, you should try to minimize the number of them. (we have just one for all loadings) On the other hand our workers seem to work stable and crash free. (after lots of fixes and optimizations of course)

[cgascons]

We can also confirm all our ANRs are gone after the ANESounds implementation.

[soccerob]

@Ankit-Adobe We'd love to hear from you guys regarding this issue as it seems we may have found the smoking gun. Can you acknowledge this as an issue? And if so, let us know if this is something that is fixable or if we should start looking at options through native extensions? I would personally prefer to see this fixed through AIR. But if there's no urgency on your end to fix it, I would love to know so I can start evaluating alternative solutions.

[gsario]

We did ANESounds implementation, but had problems on playing long sound files. Did anyone have similar issues?

[cgascons]

Hi @gevorg-gsar, we are only using ANESounds for short length sounds so can't actually answer that. We have attempted to keep using the "old way" to play our soundtracks (1 to 3 minute tracks) but sadly we still get some ANRs. So we will probably look for alternatives such as the one commented by @pnico for our soundtrack files.

[denmorgenshtern]

Great news! Any rumors from adobe about solve this problem on AIR level? Truly, in all our apps top 5 ANR caused _ZN19AJAudioTrackWrapper11deleteTrackEv+70. We have a small amount of as3-played sounds and we can easily implement it on ANE-Sounds. But main sounds in app - it is playback via NetStream. What about it, is NetStream sounds causing ANR?

[gsario]

Hi guys, We got a strange ANR on Android. Here is the link to the log: https://pastebin.com/V5AtugAx Anyone can have a clue where this leads to?

[gsario]

@pnico @cgascons one of the issues with ANE-AACPlayer is that on pressing sound button on the phone, the native sound gauge popup is shown. Another inconvenience is not supporting looped sounds.

Also, @marchbold promised a new ANE for playing sounds natively. Looking forward to use it as DIstriqts ANE-s are very high quality!

[soccerob]

Where did marchbold mention they would be releasing an ANE for native sounds? Any time frame on that?

Also wanna ping @Ankit-Adobe as we would all love to hear something, anything regarding this issue.

[cgascons]

Hey @soccerob he commented something about that here.

I'd also really like to hear about @Ankit-Adobe anytime soon about this.

[PrimaryFeather]

I directly contacted Adobe about the status last week, and they told me they are still actively investigating, but still don't have anything conclusive. 😐 If I hear anything more, I'll post it here!

[gilil12]

Hey, we have an app with 80K android daily users. our matrices show the following :

ANR - 2.06%

Broadcast of Intent { act=android.intent.action.SCREEN_OFF flg=0x50000010 (has extras) } main thread : pc 00000000005a604b /data/app/air.com.beachbumgammon-2/lib/arm/libCore.so (_ZN19AJAudioTrackWrapper11deleteTrackEv+70)

Input dispatching timed out (Waiting to send non-key event because the touched window has not finished processing certain input events that were delivered to it over 500.0ms ago. Wait queue length: 7. Wait queue head age: 10961.8ms.)

2 pc 00000000005a604b /data/app/air.com.beachbumgammon-2/lib/arm/libCore.so (_ZN19AJAudioTrackWrapper11deleteTrackEv+70)

Broadcast of Intent { act=android.intent.action.SCREEN_ON flg=0x50000010 }

2 pc 00000000005a604b /data/app/air.com.beachbumgammon-1/lib/arm/libCore.so (AJAudioTrackWrapper::deleteTrack()+70)

The above is 90% from the ANR we experience, all related to AudioTrackWrapper::deleteTrack()

I will wait for Distriqt (marchbold) to finish their MediaPlayer ANE and i will transfer all of my sounds to use the ANE.

[marchbold]

@soccerob @cgascons We have this ANE in development, it will be an addition to the Media Player ANE and planned to release in the coming weeks.

[gsario]

Guys, we have implemented https://github.com/freshplanet/ANE-AACPlayer for playing music and sounds and randomly any playing of sounds throws such exception on play: com.google.android.exoplayer.audio.AudioTrack$InitializationException: AudioTrack init failed: 0, Config(44100, 4, 24576)

Any ideas? Is FreshPlanet doing ANE-s still?

Also, @marchbold we are looking forward to your extension very much!

[viveknegi1]

Hi Everyone,

I am Vivek from Adobe. Can you guys try these SDK to compile your apps and Run . https://adobe.ly/2Jg9Jl7 ( AIR SDK for MAC ) https://adobe.ly/2LpIaT2 (AIR SDK for Windows)

Please let us know if it fixes the problem.

Thanks.

[gsario]

Wow! Adobe just did respond to a major issue! I can't believe my eyes!

Thank you @viveknegi1!

[gilil12]

Hi Vivek,

Thanks for the quick response!! There are no release notes for the new version(31) can you please explain what was changed from AIR 29.0.0.122

10x

[rewb0rn]

Hi, nice, thanks for the release and the fix! I assume this is a preview for the next Air SDK 31 beta, correct? Does this include the latest changes from Air 30.0.0.107?

Thanks

[neuronix]

Thank you @viveknegi1 ! :)

[viveknegi1]

Just to let you know, we want your help to identify if the problem is fixed. Please try this SDK/Runtime and share your feedback.

[smileygamer]

I updated one of our games with AIR 31 and deployed it to 5% of players. There's very little feedback yet, but for now I don't have ANR's pointing to _ZN19AJAudioTrackWrapper11deleteTrackEv.

However, now I have new crashes:

signal 11 (SIGSEGV), code 1 (SEGV_MAPERR) _ZN19AJAudioTrackWrapper13AudioCallbackEv

signal 11 (SIGSEGV), code 1 (SEGV_MAPERR) _ZN19AJAudioTrackWrapper17AudioCallbackPollEv

signal 11 (SIGSEGV), code 1 (SEGV_MAPERR) _ZN15AndroidSoundMix13AudioCallbackEiP14ANPAudioBuffer

Can it be the ANR's are now replaced by crashes with the same cause?

This week we received an email from Google saying our apps were flagged for being significantly above the “bad behavior threshold” for select core vitals. We have between 3% and 5% ANR's in our games, so we really look forward to a solution!

In fact we were just about to start porting them to Unity, as staying on AIR has become a serious risk to our business...

[Elintondm]

The same here. We have 28 air games and the anr rate is between 2% and 4%. We already ported 2 of them to unity and the ported games have anr rate below 0.08% !!!! So is comproved that air games have problems with anr.

We would love if adobe fix this problem. We still like make games with adobe air. But is impossible with this unacceptable anr rate.

Em sex, 8 de jun de 2018 07:18, smileygamer notifications@github.com escreveu:

I updated one of our games with AIR 31 and deployed it to 5% of players. There's very little feedback yet, but for now I don't have ANR's pointing to _ZN19AJAudioTrackWrapper11deleteTrackEv.

However, now I have new crashes:

signal 11 (SIGSEGV), code 1 (SEGV_MAPERR) _ZN19AJAudioTrackWrapper13AudioCallbackEv

signal 11 (SIGSEGV), code 1 (SEGV_MAPERR) _ZN19AJAudioTrackWrapper17AudioCallbackPollEv

signal 11 (SIGSEGV), code 1 (SEGV_MAPERR) _ZN15AndroidSoundMix13AudioCallbackEiP14ANPAudioBuffer

Can it be the ANR's are now replaced by crashes with the same cause?

This week we received an email from Google saying our apps were flagged for being significantly above the “bad behavior threshold” for select core vitals. We have between 3% and 5% ANR's in our games, so we really look forward to a solution!

In fact we were just about to start porting them to Unity, as staying on AIR has become a serious risk to our business...

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/Gamua/Adobe-Runtime-Support/issues/29#issuecomment-395718036, or mute the thread https://github.com/notifications/unsubscribe-auth/AUWTYTw1R_F2uNh3S-s-JXQwLxYhRGIrks5t6k9ugaJpZM4QwWh4 .

[marchbold]

Just FYI for anyone waiting for our ANE update, it's now available in the beta test thread: https://github.com/distriqt/ANE-MediaPlayer/issues/95

Hoping the Adobe solution works though! Anyone have any feedback from the AIR SDK @viveknegi1 linked? Will be trying it with our next beta release but that's not for a few weeks.

[gsario]

@marchbold we think that your ANE will be necessary either Adobe fixes their SDK or not (it might take weeks/months). I personally like the idea of playing sounds using native resources. You should just take care that the load and dispose for sounds, the playback of multiple short sound instances and long sounds looping works correctly. There are problems with those in AACPlayer ANE.

[Megabyteceer]

I has same ANR rate about 2-3% in all my AIR games. Maybe Unity just avoid to launch game on unsupported devices?

[viveknegi1]

Hi All,

Thanks for the update. Can you guys share detail logs/dumps of the crashes that you are facing after using the SDK ( that I shared ).

[Megabyteceer]

I updated to regular AIR30 couple of days ago. 50% of all ANR have same type. Log for this most frequent crash is here: https://lm.pixel-cave.com/uploads/file/98/987144719509/anr_crash_log.txt