Input dispatching timed out ANRs & ANR rate survey

[neuronix]

Hi guys,

I am mirorring my post from the Starling forum here with Daniel's permission.

We have ANRs on Android in all our apps and would like to know if it's maybe our apps particularly, or if it's AIR related.

Could you please check your Google Dev Console and go to "Android Vitals" > "Overview".

The first section is "Bad Behaviors".

Please share the following values (no need to mention the app, just pick the biggest app you have or the average of all your apps please):

• ANR Rate (around 2.6% on our apps)
• Crash Rate (1.1-1.8% on our apps)

The main ANR we are experiencing is "Input dispatching timed out" triggered by "com.adobe.air.customHandler.callTimeoutFunction". There are heaps of topics on it on the web, and they only proper response by Adobe is that we should avoid doing heavy calculations on user input.

But all our apps use Starling, and Starling defers all user inputs to next frame so this should not be an issue.

[cgascons]

Hi Daniel,

Glad you heard back from Adobe, here's the summarized information from the last 7 days (right after the last update of one of our games)

1 Broadcast of Intent { act=android.intent.action.SCREEN_OFF flg=0x50000010 }, VisibleToUser

Device | Amount of reports | Percentage

Moto G (2nd Gen) (titan_udstv)  7   24,1%
Galaxy J2 (j2lte)   2   6,9%
LG Stylo 2 (ph1)    2   6,9%
Galaxy S5 mini (kminilte)   2   6,9%
Galaxy J1 Mini (j1mini3g)   1   3,4%
Galaxy S6 (zeroflte)    1   3,4%
Galaxy S6 Edge (zeroltevzw)     1   3,4%
Moto G (2nd Gen) (titan_umtsds)     1   3,4%
Galaxy A3 (a33g)    1   3,4%
Galaxy J1 Ace (j1acevelte)  1   3,4%
Galaxy J1(2016) (j1x3g)     1   3,4%
LG K10 (2017) (mlv5)    1   3,4%
Moto G(4) Plus (athene_f)   1   3,4%
Galaxy J7(2016) (j7xelte)   1   3,4%
Galaxy Grand Prime (fortuna3g)  1   3,4%
LG K4 LTE (me1)     1   3,4%
Grand Prime Plus (grandppltedtv)    1   3,4%
Galaxy Grand Prime (grandprimelte)  1   3,4%
Moto G (5th Gen) (cedric)   1   3,4%
Galaxy S4 (jaltektt)    1   3,4%

OS

Android 6.0     13  44,8%
Android 5.1     6   20,7%
Android 7.0     6   20,7%
Android 5.0     4   13,8%

Here's the full report for this ANR

2 Input dispatching timed out (Waiting because the focused window has not finished processing the input events that were previously delivered to it.)

Device | Amount of reports | Percentage

TR10RS1_1   9   31,0%
ZenFone C (ZC451CG) (ASUS_Z007)     6   20,7%
BGH Y230 (T760)     4   13,8%
MediaPad (hwt1701)  2   6,9%
CHC-U23 (hwCHC-H)   1   3,4%
hlteuc  1   3,4%
RCT6973W43 (RCT6973W43)     1   3,4%
LG K4 LTE (me1ds)   1   3,4%
V975 (redhookbay)   1   3,4%
hudl 2 (HTF8A4)     1   3,4%
Galaxy Tab3 10.1 (santos10wifi)     1   3,4%
ZenFone 5 (ASUS_T00J)   1   3,4%

OS

Android 4.4     27  93,1%
Android 5.1     1   3,4%
Android 6.0     1   3,4%

Here's the full report for this ANR

3 Input dispatching timed out (Waiting because the touched window has not finished processing the input events that were previously delivered to it.)

Device | Amount of reports | Percentage

Galaxy J1 Ace (j1pop3g)     19  82,6%
M7I-3G (astar-ococci)   2   8,7%
Galaxy J1 (j13g)    2   8,7%

OS Android 4.4 23 100,0%

Here's the full report for this ANR

4 Input dispatching timed out (Waiting to send non-key event because the touched window has not finished processing certain input events that were delivered to it over 500.0ms ago. Wait queue length: 47. Wait queue head age: 5523.3ms.)

Device | Amount of reports | Percentage

Redmi Note 3 (kenzo)    5   35,7%
Moto G (2nd Gen) (titan_udstv)  3   21,4%
Galaxy J2 (j2lte)   3   21,4%
Galaxy Grand Prime (grandprimelte)  1   7,1%
Moto G Plus (5th Gen) (potter_nt)   1   7,1%
5098S (Pixi4-6_4G)  1   7,1%

OS

Android 5.1     8   57,1%
Android 6.0     4   28,6%
Android 7.0     1   7,1%
Android 5.0     1   7,1%

Here's the full report for this ANR

5 Broadcast of Intent { act=android.intent.action.SCREEN_ON flg=0x50000010 }

Device | Amount of reports | Percentage

Galaxy Tab3 Lite (goyawifi)     3   23,1%
Galaxy Tab3 7.0 (lt02wifi)  2   15,4%
Galaxy Grand Prime (fortunave3g)    2   15,4%
Galaxy Tab4 7.0 (degaswifi)     2   15,4%
M7I-3G (astar-ococci)   1   7,7%
Galaxy Note2 (t0lteskt)     1   7,7%
LG G Pad 8.3 homeBoy (awifi070u)    1   7,7%
Galaxy Grand Prime (fortuna3g)  1   7,7%

OS

Android 4.4     9   69,2%
Android 4.2     4   30,8%

Here's the full report for this ANR

6 Broadcast of Intent { act=android.intent.action.SCREEN_ON flg=0x50000010 }

Device | Amount of reports | Percentage

Canvas Spark 3 (Q385)   3   27,3%
One (M8) (htc_m8)   2   18,2%
Galaxy S5 (klte)    1   9,1%
G2 MINI (g2m)   1   9,1%
Moto G (1st Gen) (falcon_umtsds)    1   9,1%
G2 MINI (g2mds)     1   9,1%
L-01F (g2)  1   9,1%
TOMMY (P4901AC)     1   9,1%

OS

Android 5.1     4   36,4%
Android 5.0     4   36,4%
Android 6.0     3   27,3%

Here's the full report for this ANR

7 Input dispatching timed out (Waiting to send non-key event because the touched window has not finished processing certain input events that were delivered to it over 500.0ms ago. Wait queue length: 4. Wait queue head age: 6469.8ms.)

Device | Amount of reports | Percentage

sofia3gr    4   66,7%
7 Voyager (RCT6873W42)  2   33,3%

OS Android 6.0 6 100,0%

Here's the full report for this ANR

I'm hoping this info helps them find the issue! Best, Chris

[cgascons]

Hi @PrimaryFeather, any news from the folks from Adobe regarding this? Was the information useful or they need anything else?

[PrimaryFeather]

Yes, apparently that was helpful!

We are looking into it on priority. I or my team will follow up on github for more info.

This is the message I received from Ankit.

[cgascons]

Thanks for the quick response, looking forward hearing back from them, then!

[zuludev]

@chichilatte asked me to post in details of our ANRs to assist @Ankit-Adobe (I've included info in same format as @cgascons) Summarised information from the last 7 days (after last update of our app)

(fyi ANR 15 includes a garbage collection error.)

[cgascons]

Just a friendly reminder to Adobe's people, any progress? Can't wait to hear about this.

[neuronix]

Also, quite eager to have this fixed. I'm pretty sure that if Google is putting more and more of an emphasis on application behavior by underlining crashes & ANRs and providing universal benchmarks, they are already or soon will be reducing visibility of poorly behaving apps (and rightfully so).

[cgascons]

They are already doing it: https://blog.waypedia.com/googles-new-punishment-poorly-made-apps-play-store/

[zuludev]

Very keen for this to be looked at by Adobe folk as we're sure our app's visibility is taking a hit.

[NB13]

The same issues

-Broadcast of Intent { act=android.intent.action.SCREEN_OFF flg=0x50000010 } -Input dispatching timed out (Waiting because the touched window has not finished processing the input events that were previously delivered to it.) -Input dispatching timed out (Waiting to send non-key event because the touched window has not finished processing certain input events that were delivered to it over 500.0ms ago. Wait queue length:

Details for Adobe folks

Top0.pdf Top1.pdf Top2.pdf Top3.pdf Top4.pdf

Having about 200k affected users and 2M ANRs for the last 7 days. Can provide you more and more details

[cgascons]

Hi @Ankit-Adobe , any news regarding this issue? Do you need any extra information, I'm sure there will be plenty of volunteers to provide anything you request to get this sorted out.

Cheers,

[laFunkhh]

Hey @Ankit-Adobe! yeah, anything you need to help sort this issue, just ask! My ANRs are sitting around 1.5-2%. Something that may be worth noting is Android OS 8.1 seems to be the most problematic (around 3-3.5% ANRs). Got reasons to believe the app visibility is being penalized for this! Would be awesome to fix it :)

Cheers!

[soccerob]

We are also being penalized for this. Our game is a top-rated Bingo game, and we can't get listed under the Casino -> Bingo Games category.

[chichilatte]

Can nobody reliably replicate this problem on a specific device? I'm sure @Ankit-Adobe wld welcome an apk/ipa like that.

[neuronix]

There is already a reproductible case : https://github.com/Gamua/Adobe-Runtime-Support/issues/29#issuecomment-355358060

Sorry to summon you yet again @PrimaryFeather could you please ask Adobe to look into this? We have provided the requested information and we're being ignored.

[PrimaryFeather]

I'll see what I can do, @neuronix! I totally agree with you guys, this should be a top priority.

[Rohitpoddar92]

Hey guys,

we have tried to investigate the issue and here are our finding.

var temp:Array = new Array(); 
for (var i:int = 0; i < 1000000; i++) { 
    var s:Sprite = new Sprite(); temp.push( s ); 
}

#29 (comment) is not of much a help. There are a million objects. A device can handle only a limited amount of objects. If you try to run a similar case in native android app, ANR will be thrown there as well.

Here is a table for comparison of native and AIR android apps: [These are just indicative results, might vary device to device]

We have spent quite some time to try to figure out root cause of this issue. Unfortunately, since we don't have a concrete use case to reproduce the issue at our end, it has come down to some guesses based on stack traces and code flow:

1. As mentioned by a few others, it might be a GC issue, which might have been caused by design flow in Action Script code of AIR apps, as described above.
2. It might be related to Runtime input handling code, which is possible but seems unlikely.
3. It might have been caused by Audio thread related Runtime code flow.

I think, to proceed further, we need a reproducible use case.

Thanks, Rohit AIR Adobe

[chichilatte]

Thanks @Rohitpoddar92. I think i might be able to get you a reproducible use case (which will produce an ANR within half an hour or so). Have been trying for ages to find the time to build a stripped down apk. Will try to get one to you in the next few days. What wld be the best way of sending you a link to the apk? Also, just to be clear: the AIR code above is firing an ANR because the temp array is being garbage collected? I mean, it could just be a million sprites is too many to create in one frame. Lots of apps create a million objects, but very rare +foolhardy that an app would create them in one frame.

[PrimaryFeather]

@Rohitpoddar92 The example with the long for loop is a little misleading. Soccerob writes that the loop itself causes one ANR — which is to be expected, since it takes so long. But he says there's another ANR when the Garbage Collector cleans up all those objects.

@chichilatte: It would be really helpful if you could get that sample (plus source) to Adobe, thanks! I'll ask them to post again in this thread.

[Rohitpoddar92]

@PrimaryFeather You are right, I should have been more specific with the later part of GC kicking off. I just wanted to convey that the long loop example is not too much of a help.

@chichilatte You can upload the app and source on dropbox and provide me a link.

[cgascons]

Hi @Rohitpoddar92

Sadly I'm unable to to create a reproducible case at all since it's 100% random on our end. I've tried many times to reproduce it and we just can't, but we keep on receiving ANR reports even after having updated our AIR SDK to the latest (AIR 29 v108), and all our ANE's to their most updated versions as well.

It's quite frustrating situation since (at least) we are just not able to create any reproducible project for you guys to test. I'd encourage any of the present members in this Issue to provide any information at all or sample project (if they manage to create one), even though I can imagine they'd have done it already if they did.

We will keep trying to find some repro steps and will update this thread if we succeed.

Thanks,

[NB13]

Hi to all. Recently I found out that usual file.deleteFile() (sync method not file.deleteFileAsync()) Can cause big lag and ANR on some cheap android devices.

[NB13]

Hi @Rohitpoddar92 . I think the main issue is not concrete ANRs but impossibility to find cause in every concrete case. We've tried to use ANRWatchDog and Crashlytics -but haven't got something useful, the most popular ANR native stacktrace:   | com.adobe.air.customHandler.handleMessage (customHandler.java:22)   | android.os.Handler.dispatchMessage (Handler.java:111)   | android.os.Looper.loop (Looper.java:194)   | android.app.ActivityThread.main (ActivityThread.java:5576)   | java.lang.reflect.Method.invoke (Method.java)   | java.lang.reflect.Method.invoke (Method.java:372)   | com.android.internal.os.ZygoteInit$MethodAndArgsCa

So, @Rohitpoddar92 can you provide some mechanism helping app developers to find cause? For instance you can add native api (maybe to frecontext) to get current as3 stacktrace. It's not solution for GC case, but something very useful in other cases.

[Rohitpoddar92]

@cgascons I can understand your frustration. We will also keep trying at our end to figure out a way to fix it.

@NB13, The code flow mentioned in stack trace is very common in AIR Android input handling. This is like a queueing mechanism for events. So, whenever we get any ANR with input events, this stack trace is likely to occur. As of now, we are still not sure how to reproduce it.

Regarding providing a native api, we will explore this request and get back to you.

Thanks, Rohit AIR Adobea

[neuronix]

We introduced calls to System.pauseForGCIfCollectionImminent(0); at several key points in our latest app a few versions ago and have seen a very significant reduction in ANRs from around 3% of impacted sessions before to 0.4% on average now. Luckily this app has pauses between input periods that we could exploit but it's not the case for all our apps which still suffer of high ANR rates.

I'd recommend trying to slip System.pauseForGCIfCollectionImminent(0); where ever you can to workaround the bug in the meantime.

[PrimaryFeather]

That's great news, @neuronix — very encouraging!

[pis0]

e.g. TouchProcessor.advanceTime... the "Input dispatching timed" shows up when you keep touching a device screen in a high CPU process situation ( debug mode use to help here), 'cause the "_queue" length increase at the same frame. ( we get the "Broadcast of Intent" ones like this as well. It's also a kind of input )

So, is there some problem in using an inelegant solution like this to avoid this issue?

static private const QUEUE_MAX_LENGTH:int = 30;
public function advanceTime(passedTime:Number):void{
   if (_queue.length >= QUEUE_MAX_LENGTH)   {
      cancelTouches();
      System.pauseForGCIfCollectionImminent(0);
      return;
   }
...

[PrimaryFeather]

That would definitely be worth a try. The question is how many input events will be created in such a situation, though — my guess would be that the runtime doesn't send many events, anyway, in such a high CPU situation, so it's unsure if we'd reach such a queue length. Perhaps somebody can try making this modification and log how often it's actually happening?

[cgascons]

Hi, we have tried calling System.pauseForGCIfCollectionImminent(0) as suggested by @neuronix but we didn't notice any significant change, actually we have even a little bit high ANR rate than before (3,66%). Will try @pis0 's workaround soon and will let you know about the results too.

[laFunkhh]

Hey all! Also tried the pauseForCGIfCollectionImminent(0) (used to have it at 0.25) and made another batch of optimizations, focusing on reducing the number of memory allocations/deallocations and adding a "staged" garbage collection along 2 frames. Still these two frames take around 100-200ms to process and I'm guessing that's where the ANRs happens?

After implementing these changes the ANRs dropped from 1.9% to 1.6% (maybe?) Seeing the history of builds it all feels really random honestly. The past 5 builds have between 0.83% to 2.31%

[neuronix]

@cgascons @laFunkhh Our game has quite a particular inputs/no inputs sequence so it was easy for us to time the GC calls properly. On another of our games with less distinct input phases, we still dropped from a 2.6% to a 1.8% ANR rate with a few GC calls.

It's still just a workaround, Adobe should definitely try and fix this.

[pis0]

we got the best results in our intern tests with these changes in TouchProcessor.as... ( please, be aware that we prefer to purge events than get "Input dispatching timed" ANRs, even if that means interrupting <began/moved/ended> cycles.. )

private const QUEUE_MAX_LENGTH:int = 8;
private const MAX_PASSED_TIME:Number = 0.3;      

private var ignoreEvents:Boolean = false;

public function advanceTime(passedTime:Number):void{
    ignoreEvents = passedTime > MAX_PASSED_TIME;
    while (_queue.length > QUEUE_MAX_LENGTH) _queue.pop();
    ...
}

protected function processTouches(touches:Vector.<Touch>, shiftDown:Boolean, ctrlDown:Boolean):void{
     ...
    for each (var touchData:Object in sHoveringTouchData)
        if (touchData.touch.target != touchData.target)
            if (!ignoreEvents) _touchEvent.dispatch(touchData.bubbleChain);
       ...
    for each (touch in touches)
        if (!ignoreEvents) touch.dispatchEvent(_touchEvent);
    ...
}

It's an useful workaround to decrease these ANRs, but I have a keen interest in a better solution :)

[gsario]

Hi guys, the situation with ANR-s is quite serious. We have the same issues.

Crashes: 0.76% ANR-s: 2.38%

• Broadcast of Intent { act=android.intent.action.SCREEN_OFF flg=0x50000010 }
• Input dispatching timed out (Waiting to send non-key event because the touched...

Let's keep actively working together on solving these.

Did anyone else try @pis0 's solution? Did it help with "Input dispatching timed out"?

[chichilatte]

Hi folks. Possible breakthrough? We've just noticed that Google Play Console ANR/crash reports are now showing more info in the traces. The lines which used to say... /data/app/{APP_ID}/lib/arm/libCore.so (???) Now all say things like... /data/app/{APP_ID}/lib/arm/libCore.so (_ZN19AJAudioTrackWrapper11deleteTrackEv+70)

We updated our app to AIR 28 recently, which may be providing more handy info to Google.

Here's the full trace we're getting, same for almost all our "Input dispatching timed out" and "Broadcast of intent ... SCREEN_OFF" ANRs:

"main" prio=5 tid=1 Native
  | group="main" sCount=1 dsCount=0 obj=0x7470b390 self=0xb85208f0
  | sysTid=26165 nice=0 cgrp=default sched=0/0 handle=0xb6f8bc00
  | state=S schedstat=( 0 0 0 ) utm=8498 stm=1217 core=3 HZ=100
  | stack=0xbe6c8000-0xbe6ca000 stackSize=8MB
  | held mutexes=
  #00  pc 00000000000168ac  /system/lib/libc.so (syscall+28)
  #01  pc 0000000000041aed  /system/lib/libc.so (pthread_join+124)
  #02  pc 00000000005df623  /data/app/{APP_ID}/lib/arm/libCore.so (_ZN19AJAudioTrackWrapper11deleteTrackEv+70)
  #03  pc 00000000005df363  /data/app/{APP_ID}/lib/arm/libCore.so (_ZN19AJAudioTrackWrapperD2Ev+18)
  #04  pc 00000000005bea23  /data/app/{APP_ID}/lib/arm/libCore.so (???)
  #05  pc 00000000005ba821  /data/app/{APP_ID}/lib/arm/libCore.so (_ZN15AndroidSoundMix23PlatformCloseDeviceImplEb+40)
  #06  pc 00000000006c0071  /data/app/{APP_ID}/lib/arm/libCore.so (_ZN8SoundMix11CloseDeviceEb+40)
  #07  pc 00000000007292fb  /data/app/{APP_ID}/lib/arm/libCore.so (_ZN10CorePlayer11DoPlay_IdleEb+1686)
  #08  pc 000000000072bbab  /data/app/{APP_ID}/lib/arm/libCore.so (_ZN10CorePlayer6DoPlayEbb+1006)
  #09  pc 000000000054c6b9  /data/app/{APP_ID}/lib/arm/libCore.so (_ZN12CommonPlayer7OnTimerEv+360)
  #10  pc 000000000054f909  /data/app/{APP_ID}/lib/arm/libCore.so (_ZN14PlatformPlayer20AndroidTimerCallbackEPv+48)
  #11  pc 000000000055290d  /data/app/{APP_ID}/lib/arm/libCore.so (_Z19UnixTimeoutCallbackj+36)
  #12  pc 00000000000eaee9  /system/lib/libart.so (art_quick_generic_jni_trampoline+40)
  #13  pc 00000000000e67f1  /system/lib/libart.so (art_quick_invoke_stub_internal+64)
  #14  pc 00000000003e9ddf  /system/lib/libart.so (art_quick_invoke_stub+170)
  #15  pc 00000000007fdb8c  [stack] (???)
  at com.adobe.air.customHandler.callTimeoutFunction (Native method)
  at com.adobe.air.customHandler.handleMessage (customHandler.java:22)
  at android.os.Handler.dispatchMessage (Handler.java:102)
  at android.os.Looper.loop (Looper.java:148)
  at android.app.ActivityThread.main (ActivityThread.java:5452)
  at java.lang.reflect.Method.invoke! (Native method)
  at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run (ZygoteInit.java:762)
  at com.android.internal.os.ZygoteInit.main (ZygoteInit.java:652)

I see David Oliva reported something very similar in the Adobe bug tracker a few months ago.

Also of interest, one of our common crashes (25% of them), "signal 11 (SIGSEGV), code 1 (SEGV_MAPERR)" has something similar in the trace now...

  #01  pc 00000000005dfd1d  /data/app/{APP_ID}/lib/arm/libCore.so (_ZN19AJAudioTrackWrapper13AudioCallbackEv+44)
  #02  pc 00000000005dfcdf  /data/app/{APP_ID}/lib/arm/libCore.so (_ZN19AJAudioTrackWrapper17AudioCallbackPollEv+70)
  #03  pc 00000000005df2c1  /data/app/{APP_ID}/lib/arm/libCore.so (_ZN19AJAudioTrackWrapper18AudioCallbackEntryEPv+96)
  #04  pc 0000000000047f93  /system/lib/libc.so (_ZL15__pthread_startPv+22)
  #05  pc 000000000001a161  /system/lib/libc.so (__start_thread+6)

For us at least it looks like an audio bug. Is everyone experiencing this using audio?

My wild hunch: Maybe a synchronous file delete of a temporary audio file is causing lag.

[mumeka]

Input dispatching timed out (Waiting to send key event because the focused window has not finished processing all of the input events that were previously delivered to it. Outbound queue length: 0. Wait queue length: 2.)

"main" tid=1 Native
"main" prio=5 tid=1 Native
  | group="main" sCount=1 dsCount=0 obj=0x772ac000 self=0xb4427800
  | sysTid=2508 nice=-4 cgrp=default sched=0/0 handle=0xb6ff8bec
  | state=S schedstat=( 0 0 0 ) utm=272708 stm=13487 core=3 HZ=100
  | stack=0xbe09c000-0xbe09e000 stackSize=8MB
  | held mutexes=

  #00  pc 000000000001341c  /system/lib/libc.so (syscall+28)

  #01  pc 0000000000017be3  /system/lib/libc.so (pthread_join+90)

  #02  pc 00000000005a604b  /data/app/air.com.com.g.p-1/lib/arm/libCore.so (_ZN19AJAudioTrackWrapper11deleteTrackEv+70)

  #03  pc 00000000005a5d8b  /data/app/air.com.com.g.p-1/lib/arm/libCore.so (_ZN19AJAudioTrackWrapperD2Ev+18)

  #04  pc 000000000058545b  /data/app/air.com.com.g.p-1/lib/arm/libCore.so (???)

  #05  pc 0000000000581259  /data/app/air.com.com.g.p-1/lib/arm/libCore.so (_ZN15AndroidSoundMix23PlatformCloseDeviceImplEb+40)

  #06  pc 0000000000686b09  /data/app/air.com.com.g.p-1/lib/arm/libCore.so (_ZN8SoundMix11CloseDeviceEb+40)

  #07  pc 00000000006efd8d  /data/app/air.com.com.g.p-1/lib/arm/libCore.so (_ZN10CorePlayer11DoPlay_IdleEb+1760)

  #08  pc 00000000006f263b  /data/app/air.com.com.g.p-1/lib/arm/libCore.so (_ZN10CorePlayer6DoPlayEbb+1006)

  #09  pc 0000000000518db9  /data/app/air.com.com.g.p-1/lib/arm/libCore.so (_ZN12CommonPlayer7OnTimerEv+360)

  #10  pc 000000000051c009  /data/app/air.com.com.g.p-1/lib/arm/libCore.so (_ZN14PlatformPlayer20AndroidTimerCallbackEPv+48)

  #11  pc 000000000051ee91  /data/app/air.com.com.g.p-1/lib/arm/libCore.so (_Z19UnixTimeoutCallbackj+36)

  #12  pc 0000000000210297  /data/dalvik-cache/arm/data@app@air.com.com.g.p-1@base.apk@classes.dex (Java_com_adobe_air_customHandler_callTimeoutFunction__II+90)

  at com.adobe.air.customHandler.callTimeoutFunction (Native method)

  at com.adobe.air.customHandler.handleMessage (customHandler.java:22)

  at android.os.Handler.dispatchMessage (Handler.java:102)

  at android.os.Looper.loop (Looper.java:145)

  at android.app.ActivityThread.main (ActivityThread.java:6946)

  at java.lang.reflect.Method.invoke! (Native method)

  at java.lang.reflect.Method.invoke (Method.java:372)

  at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run (ZygoteInit.java:1404)

  at com.android.internal.os.ZygoteInit.main (ZygoteInit.java:1199)

[chichilatte]

Great to have more details @mumeka! I'm getting a sore arm scrolling this page on my phone tho. If anyone does have reams of crash reports do chuck them into a gist or something xx

[chichilatte]

@mumeka Also, is that one big ANR report you posted, starting with the line containing... _ZN19AJAudioTrackWrapper11deleteTrackEv+70?

I'm curious what percentage of your ANRs are like this.

[mumeka]

Yes one single big ANR with all lines. There are hundreds more, occurring at same place with your ANR. My game uses simple 5 sound effects, card game. So how it is related to Sound i don't know. %46 of my errors belong to this ANR. My ANR ratio is %1,21 and bad behaviour limit is at %0,47. Adobe needs to do something, app gets less visibility at searches due to ANRs.

[chichilatte]

Hi @Rohitpoddar92, I wonder if you have on opinion on the new details in our ANR reports?

[cgascons]

Hi there, after testing around a couple of days with @pis0 (2nd) workaround, we are receiving lots of #1502 errors on TouchProcessor/advanceTime() code. It looks like one of the while loops is problematic. I've read the code over and over and can't really find why would that happen, especially now that the queue is getting lighter in the first chunk of the workaround. It's like there's at some point something that is adding touch events to the queue unceasingly. Also, getting a few instances of the same error on Starling/onTouch() and VertexDataFormat/getAttribute() even though I highly doubt the last one is related.

Anyone else has experienced this?

[cgascons]

Also, as a side note, I think it performs better (and we actually get rid of the #1502 errors in TouchProcessor/advanceTime() if we actually prevent the queue from being populated with the events, rather than looping the queue and popping the events out. So, we removed this loop from TouchProcessor/advanceTime():

while (_queue.length > QUEUE_MAX_LENGTH)
   _queue.pop();

And modified TouchProcessor/enqueue() as follows:

/** Enqueues a new touch our mouse event with the given properties. */
public function enqueue(touchID:int, phase:String, globalX:Number, globalY:Number, pressure:Number=1.0, width:Number=1.0, height:Number=1.0):void
{
    if(_queue.length >= QUEUE_MAX_LENGTH)
        return;

         _queue.unshift(arguments);

         // multitouch simulation (only with mouse)
         if (_ctrlDown && _touchMarker && touchID == 0)
         {
                _touchMarker.moveMarker(globalX, globalY, _shiftDown);
                _queue.unshift([1, phase, _touchMarker.mockX, _touchMarker.mockY]);
            }
}

We expect receiving less (or none) #1502 erorrs due to the deletion of the while loop added recently. Any thoughts?

[gsario]

@cgascons can you please share the source code of your changes? If your changes are on the latest Starling version, we can try to build our game with those and check if that fix reduces the ANR-s.

[cgascons]

Hi @gevorg-gsar, the changes we did to the code are detailed on my last post in this issue, we didn't change anything else. Anyways, let me share the statistics of the errors with this recently changed code in my production app:

• ANR errors: 2,96%
• Multiple ANR errors: 0,81%

As you can see, this is still unacceptable and beyond the limits Google Play consider acceptable so we are still suffering the consequences from the low visibility.

[chichilatte]

@cgascons I just noticed that in the 5 ANR reports that you posted back in January, three of them have a very prominent line mentioning audio track deletion, e.g. _ZN19AJAudioTrackWrapper11deleteTrackEv+70. Same line we're getting. Can i ask, does your app use much sound + are you using Starling's AssetManager.playSound()?

EDIT: I see it's called Monster Battles, it's gonna have plenty of sound surely!

[cgascons]

Hey @chichilatte Indeed, it has lots of sounds :)

We are not using AssetManager to play them though, we are doing it "the old way". In fact, until now we didn't even know that could be done via AssetManager, at least on my current Starling Version (2.3) that function doesn't even exist. Are you guys using it?

[chichilatte]

Yep we're using it (we're still on Starling 1.8). I just checked Starling 2.3 on github – looks like playSound() is still in there. Useful to know that we're both getting ANRs despite using two different ways to play sounds.

When i get a mo I'll try to replicate the ANRs on a version of our app with sounds turned off.

[cgascons]

Hey @chichilatte, it would definitely be nice if you have the chance to actually not being able to replicate it with the sounds turned off, that would mean we could have definitely found the very source of the issue. Fingers crossed!

[cgascons]

By the way, today I've been Googling around and found some quite old posts about (I believe) this issue, some of them are from 2013, I even found a post of myself from 3 years ago which I didn't remember I had written. All of them have one thing in common, the AudioSource traces. I'm not saying that really has to be the issue, but it's difficult to think otherwise by looking at so many traces pointing to the same. Perhaps taking a look at that gives us some clues @Ankit-Adobe

These are the links I found:

https://forums.adobe.com/thread/1283716 https://forum.starling-framework.org/topic/keydispatchingtimedout-anr-on-android https://forums.adobe.com/thread/1080235 https://stackoverflow.com/questions/25627608/anr-android-keydispatchingtimedout-air-4-0

I'm sorry for being so insistent, I just really want to get rid of these ANRs real bad.

[NB13]

Hi to all, we've got quite many ANR in new game with small audience (2-3k users) - main problem was _ZN19AJAudioTrackWrapper11deleteTrackE In new releases we've changed AS3 sounds to Android MediaPlayer - https://developer.android.com/reference/android/media/MediaPlayer

As I can see last two weeks - all ANRs of this type disappeared. Unfortunately I can not see ANR rate in Google Play Console, too small dataset I think. We will try this fix in old project with 150k dau in a week or two maybe.

[soccerob]

How are you calling the Android MediaPlayer from as3? Are you using a native extension?