Open mohossama opened 8 months ago
Hi
Looking at the backtrace, this is a crash happening during the garbage collection process. But the nature of that process is such that we can't tell what object is being cleaned up.. it's likely that the error happens where an object is being processed which still has a hanging reference open to an object which itself had been cleaned up previously...
It was quite some jump between versions there. Can I check whether anything else changed or is the runtime update the only change that was made?
Also just wondering whether you use any ANEs, and in particular whether you maintain any references to ActionScript objects from within the extension objects (within the C or Java code, if you are writing these yourselves)?
If you have any other crash logs it may be useful, sometimes we'd spot the object that's being corrupted if the crash is timing-related.
thanks
Hi thanks for the followup
Can I check whether anything else changed or is the runtime update the only change that was made?
We needed to update some ANEs with the Air runtime update to 50.x. Here's a list of the ones that were updated, just in case anything rings a bell. They were mostly native dependency updates as far as I can tell:
extensionID>com.appsflyer.adobeair</extensionID>
<extensionID>com.ironsource.adobeair</extensionID>
<extensionID>com.ironsource.adobeair.admob</extensionID>
<extensionID>com.ironsource.adobeair.facebook</extensionID>
<extensionID>com.ironsource.adobeair.fbaudiencenetwork</extensionID>
<extensionID>com.ironsource.adobeair.unityads</extensionID>
<extensionID>com.ironsource.adobeair.unityadssdk</extensionID>
<extensionID>com.ironsource.adobeair.applovin</extensionID>
<extensionID>com.ironsource.adobeair.applovinsdk</extensionID>
in particular whether you maintain any references to ActionScript objects from within the extension objects (within the C or Java code, if you are writing these yourselves)?
We do maintain a few ANEs that we also touched with this update. I don't think we introduced new AS3 code in them, but we can review them. I just want to make sure I understood correctly what is meant by maintaining a reference to AS object from within the native code (Java in our case). Do you mean like taking the FREOject
from a function parameter and keeping that reference somewhere? Does retaining a reference of the FREContext
object in the Java code also count?
If you have any other crash logs it may be useful
Here are some more samples that look different that were correlated to the same crash by GooglePlay. Is that what you meant?
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 6660 >>> air.com.... <<<
backtrace:
#00 pc 0x0000000000868d98 /data/app/~~nhVtyb9EFxkbwLnZnSdu-A==/air.com....-b-XQcELpna0ScQGmMLLbsg==/lib/arm64/libCore.so (BuildId: 0941d34cf16b317780fd8fcb701e876875cc8447)
#01 pc 0x00000000008664a4 /data/app/~~nhVtyb9EFxkbwLnZnSdu-A==/air.com....-b-XQcELpna0ScQGmMLLbsg==/lib/arm64/libCore.so (BuildId: 0941d34cf16b317780fd8fcb701e876875cc8447)
#02 pc 0x00000000007bed64 /data/app/~~nhVtyb9EFxkbwLnZnSdu-A==/air.com....-b-XQcELpna0ScQGmMLLbsg==/lib/arm64/libCore.so (BuildId: 0941d34cf16b317780fd8fcb701e876875cc8447)
#03 pc 0x000000000086610c /data/app/~~nhVtyb9EFxkbwLnZnSdu-A==/air.com....-b-XQcELpna0ScQGmMLLbsg==/lib/arm64/libCore.so (BuildId: 0941d34cf16b317780fd8fcb701e876875cc8447)
#04 pc 0x0000000000865518 /data/app/~~nhVtyb9EFxkbwLnZnSdu-A==/air.com....-b-XQcELpna0ScQGmMLLbsg==/lib/arm64/libCore.so (BuildId: 0941d34cf16b317780fd8fcb701e876875cc8447)
#05 pc 0x0000000000865634 /data/app/~~nhVtyb9EFxkbwLnZnSdu-A==/air.com....-b-XQcELpna0ScQGmMLLbsg==/lib/arm64/libCore.so (BuildId: 0941d34cf16b317780fd8fcb701e876875cc8447)
#06 pc 0x00000000008641b4 /data/app/~~nhVtyb9EFxkbwLnZnSdu-A==/air.com....-b-XQcELpna0ScQGmMLLbsg==/lib/arm64/libCore.so (BuildId: 0941d34cf16b317780fd8fcb701e876875cc8447)
#07 pc 0x0000000000863f78 /data/app/~~nhVtyb9EFxkbwLnZnSdu-A==/air.com....-b-XQcELpna0ScQGmMLLbsg==/lib/arm64/libCore.so (BuildId: 0941d34cf16b317780fd8fcb701e876875cc8447)
#08 pc 0x0000000000871250 /data/app/~~nhVtyb9EFxkbwLnZnSdu-A==/air.com....-b-XQcELpna0ScQGmMLLbsg==/lib/arm64/libCore.so (BuildId: 0941d34cf16b317780fd8fcb701e876875cc8447)
#09 pc 0x0000000000785f88 /data/app/~~nhVtyb9EFxkbwLnZnSdu-A==/air.com....-b-XQcELpna0ScQGmMLLbsg==/lib/arm64/libCore.so (BuildId: 0941d34cf16b317780fd8fcb701e876875cc8447)
#10 pc 0x0000000000771b48 /data/app/~~nhVtyb9EFxkbwLnZnSdu-A==/air.com....-b-XQcELpna0ScQGmMLLbsg==/lib/arm64/libCore.so (BuildId: 0941d34cf16b317780fd8fcb701e876875cc8447)
#11 pc 0x000000000040a810 /data/app/~~nhVtyb9EFxkbwLnZnSdu-A==/air.com....-b-XQcELpna0ScQGmMLLbsg==/lib/arm64/libCore.so (BuildId: 0941d34cf16b317780fd8fcb701e876875cc8447)
#12 pc 0x000000000028be58 /data/app/~~nhVtyb9EFxkbwLnZnSdu-A==/air.com....-b-XQcELpna0ScQGmMLLbsg==/lib/arm64/libCore.so (BuildId: 0941d34cf16b317780fd8fcb701e876875cc8447)
#13 pc 0x000000000028bdf4 /data/app/~~nhVtyb9EFxkbwLnZnSdu-A==/air.com....-b-XQcELpna0ScQGmMLLbsg==/lib/arm64/libCore.so (BuildId: 0941d34cf16b317780fd8fcb701e876875cc8447)
#14 pc 0x000000000028b4cc /data/app/~~nhVtyb9EFxkbwLnZnSdu-A==/air.com....-b-XQcELpna0ScQGmMLLbsg==/lib/arm64/libCore.so (BuildId: 0941d34cf16b317780fd8fcb701e876875cc8447)
#15 pc 0x000000000028b53c /data/app/~~nhVtyb9EFxkbwLnZnSdu-A==/air.com....-b-XQcELpna0ScQGmMLLbsg==/lib/arm64/libCore.so (BuildId: 0941d34cf16b317780fd8fcb701e876875cc8447)
#16 pc 0x000000000028b270 /data/app/~~nhVtyb9EFxkbwLnZnSdu-A==/air.com....-b-XQcELpna0ScQGmMLLbsg==/lib/arm64/libCore.so (BuildId: 0941d34cf16b317780fd8fcb701e876875cc8447)
#17 pc 0x00000000000c163c /apex/com.android.runtime/lib64/bionic/libc.so (__pthread_start(void*)+204)
#18 pc 0x0000000000054930 /apex/com.android.runtime/lib64/bionic/libc.so (__start_thread+64)
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 8172 >>> air.com.... <<<
backtrace:
#00 pc 0x0000000001372708 /data/app/~~6_Hihc-gYiAm3y1ZZth70A==/air.com....-9lTosdzsD9i4oDLAwz4zow==/lib/arm64/libCore.so (BuildId: 0941d34cf16b317780fd8fcb701e876875cc8447)
#01 pc 0x000000000086fe04 /data/app/~~6_Hihc-gYiAm3y1ZZth70A==/air.com....-9lTosdzsD9i4oDLAwz4zow==/lib/arm64/libCore.so (BuildId: 0941d34cf16b317780fd8fcb701e876875cc8447)
#02 pc 0x0000000000867b1c /data/app/~~6_Hihc-gYiAm3y1ZZth70A==/air.com....-9lTosdzsD9i4oDLAwz4zow==/lib/arm64/libCore.so (BuildId: 0941d34cf16b317780fd8fcb701e876875cc8447)
#03 pc 0x00000000008678e8 /data/app/~~6_Hihc-gYiAm3y1ZZth70A==/air.com....-9lTosdzsD9i4oDLAwz4zow==/lib/arm64/libCore.so (BuildId: 0941d34cf16b317780fd8fcb701e876875cc8447)
#04 pc 0x0000000000867800 /data/app/~~6_Hihc-gYiAm3y1ZZth70A==/air.com....-9lTosdzsD9i4oDLAwz4zow==/lib/arm64/libCore.so (BuildId: 0941d34cf16b317780fd8fcb701e876875cc8447)
#05 pc 0x00000000002789e8 /data/app/~~6_Hihc-gYiAm3y1ZZth70A==/air.com....-9lTosdzsD9i4oDLAwz4zow==/lib/arm64/libCore.so (BuildId: 0941d34cf16b317780fd8fcb701e876875cc8447)
#06 pc 0x000000000027c228 /data/app/~~6_Hihc-gYiAm3y1ZZth70A==/air.com....-9lTosdzsD9i4oDLAwz4zow==/lib/arm64/libCore.so (BuildId: 0941d34cf16b317780fd8fcb701e876875cc8447)
#07 pc 0x000000000027f698 /data/app/~~6_Hihc-gYiAm3y1ZZth70A==/air.com....-9lTosdzsD9i4oDLAwz4zow==/lib/arm64/libCore.so (BuildId: 0941d34cf16b317780fd8fcb701e876875cc8447)
#08 pc 0x000000000004a084 /data/app/~~6_Hihc-gYiAm3y1ZZth70A==/air.com....-9lTosdzsD9i4oDLAwz4zow==/oat/arm64/base.odex (art_jni_trampoline+116)
#09 pc 0x00000000005ba6b0 /apex/com.android.art/lib64/libart.so (nterp_helper+4016)
#10 pc 0x000000000036197c /data/app/~~6_Hihc-gYiAm3y1ZZth70A==/air.com....-9lTosdzsD9i4oDLAwz4zow==/oat/arm64/base.vdex (com.adobe.air.customHandler.handleMessage+20)
#11 pc 0x0000000000a336f8 /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.oat (android.os.Handler.dispatchMessage+168)
#12 pc 0x0000000000a36f98 /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.oat (android.os.Looper.loopOnce+1000)
#13 pc 0x0000000000a36b08 /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.oat (android.os.Looper.loop+1112)
#14 pc 0x0000000000771210 /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.oat (android.app.ActivityThread.main+2432)
#15 pc 0x000000000033f080 /apex/com.android.art/lib64/libart.so (art_quick_invoke_static_stub+640)
#16 pc 0x00000000003884dc /apex/com.android.art/lib64/libart.so (_jobject* art::InvokeMethod<(art::PointerSize)8>(art::ScopedObjectAccessAlreadyRunnable const&, _jobject*, _jobject*, _jobject*, unsigned long)+1588)
#17 pc 0x0000000000387e98 /apex/com.android.art/lib64/libart.so (art::Method_invoke(_JNIEnv*, _jobject*, _jobject*, _jobjectArray*) (.__uniq.165753521025965369065708152063621506277)+32)
#18 pc 0x00000000003336a8 /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.oat (art_jni_trampoline+120)
#19 pc 0x00000000009a9048 /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.oat (com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run+136)
#20 pc 0x00000000009b320c /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.oat (com.android.internal.os.ZygoteInit.main+3548)
#21 pc 0x000000000033f080 /apex/com.android.art/lib64/libart.so (art_quick_invoke_static_stub+640)
#22 pc 0x00000000004e1ea8 /apex/com.android.art/lib64/libart.so (art::JValue art::InvokeWithVarArgs<_jmethodID*>(art::ScopedObjectAccessAlreadyRunnable const&, _jobject*, _jmethodID*, std::__va_list)+728)
#23 pc 0x000000000057b930 /apex/com.android.art/lib64/libart.so (art::JNI<true>::CallStaticVoidMethodV(_JNIEnv*, _jclass*, _jmethodID*, std::__va_list)+156)
#24 pc 0x00000000000bead0 /system/lib64/libandroid_runtime.so (_JNIEnv::CallStaticVoidMethod(_jclass*, _jmethodID*, ...)+120)
#25 pc 0x00000000000cadf8 /system/lib64/libandroid_runtime.so (android::AndroidRuntime::start(char const*, android::Vector<android::String8> const&, bool)+852)
#26 pc 0x0000000000002568 /system/bin/app_process64 (main+1300)
#27 pc 0x0000000000083198 /apex/com.android.runtime/lib64/bionic/libc.so (__libc_init+96)
Please let me know if you need more info
@ajwfrost sorry for the direct mention, not my intention to spam you, but this question from my last comment is quite important and I fear it may have gotten lost in the wall of text. It would really help our investigation if you can confirm what you meant by "maintain references to ActionScript objects from within the extension objects"
in particular whether you maintain any references to ActionScript objects from within the extension objects (within the C or Java code, if you are writing these yourselves)?
[...] Do you mean like taking the FREOject from a function parameter and keeping that reference somewhere? Does retaining a reference of the FREContext object in the Java code also count?
Hi - thanks for the ping, I'd not actually seen that update earlier, got lost in the mass of notifications :-(
But yes, keeping any reference to an FREObject would be a bad idea, ActionScript references should only be used within the single function call (unless they're also pinned via an ActionScript property elsewhere). I think Adobe had tried to ensure that references got held open but I would still think it depends a bit on how/when the garbage collector works...
We can check those other crash logs to see if they give more information, thanks!
Thanks for the quick response! We will inspect the Java code in the ANEs we have the source code for to see if we can identify such reference-holding patterns.
Problem Description
Upgrading from Air SDK 33.1.1.575 to 50.2.4.1 caused our GooglePlay user-perceived crash rates to spike from <0.5% to ~8% and rising.
Which AIR SDK did you use? Air SDK 50.2.4.1. Building on windows.
Describe the affected environment: Android vs. iOS, all devices vs. just specific models. Android. No clear pattern of device models or Android versions being affected more than others. Both our 64-bit and 32-bit APKs seem to be similarly affected.
Can you reproduce it yourself, or did you get reports from other users? If so, how many users are affected (percentage)? We're unable to reproduce it locally.
What we have now is the backtrace from GooglePlay. They mostly look like this:
We cannot tell from the backtrace what possible behavior can be triggering this, so any hints to help reproduce it are welcome, if the backtrace tells you anything about that. We're also willing to help in other ways. e.g. if you can provide us (privately?) with debug symbols we can upload to GooglePlay to get clearer stack traces.