Open neuronix opened 4 years ago
ajwfrost
commented
4 years ago Hi - there may be a lot of different reasons for crashing with sigsegv but the above trace would likely allow us to eliminate at least that one! What build is this using?
Thanks
neuronix
commented
4 years ago Hi Andrew, thanks for the prompt answer! This particular app is .288 but we have the same issue with another one built with .315 and same issue with our apps on AIR32.
Here are a few other traces (and there are many in the starling forum topic)
` pid: 0, tid: 0 >>> air.com.company.application <<<
backtrace:
` pid: 0, tid: 0 >>> air.com.company.application <<<
backtrace:
` pid: 0, tid: 0 >>> air.com.company.application <<<
backtrace:
This one on Android 8.0 ` pid: 0, tid: 0 >>> air.com.company.application <<<
backtrace:
` pid: 0, tid: 0 >>> air.com.company.application <<<
backtrace:
` pid: 0, tid: 0 >>> air.com.company.application <<<
backtrace:
ajwfrost
commented
4 years ago Looks like lots of different problems! The first one is in openssl .. we can look at adding some protection around this or see whether there's a bad call being made to it. I don't suppose these are reproducible at all..? it's always nicer if we can get confirmation that something had been crashing and then has been fixed..!
neuronix
commented
4 years ago Unfortunately we don't know how/when these errors occur, we just have the reports in the Google Playstore Console. But we will definitely see a drop in crash % once things are fixed :)
Here are a few more from another app with minor differences:
` pid: 0, tid: 0 >>> air.com.company.application2 <<<
backtrace:
` pid: 0, tid: 0 >>> air.com.company.application2 <<<
backtrace:
There are many more different ones but then sometimes only one occurrence. So far the most "popular" one is the "strcmp" one, the one in my first post.
Here are some from another app that is still running on AIR 32. As I stated, the issues arise with targetSDK=28, we have other apps still running with targetSDK=19 on the same AIR 32 build and they don't have any major issues.
Android 9 & 8.1 ` pid: 0, tid: 0 >>> air.com.company.application <<<
backtrace:
Multiple Android versions ` pid: 0, tid: 0 >>> air.com.company.application <<<
backtrace:
Android 9 & 8.1 ` pid: 0, tid: 0 >>> air.com.company.application <<<
backtrace:
Android 9 only ` pid: 0, tid: 0 >>> air.com.company.application <<<
backtrace:
Multiple Android versions ` pid: 0, tid: 0 >>> air.com.company.application <<<
backtrace:
Android 6 only (Starling forum topic started with issues on Android 6) ` pid: 0, tid: 0 >>> air.com.company.application <<<
backtrace:
marcanw
commented
4 years ago Yep same issue Signal 11 or Signal 5 And I confirm majority of crashes occur on android 9
neuronix
commented
4 years ago Hi @ajwfrost
Is there any news on this topic ? I don't see anything in the release notes regarding this. It really is plaguing our apps with insane crash rates: ~6% of sessions.. ! :(
ajwfrost
commented
4 years ago Hi - sorry for the delay. We had a change go in to a build recently that may have improved things, a lot of these issues on Android 9 appear to be due to a change in the workflow and the fact we're not getting a valid activity/context from some of our calls (e.g. when in the background). Looking at it, this was in a 33.1 beta release but not in the current 33.0 branch.. the plan is to release the next production version at the end of next week.
The calls stacks you have above seem to fall into one of three camps:
_ZN9HashTable10LookupItemEPKvPPv or VarStrEqual: this appears to be due to a corrupt (or null) string being used for a filename, where it's trying to get the application directory (which is likely to rely on the context..)
ZIP_GetNextEntry: this may be slightly different if it's accessing resources within the main APK file itself, there were some ANRs relating to this too I think so we were looking to check how we access these and ensure we clean up properly in case of any exceptions on the Java side.
je_free: this is most likely caused by string corruption and is probably linked to the first of the issues here.
There's always another possibility so please keep the call stacks coming (and any register/memory information that you might get as part of these). It's also really helpful if you're able to identify the exact runtime used in each of the crash logs, we trace this out during startup but I'm guessing you don't actually get the full logcat output from Google Play...
thanks
Andrew
neuronix
commented
4 years ago Hi Andrew,
We updated an app with 33.1.1.63 and unfortunately the issue has not changed much, please find more stack traces here:
signal 11 (SIGSEGV), code 1 (SEGV_MAPERR) strcmp
Huawei HUAWEI P30 lite (HWMAR), 3840MB RAM, Android 9
Report 1
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> air.com.company.app <<<
backtrace:
#00 pc 000000000001e748 /system/lib64/libc.so (strcmp+136)
#01 pc 00000000003bad8c /data/app/air.com.company.app-s53aJSh7H_JH1I25gz1MSA==/lib/arm64/libCore.so
#02 pc 00000000003be700 /data/app/air.com.company.app-s53aJSh7H_JH1I25gz1MSA==/lib/arm64/libCore.so
#03 pc 00000000002d38a0 /data/app/air.com.company.app-s53aJSh7H_JH1I25gz1MSA==/lib/arm64/libCore.so
#04 pc 00000000002d4944 /data/app/air.com.company.app-s53aJSh7H_JH1I25gz1MSA==/lib/arm64/libCore.so
#05 pc 00000000002d731c /data/app/air.com.company.app-s53aJSh7H_JH1I25gz1MSA==/lib/arm64/libCore.so
#06 pc 00000000002da084 /data/app/air.com.company.app-s53aJSh7H_JH1I25gz1MSA==/lib/arm64/libCore.so
#07 pc 00000000006f4ac8 /data/app/air.com.company.app-s53aJSh7H_JH1I25gz1MSA==/lib/arm64/libCore.so
#08 pc 00000000006f3d68 /data/app/air.com.company.app-s53aJSh7H_JH1I25gz1MSA==/lib/arm64/libCore.so
#09 pc 0000000000272960 /data/app/air.com.company.app-s53aJSh7H_JH1I25gz1MSA==/lib/arm64/libCore.so
#10 pc 00000000002729d0 /data/app/air.com.company.app-s53aJSh7H_JH1I25gz1MSA==/lib/arm64/libCore.so
#11 pc 0000000000272704 /data/app/air.com.company.app-s53aJSh7H_JH1I25gz1MSA==/lib/arm64/libCore.so
#12 pc 0000000000083588 /system/lib64/libc.so (__pthread_start(void*)+36)
#13 pc 00000000000241dc /system/lib64/libc.so (__start_thread+68)Samsung Galaxy S8 (dreamlte), 3840MB RAM, Android 9
Report 2
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> air.com.company.app <<<
backtrace:
#00 pc 000000000001e2d8 /system/lib64/libc.so (strcmp+24)
#01 pc 00000000003bad8c /data/app/air.com.company.app-IRUfmMRZHqSAgDbfYiWlKQ==/lib/arm64/libCore.so
#02 pc 00000000003be504 /data/app/air.com.company.app-IRUfmMRZHqSAgDbfYiWlKQ==/lib/arm64/libCore.so
#03 pc 00000000002d38e0 /data/app/air.com.company.app-IRUfmMRZHqSAgDbfYiWlKQ==/lib/arm64/libCore.so
#04 pc 00000000002d4944 /data/app/air.com.company.app-IRUfmMRZHqSAgDbfYiWlKQ==/lib/arm64/libCore.so
#05 pc 00000000002d731c /data/app/air.com.company.app-IRUfmMRZHqSAgDbfYiWlKQ==/lib/arm64/libCore.so
#06 pc 00000000002da084 /data/app/air.com.company.app-IRUfmMRZHqSAgDbfYiWlKQ==/lib/arm64/libCore.so
#07 pc 00000000006f4ac8 /data/app/air.com.company.app-IRUfmMRZHqSAgDbfYiWlKQ==/lib/arm64/libCore.so
#08 pc 00000000006f3d68 /data/app/air.com.company.app-IRUfmMRZHqSAgDbfYiWlKQ==/lib/arm64/libCore.so
#09 pc 0000000000272960 /data/app/air.com.company.app-IRUfmMRZHqSAgDbfYiWlKQ==/lib/arm64/libCore.so
#10 pc 00000000002729d0 /data/app/air.com.company.app-IRUfmMRZHqSAgDbfYiWlKQ==/lib/arm64/libCore.so
#11 pc 0000000000272704 /data/app/air.com.company.app-IRUfmMRZHqSAgDbfYiWlKQ==/lib/arm64/libCore.so
#12 pc 0000000000084e00 /system/lib64/libc.so (__pthread_start(void*)+208)
#13 pc 0000000000023a8c /system/lib64/libc.so (__start_thread+68)Google Pixel 2 (walleye), 3840MB RAM, Android 10
Reports
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> air.com.company.app <<<
backtrace:
#00 pc 000000000007e858 /apex/com.android.runtime/lib64/bionic/libc.so (strcmp+24)
#01 pc 00000000003bad8c /data/app/air.com.company.app-XZjr6fKxmHt2iUMgWbfb0Q==/lib/arm64/libCore.so
#02 pc 00000000003be700 /data/app/air.com.company.app-XZjr6fKxmHt2iUMgWbfb0Q==/lib/arm64/libCore.so
#03 pc 00000000002d38a0 /data/app/air.com.company.app-XZjr6fKxmHt2iUMgWbfb0Q==/lib/arm64/libCore.so
#04 pc 00000000002d4944 /data/app/air.com.company.app-XZjr6fKxmHt2iUMgWbfb0Q==/lib/arm64/libCore.so
#05 pc 00000000002d731c /data/app/air.com.company.app-XZjr6fKxmHt2iUMgWbfb0Q==/lib/arm64/libCore.so
#06 pc 00000000002da084 /data/app/air.com.company.app-XZjr6fKxmHt2iUMgWbfb0Q==/lib/arm64/libCore.so
#07 pc 00000000006f4ac8 /data/app/air.com.company.app-XZjr6fKxmHt2iUMgWbfb0Q==/lib/arm64/libCore.so
#08 pc 00000000006f3d68 /data/app/air.com.company.app-XZjr6fKxmHt2iUMgWbfb0Q==/lib/arm64/libCore.so
#09 pc 0000000000272960 /data/app/air.com.company.app-XZjr6fKxmHt2iUMgWbfb0Q==/lib/arm64/libCore.so
#10 pc 00000000002729d0 /data/app/air.com.company.app-XZjr6fKxmHt2iUMgWbfb0Q==/lib/arm64/libCore.so
#11 pc 0000000000272704 /data/app/air.com.company.app-XZjr6fKxmHt2iUMgWbfb0Q==/lib/arm64/libCore.so
#12 pc 00000000000e2364 /apex/com.android.runtime/lib64/bionic/libc.so (__pthread_start(void*)+36)
#13 pc 0000000000083d98 /apex/com.android.runtime/lib64/bionic/libc.so (__start_thread+64)signal 11 (SIGSEGV), code 1 (SEGV_MAPERR) libCore.so
Huawei HUAWEI P30 lite (HWMAR), 3840MB RAM, Android 9
Reports
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> air.com.company.app <<<
backtrace:
#00 pc 00000000003be6f0 /data/app/air.com.company.app-T7O2dUiLEHNqTwzrTWKD5g==/lib/arm64/libCore.so
#01 pc 00000000002d38a0 /data/app/air.com.company.app-T7O2dUiLEHNqTwzrTWKD5g==/lib/arm64/libCore.so
#02 pc 00000000002d4944 /data/app/air.com.company.app-T7O2dUiLEHNqTwzrTWKD5g==/lib/arm64/libCore.so
#03 pc 00000000002d731c /data/app/air.com.company.app-T7O2dUiLEHNqTwzrTWKD5g==/lib/arm64/libCore.so
#04 pc 00000000002da084 /data/app/air.com.company.app-T7O2dUiLEHNqTwzrTWKD5g==/lib/arm64/libCore.so
#05 pc 00000000006f4ac8 /data/app/air.com.company.app-T7O2dUiLEHNqTwzrTWKD5g==/lib/arm64/libCore.so
#06 pc 00000000006f3d68 /data/app/air.com.company.app-T7O2dUiLEHNqTwzrTWKD5g==/lib/arm64/libCore.so
#07 pc 0000000000272960 /data/app/air.com.company.app-T7O2dUiLEHNqTwzrTWKD5g==/lib/arm64/libCore.so
#08 pc 00000000002729d0 /data/app/air.com.company.app-T7O2dUiLEHNqTwzrTWKD5g==/lib/arm64/libCore.so
#09 pc 0000000000272704 /data/app/air.com.company.app-T7O2dUiLEHNqTwzrTWKD5g==/lib/arm64/libCore.so
#10 pc 0000000000083588 /system/lib64/libc.so (__pthread_start(void*)+36)
#11 pc 00000000000241dc /system/lib64/libc.so (__start_thread+68)signal 11 (SIGSEGV), code 1 (SEGV_MAPERR) arena_run_reg_alloc
GE LG X charge (mlv7), 2048MB RAM, Android 7.1
Reports
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> air.com.company.app <<<
backtrace:
#00 pc 0000000000054282 /system/lib/libc.so (arena_run_reg_alloc+101)
#01 pc 00000000000548a9 /system/lib/libc.so (je_arena_malloc_hard+76)
#02 pc 0000000000061c0f /system/lib/libc.so (je_malloc+122)
#03 pc 0000000000048835 /system/lib/libc++.so (operator new(unsigned int)+16)
#04 pc 000000000002618d /system/lib/libandroidfw.so (OpenArchive+24)
#05 pc 0000000000022a25 /system/lib/libandroidfw.so (android::ZipFileRO::open(char const*)+20)
#06 pc 0000000000016777 /system/lib/libandroidfw.so (android::AssetManager::SharedZip::SharedZip(android::String8 const&, long)+70)
#07 pc 0000000000016837 /system/lib/libandroidfw.so (android::AssetManager::SharedZip::get(android::String8 const&, bool)+146)
#08 pc 0000000000015049 /system/lib/libandroidfw.so (android::AssetManager::ZipSet::getZipResourceTableAsset(android::String8 const&)+52)
#09 pc 0000000000013b69 /system/lib/libandroidfw.so (android::AssetManager::appendPathToResTable(android::AssetManager::asset_path const&, bool) const+200)
#10 pc 0000000000013887 /system/lib/libandroidfw.so (android::AssetManager::addAssetPath(android::String8 const&, int*, bool, bool)+302)
#11 pc 000000000009b993 /system/lib/libandroid_runtime.so
#12 pc 0000000001d8a215 /system/framework/arm/boot-framework.oat (offset 0x1af3000) (android.content.res.AssetManager.addAssetPathNative+104)
#13 pc 0000000001d8a13b /system/framework/arm/boot-framework.oat (offset 0x1af3000) (android.content.res.AssetManager.addAssetPathInternal+70)
#14 pc 0000000001d8ba43 /system/framework/arm/boot-framework.oat (offset 0x1af3000) (android.content.res.AssetManager.addAssetPath+46)
#15 pc 0000000001c2721b /system/framework/arm/boot-framework.oat (offset 0x1af3000) (android.app.ResourcesManager.createAssetManager+302)
#16 pc 0000000001c24ebb /system/framework/arm/boot-framework.oat (offset 0x1af3000) (android.app.ResourcesManager.createResourcesImpl+118)
#17 pc 0000000001c256bf /system/framework/arm/boot-framework.oat (offset 0x1af3000) (android.app.ResourcesManager.getOrCreateResources+482)
#18 pc 0000000001c27bd7 /system/framework/arm/boot-framework.oat (offset 0x1af3000) (android.app.ResourcesManager.getResources+282)
#19 pc 0000000001b81153 /system/framework/arm/boot-framework.oat (offset 0x1af3000) (android.app.ActivityThread.getTopLevelResources+150)
#20 pc 0000000001c04ce1 /system/framework/arm/boot-framework.oat (offset 0x1af3000) (android.app.LoadedApk.getResources+84)
#21 pc 0000000001bba39d /system/framework/arm/boot-framework.oat (offset 0x1af3000) (android.app.ContextImpl.<init>+768)
#22 pc 0000000001bbe229 /system/framework/arm/boot-framework.oat (offset 0x1af3000) (android.app.ContextImpl.createApplicationContext+268)
#23 pc 0000000001af665b /system/framework/arm/boot-framework.oat (offset 0x1af3000) (android.content.ContextWrapper.createApplicationContext+46)
#24 pc 00000000024b2fa9 /system/framework/arm/boot-framework.oat (offset 0x1af3000) (android.webkit.WebViewFactory.getWebViewContextAndSetProvider+796)
#25 pc 00000000024b2775 /system/framework/arm/boot-framework.oat (offset 0x1af3000) (android.webkit.WebViewFactory.getProviderClass+104)
#26 pc 00000000024b246b /system/framework/arm/boot-framework.oat (offset 0x1af3000) (android.webkit.WebViewFactory.getProvider+182)
#27 pc 00000000024ab631 /system/framework/arm/boot-framework.oat (offset 0x1af3000) (android.webkit.WebView.getFactory+52)
#28 pc 00000000024ab481 /system/framework/arm/boot-framework.oat (offset 0x1af3000) (android.webkit.WebView.ensureProviderCreated+68)
#29 pc 00000000024af9d5 /system/framework/arm/boot-framework.oat (offset 0x1af3000) (android.webkit.WebView.setOverScrollMode+64)
#30 pc 00000000023a3733 /system/framework/arm/boot-framework.oat (offset 0x1af3000) (android.view.View.<init>+486)
#31 pc 00000000023a3987 /system/framework/arm/boot-framework.oat (offset 0x1af3000) (android.view.View.<init>+66)
#32 pc 00000000023f9761 /system/framework/arm/boot-framework.oat (offset 0x1af3000) (android.view.ViewGroup.<init>+76)
#33 pc 00000000024a9e9d /system/framework/arm/boot-framework.oat (offset 0x1af3000) (android.widget.AbsoluteLayout.<init>+64)
#34 pc 00000000024aae5f /system/framework/arm/boot-framework.oat (offset 0x1af3000) (android.webkit.WebView.<init>+74)
#35 pc 00000000024aaddd /system/framework/arm/boot-framework.oat (offset 0x1af3000) (android.webkit.WebView.<init>+80)
#36 pc 00000000024aad55 /system/framework/arm/boot-framework.oat (offset 0x1af3000) (android.webkit.WebView.<init>+64)
#37 pc 00000000024aace1 /system/framework/arm/boot-framework.oat (offset 0x1af3000) (android.webkit.WebView.<init>+60)
#38 pc 00000000024aac73 /system/framework/arm/boot-framework.oat (offset 0x1af3000) (android.webkit.WebView.<init>+46)
#39 pc 00000000006b42ff /data/user_de/0/com.google.android.gms/app_chimera/m/000000d9/oat/arm/dl-AdsFdrDynamite.integ_21207000006.odex (offset 0x4b0000)tgkill
Huawei Mate 10 lite (HWRNE), 3840MB RAM, Android 8.0
Reports
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> air.com.company.app <<<
backtrace:
#00 pc 000000000006a808 /system/lib64/libc.so (tgkill+8)
#01 pc 000000000001db50 /system/lib64/libc.so (abort+88)
#02 pc 000000000002532c /system/lib64/libc.so (__libc_fatal+116)
#03 pc 0000000000091cf0 /system/lib64/libc.so (ifree+812)
#04 pc 0000000000091f70 /system/lib64/libc.so (je_free+120)
#05 pc 000000000035d2c0 /system/lib64/libart.so (void art::JNI::ReleasePrimitiveArray<_jcharArray*, unsigned short, art::mirror::PrimitiveArray<unsigned short> >(_JNIEnv*, _jcharArray*, unsigned short*, int)+652)
#06 pc 0000000000027ec8 /system/lib64/libjavacore.so (NativeConverter_decode(_JNIEnv*, _jclass*, long, _jbyteArray*, int, _jcharArray*, int, _jintArray*, unsigned char)+872)
#07 pc 00000000003149cc /system/framework/arm64/boot-core-libart.oat (offset 0xd6000) (libcore.icu.NativeConverter.decode [DEDUPED]+268)
#08 pc 0000000000308460 /system/framework/arm64/boot-core-libart.oat (offset 0xd6000) (java.nio.charset.CharsetDecoderICU.implFlush+288)
#09 pc 00000000006b9ee8 /system/framework/arm64/boot.oat (offset 0x1dc000) (java.util.zip.ZipCoder.toString+1160)
#10 pc 0000000000620240 /system/framework/arm64/boot.oat (offset 0x1dc000) (java.util.zip.ZipFile.getZipEntry+592)
#11 pc 00000000006bab10 /system/framework/arm64/boot.oat (offset 0x1dc000) (java.util.zip.ZipFile$ZipEntryIterator.next+1120)
#12 pc 000000000061d588 /system/framework/arm64/boot.oat (offset 0x1dc000) (java.util.jar.JarFile$JarEntryIterator.next [DEDUPED]+40)
#13 pc 00000000003c50b8 /system/framework/arm64/boot.oat (offset 0x1dc000) (java.security.cert.PKIXRevocationChecker.clone [DEDUPED]+40)
#14 pc 0000000000005198 /dev/ashmem/dalvik-jit-code-cache (deleted)Hi @ajwfrost
Please find a new batch of traces here from another app built with .63. It seems like some of these are more detailed and maybe more useful:
signal 11 (SIGSEGV), code 1 (SEGV_MAPERR) strcmp
Huawei P20 lite (HWANE), 3840MB RAM, Android 9
Report 1
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> air.com.company.app2 <<<
backtrace:
#00 pc 000000000001e748 /system/lib64/libc.so (strcmp+136)
#01 pc 00000000003bad8c /data/app/air.com.company.app2-4hNERi4AQP7Gw9VlYL-frg==/lib/arm64/libCore.so
#02 pc 00000000003be700 /data/app/air.com.company.app2-4hNERi4AQP7Gw9VlYL-frg==/lib/arm64/libCore.so
#03 pc 00000000002d38a0 /data/app/air.com.company.app2-4hNERi4AQP7Gw9VlYL-frg==/lib/arm64/libCore.so
#04 pc 00000000002d4944 /data/app/air.com.company.app2-4hNERi4AQP7Gw9VlYL-frg==/lib/arm64/libCore.so
#05 pc 00000000002d731c /data/app/air.com.company.app2-4hNERi4AQP7Gw9VlYL-frg==/lib/arm64/libCore.so
#06 pc 00000000002da084 /data/app/air.com.company.app2-4hNERi4AQP7Gw9VlYL-frg==/lib/arm64/libCore.so
#07 pc 00000000006f4ac8 /data/app/air.com.company.app2-4hNERi4AQP7Gw9VlYL-frg==/lib/arm64/libCore.so
#08 pc 00000000006f3d68 /data/app/air.com.company.app2-4hNERi4AQP7Gw9VlYL-frg==/lib/arm64/libCore.so
#09 pc 0000000000272960 /data/app/air.com.company.app2-4hNERi4AQP7Gw9VlYL-frg==/lib/arm64/libCore.so
#10 pc 00000000002729d0 /data/app/air.com.company.app2-4hNERi4AQP7Gw9VlYL-frg==/lib/arm64/libCore.so
#11 pc 0000000000272704 /data/app/air.com.company.app2-4hNERi4AQP7Gw9VlYL-frg==/lib/arm64/libCore.so
#12 pc 0000000000083588 /system/lib64/libc.so (__pthread_start(void*)+36)
#13 pc 00000000000241dc /system/lib64/libc.so (__start_thread+68)
Samsung Galaxy A6 (a6lte), 2816MB RAM, Android 8.0
Report 1
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> air.com.company.app2 <<<
backtrace:
#00 pc 0000000000019a8c /system/lib/libc.so (strcmp+107)
#01 pc 00000000001e1e6d /data/app/air.com.company.app2-H2qOn1114nQ3KxB3brV35w==/lib/arm/libCore.so
#02 pc 00000000f42c0064 <unknown>signal 11 (SIGSEGV), code 1 (SEGV_MAPERR) arena_dalloc_bin_locked_impl
Huawei honor 10 Lite (HWHRY-H), 2816MB RAM, Android 10
Report 1
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> air.com.company.app2 <<<
backtrace:
#00 pc 0000000000044838 /apex/com.android.runtime/lib64/bionic/libc.so (arena_dalloc_bin_locked_impl+360)
#01 pc 00000000000658d8 /apex/com.android.runtime/lib64/bionic/libc.so (je_tcache_bin_flush_small+340)
#02 pc 000000000006681c /apex/com.android.runtime/lib64/bionic/libc.so (tcache_destroy+304)
#03 pc 00000000000666d8 /apex/com.android.runtime/lib64/bionic/libc.so (je_tcache_cleanup+24)
#04 pc 0000000000066f84 /apex/com.android.runtime/lib64/bionic/libc.so (je_tsd_cleanup+64)
#05 pc 0000000000067dd8 /apex/com.android.runtime/lib64/bionic/libc.so (je_tsd_cleanup_wrapper+32)
#06 pc 00000000000ce850 /apex/com.android.runtime/lib64/bionic/libc.so (pthread_key_clean_all()+116)
#07 pc 00000000000ce2b4 /apex/com.android.runtime/lib64/bionic/libc.so (pthread_exit+72)
#08 pc 00000000000ce194 /apex/com.android.runtime/lib64/bionic/libc.so (__pthread_start(void*)+40)
#09 pc 0000000000070ba8 /apex/com.android.runtime/lib64/bionic/libc.so (__start_thread+64)signal 11 (SIGSEGV), code 1 (SEGV_MAPERR) pthread_mutex_lock
Huawei HUAWEI MediaPad T3 10 (HWAGS-Q), 2048MB RAM, Android 8.0
Report 1 of 4
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> air.com.company.app2 <<<
backtrace:
#00 pc 0000000000075c20 /system/lib64/libc.so (pthread_mutex_lock)
#01 pc 00000000000b5084 /system/lib64/libc.so (je_tcache_bin_flush_small+216)
#02 pc 00000000000b7998 /system/lib64/libc.so (tcache_destroy+2676)
#03 pc 00000000000b6f10 /system/lib64/libc.so (je_tcache_cleanup+28)
#04 pc 00000000000b8a9c /system/lib64/libc.so (je_tsd_cleanup+60)
#05 pc 00000000000b9590 /system/lib64/libc.so (je_tsd_cleanup_wrapper+32)
#06 pc 000000000007588c /system/lib64/libc.so (pthread_key_clean_all()+128)
#07 pc 0000000000075340 /system/lib64/libc.so (pthread_exit+72)
#08 pc 00000000000751f8 /system/lib64/libc.so (__pthread_start(void*)+40)
#09 pc 000000000001f83c /system/lib64/libc.so (__start_thread+68)signal 11 (SIGSEGV), code 1 (SEGV_MAPERR) art::gc::collector::ConcurrentCopying::ProcessThreadLocalMarkStacks(bool, art::Closure*)
Huawei P20 lite (HWANE), 3840MB RAM, Android 9
Report 1 of 2
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> air.com.company.app2 <<<
backtrace:
#00 pc 00000000001c9bfc /system/lib64/libart.so (art::gc::collector::ConcurrentCopying::ProcessThreadLocalMarkStacks(bool, art::Closure*)+324)
#01 pc 00000000001c97c4 /system/lib64/libart.so (art::gc::collector::ConcurrentCopying::ProcessMarkStackOnce()+960)
#02 pc 00000000001c93ec /system/lib64/libart.so (art::gc::collector::ConcurrentCopying::ProcessMarkStack()+28)
#03 pc 00000000001c2908 /system/lib64/libart.so (art::gc::collector::ConcurrentCopying::MarkingPhase()+1756)
#04 pc 00000000001c0d68 /system/lib64/libart.so (art::gc::collector::ConcurrentCopying::RunPhases()+1176)
#05 pc 00000000001d7018 /system/lib64/libart.so (art::gc::collector::GarbageCollector::Run(art::gc::GcCause, bool)+320)
#06 pc 00000000001fcde8 /system/lib64/libart.so (art::gc::Heap::CollectGarbageInternal(art::gc::collector::GcType, art::gc::GcCause, bool)+3404)
#07 pc 000000000020e404 /system/lib64/libart.so (art::gc::Heap::ConcurrentGC(art::Thread*, art::gc::GcCause, bool)+128)
#08 pc 0000000000213aec /system/lib64/libart.so (art::gc::Heap::ConcurrentGCTask::Run(art::Thread*)+40)
#09 pc 0000000000235e9c /system/lib64/libart.so (art::gc::TaskProcessor::RunAllTasks(art::Thread*)+68)
#10 pc 0000000000092a6c /system/framework/arm64/boot-core-libart.oat (offset 0x90000) (dalvik.system.VMRuntime.clampGrowthLimit [DEDUPED]+124)
#11 pc 00000000001d79d4 /system/framework/arm64/boot-core-libart.oat (offset 0x90000) (java.lang.Daemons$HeapTaskDaemon.runInternal+196)
#12 pc 000000000015978c /system/framework/arm64/boot-core-libart.oat (offset 0x90000) (java.lang.Daemons$Daemon.run+76)
#13 pc 00000000002c1038 /system/framework/arm64/boot.oat (offset 0x13b000) (java.lang.Thread.run+72)
#14 pc 0000000000571d88 /system/lib64/libart.so (art_quick_invoke_stub+584)
#15 pc 00000000000d4d2c /system/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+200)
#16 pc 0000000000475d3c /system/lib64/libart.so (art::(anonymous namespace)::InvokeWithArgArray(art::ScopedObjectAccessAlreadyRunnable const&, art::ArtMethod*, art::(anonymous namespace)::ArgArray*, art::JValue*, char const*)+104)
#17 pc 0000000000476df8 /system/lib64/libart.so (art::InvokeVirtualOrInterfaceWithJValues(art::ScopedObjectAccessAlreadyRunnable const&, _jobject*, _jmethodID*, jvalue*)+424)
#18 pc 00000000004a23ec /system/lib64/libart.so (art::Thread::CreateCallback(void*)+1120)
#19 pc 0000000000083588 /system/lib64/libc.so (__pthread_start(void*)+36)
#20 pc 00000000000241dc /system/lib64/libc.so (__start_thread+68)signal 11 (SIGSEGV), code 1 (SEGV_MAPERR) je_arena_malloc_hard
Wiko W_C200SN (W_C200SN), 1024MB RAM, Android 8.1
Report 1 of 2
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> air.com.company.app2 <<<
backtrace:
#00 pc 0000000000057756 /system/lib/libc.so (je_arena_malloc_hard+201)
#01 pc 0000000000065e8b /system/lib/libc.so (a0ialloc+58)
#02 pc 00000000000668bd /system/lib/libc.so (je_malloc+320)
#03 pc 0000000000081781 /data/app/air.com.company.app2-HOIf_QkuhUUyKg8FzcqIMw==/lib/arm/libc++_shared.so (operator new(unsigned int)+24)
#04 pc 00000000000817b1 /data/app/air.com.company.app2-HOIf_QkuhUUyKg8FzcqIMw==/lib/arm/libc++_shared.so (operator new(unsigned int, std::nothrow_t const&)+4)
#05 pc 0000000000520d8b /data/app/air.com.company.app2-HOIf_QkuhUUyKg8FzcqIMw==/lib/arm/libCore.so
#06 pc 00000000005212a1 /data/app/air.com.company.app2-HOIf_QkuhUUyKg8FzcqIMw==/lib/arm/libCore.so
#07 pc 00000000000ed2e1 /data/app/air.com.company.app2-HOIf_QkuhUUyKg8FzcqIMw==/lib/arm/libCore.so
#08 pc 00000000000ed343 /data/app/air.com.company.app2-HOIf_QkuhUUyKg8FzcqIMw==/lib/arm/libCore.so
#09 pc 00000000000ed0b7 /data/app/air.com.company.app2-HOIf_QkuhUUyKg8FzcqIMw==/lib/arm/libCore.so
#10 pc 0000000000047627 /system/lib/libc.so (__pthread_start(void*)+22)
#11 pc 000000000001af9d /system/lib/libc.so (__start_thread+32)signal 11 (SIGSEGV), code 1 (SEGV_MAPERR) arena_run_reg_alloc
Samsung Galaxy Tab A (2016) (gtaxllte), 2048MB RAM, Android 6.0
Report 1 of 2
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> air.com.company.app2 <<<
backtrace:
#00 pc 0000000000049948 /system/lib/libc.so (arena_run_reg_alloc+87)
#01 pc 000000000004b70f /system/lib/libc.so (je_arena_tcache_fill_small+94)
#02 pc 000000000005ca4b /system/lib/libc.so (je_tcache_alloc_small_hard+18)
#03 pc 000000000005753d /system/lib/libc.so (je_malloc+1112)
#04 pc 000000000051d269 /data/app/air.com.company.app2-1/lib/arm/libCore.so
#05 pc 00000000000e25e7 /data/app/air.com.company.app2-1/lib/arm/libCore.so
#06 pc 00000000002a3b0f /data/app/air.com.company.app2-1/lib/arm/libCore.so
#07 pc 000000000042cfe9 /data/app/air.com.company.app2-1/lib/arm/libCore.so
#08 pc 000000000042dac3 /data/app/air.com.company.app2-1/lib/arm/libCore.so
#09 pc 000000000042cf9b /data/app/air.com.company.app2-1/lib/arm/libCore.so
#10 pc 00000000000ed2ed /data/app/air.com.company.app2-1/lib/arm/libCore.so
#11 pc 00000000000ed343 /data/app/air.com.company.app2-1/lib/arm/libCore.so
#12 pc 00000000000ed0b7 /data/app/air.com.company.app2-1/lib/arm/libCore.so
#13 pc 000000000003fc73 /system/lib/libc.so (__pthread_start(void*)+30)
#14 pc 000000000001a3ab /system/lib/libc.so (__start_thread+6)signal 11 (SIGSEGV), code 1 (SEGV_MAPERR) ifree
Huawei P20 lite (HWANE), 3840MB RAM, Android 9
Report 1 of 2
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> air.com.company.app2 <<<
backtrace:
#00 pc 00000000000a0954 /system/lib64/libc.so (ifree+88)
#01 pc 00000000000a0edc /system/lib64/libc.so (je_free+120)
#02 pc 00000000007f874c /data/app/air.com.company.app2-cHIQ0tuoGdkdpAdylEvgfQ==/lib/arm64/libCore.so
#03 pc 00000000007f8b80 /data/app/air.com.company.app2-cHIQ0tuoGdkdpAdylEvgfQ==/lib/arm64/libCore.so
#04 pc 00000000007b6b20 /data/app/air.com.company.app2-cHIQ0tuoGdkdpAdylEvgfQ==/lib/arm64/libCore.so
#05 pc 00000000007b694c /data/app/air.com.company.app2-cHIQ0tuoGdkdpAdylEvgfQ==/lib/arm64/libCore.so
#06 pc 00000000007b6774 /data/app/air.com.company.app2-cHIQ0tuoGdkdpAdylEvgfQ==/lib/arm64/libCore.so
#07 pc 00000000007b61a0 /data/app/air.com.company.app2-cHIQ0tuoGdkdpAdylEvgfQ==/lib/arm64/libCore.so
#08 pc 00000000000051b4 <anonymous>signal 11 (SIGSEGV), code 2 (SEGV_ACCERR)
art::JNIEnvExt::JNIEnvExt(art::Thread, art::JavaVMExt, std::1::basic_string<char, std::__1::char_traits
Huawei FIG-LX1 (HWFIG-H), 2816MB RAM, Android 9
Report 1 of 2
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> air.com.company.app2 <<<
backtrace:
#00 pc 000000000030ae34 /system/lib64/libart.so (art::JNIEnvExt::JNIEnvExt(art::Thread*, art::JavaVMExt*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>*)+28)
#01 pc 000000000030adb4 /system/lib64/libart.so (art::JNIEnvExt::Create(art::Thread*, art::JavaVMExt*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>*)+52)
#02 pc 00000000004a2ac8 /system/lib64/libart.so (art::Thread::Init(art::ThreadList*, art::JavaVMExt*, art::JNIEnvExt*)+344)
#03 pc 00000000004a4db8 /system/lib64/libart.so (art::Thread::Attach(char const*, bool, _jobject*, bool)+452)
#04 pc 000000000047c7bc /system/lib64/libart.so (art::Runtime::AttachCurrentThread(char const*, bool, _jobject*, bool)+112)
#05 pc 00000000002f4708 /system/lib64/libart.so (art::JII::AttachCurrentThreadInternal(_JavaVM*, _JNIEnv**, void*, bool)+244)
#06 pc 000000000026ef00 /data/app/air.com.company.app2-j_F-pNDIrZBq7F2J0M5-1w==/lib/arm64/libCore.so
#07 pc 0000000000272958 /data/app/air.com.company.app2-j_F-pNDIrZBq7F2J0M5-1w==/lib/arm64/libCore.so
#08 pc 00000000002729d0 /data/app/air.com.company.app2-j_F-pNDIrZBq7F2J0M5-1w==/lib/arm64/libCore.so
#09 pc 0000000000272704 /data/app/air.com.company.app2-j_F-pNDIrZBq7F2J0M5-1w==/lib/arm64/libCore.so
#10 pc 0000000000083588 /system/lib64/libc.so (__pthread_start(void*)+36)
#11 pc 00000000000241dc /system/lib64/libc.so (__start_thread+68)Wow, thanks for all the details there! we'll go through some of those to try to see what we can do to protect from the crashes..
neuronix
commented
4 years ago Hi @ajwfrost this issue is getting out of hand. The last app we updated now has a 9.5% crash rate! This really needs to be resolved urgently. It's compromising all our marketing efforts.
Please understand that most of your clients like us are long term AIR users. We don't care about features, we don't need any new features, ANEs can handle those. We need stability, we need bug fixes and we need store guideline deadlines to be met. Please focus on these points, AIR 33.1 is making us nervous.. :(
Last version AIR 33.1.0.63
Previous version AIR 32.0.0.116
neuronix
commented
4 years ago Here is the crash report for the most regular crash (strcmp):
Here is the entire developer console crash list, if there are any ones in here you'd like me to detail, please ask:
ajwfrost
commented
4 years ago Hi
Thanks for this - we've been looking into this and can see where it's crashing - but we can't actually see why there would be a problem. The code all looks fine and has null-checks and appears thread-safe. There is obviously something happening here which is causing a problem but we could do with being able to reproduce it ourselves so that we can dig deeper..
From the logs and call stacks we can see what sort of scenarios are happening when the crashes happen - so I'm assuming you're doing a lot of accesses to resources that are obtained via an "app://" URL (i.e. from the application folder) which are asynchronous. Am just wondering whether any of them are also synchronous accesses.. but I can't see how a mixture of the two would be a problem. However our working assumption currently is that there is something wrong with the memory accesses when the elements have been created in various threads and are added into a 'master' cache. We'll keep investigating, but unfortunately I don't think there's an immediate fix coming.
Was there an earlier version where the rate was lower? 32.0.0.116 still looks quite a bad crash rate to me..
thanks
Andrew
neuronix
commented
4 years ago Hi Andrew,
Thanks for your prompt reply!
Indeed this is the most resource demanding app we've ever made, loading 100+ atfs, sounds and other files. It has a huge scrollable map with lots and lots of images loaded from the application/storage directories. ATFs are the most loaded file type and we are reading files asynchronously to load them since it is a bit faster and less resource intensive than using urlloaders. There are also lots of File.exists checks to cancel the load if the file is missing and I guess the exists() is synchronous.
Our largest app still runs on 32.0.0.100 (we have been delaying updating to avoid this issue) and has a 0.13% crash rate for the last 30 days which is negligible. Something happened after 32.0.0.100 (we tried many 32 versions before staying on 116) that introduced these crashes.
One remark though, there was no big change in the number of resources loaded between our two app versions (5.27 on 32.0.0.116 and 5.39 on 33.1.1.63) so the increase in crash rate is weird if it's a async/sync file access issue.
We will try skipping the File.exists checks in next release and report back our findings!
ajwfrost
commented
4 years ago Okay thanks, that's an interesting clue as there was a fix that went into the async file handling code around that time - we can go back through the changes to see whether there's anything that should impact this (from memory it was mostly around synchronisation where there had been some deadlock in a background thread that prevented some of the progress events when loading large files.. so not really in the same area as where these crashes are happening, but it's suspicious enough..)
thanks
neuronix
commented
4 years ago Hi @ajwfrost
Please find some more stack traces from another of our apps which is smaller and has a very reasonable amount of files to load (probably 20-30 all file types combined). Also built with 33.1.1.63 and we also noticed an increase in the crash rate moving from 5.47% (not shown on the screenshot) to now up to 11.21% of sessions....
The strcmp error is also the most frequent but I don't see anything new in the trace. Here are some new traces though.
Please let me know if there is anything else we could provide you with. I know I keep on adding things here, I'm not trying to be annoying, just adding the info as it comes :)
signal 11 (SIGSEGV), code 2 (SEGV_ACCERR) je_tcache_bin_flush_small
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> air.com.company.app <<<
backtrace:
#00 pc 000000000006588c /apex/com.android.runtime/lib64/bionic/libc.so (je_tcache_bin_flush_small+264)
#01 pc 000000000006681c /apex/com.android.runtime/lib64/bionic/libc.so (tcache_destroy+304)
#02 pc 00000000000666d8 /apex/com.android.runtime/lib64/bionic/libc.so (je_tcache_cleanup+24)
#03 pc 0000000000066f84 /apex/com.android.runtime/lib64/bionic/libc.so (je_tsd_cleanup+64)
#04 pc 0000000000067dd8 /apex/com.android.runtime/lib64/bionic/libc.so (je_tsd_cleanup_wrapper+32)
#05 pc 00000000000ce850 /apex/com.android.runtime/lib64/bionic/libc.so (pthread_key_clean_all()+116)
#06 pc 00000000000ce2b4 /apex/com.android.runtime/lib64/bionic/libc.so (pthread_exit+72)
#07 pc 00000000000ce194 /apex/com.android.runtime/lib64/bionic/libc.so (__pthread_start(void*)+40)
#08 pc 0000000000070ba8 /apex/com.android.runtime/lib64/bionic/libc.so (__start_thread+64) signal 11 (SIGSEGV), code 1 (SEGV_MAPERR) je_arena_tcache_fill_small
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> air.com.company.app <<<
backtrace:
#00 pc 0000000000057dee /system/lib/libc.so (je_arena_tcache_fill_small+281)
#01 pc 0000000000078141 /system/lib/libc.so (je_tcache_alloc_small_hard+16)
#02 pc 000000000006922f /system/lib/libc.so (je_malloc+630)
#03 pc 0000000000010d0b /system/lib/libopenjdk.so (newEntry+190)
#04 pc 0000000000010fc7 /system/lib/libopenjdk.so (ZIP_GetNextEntry+32)
#05 pc 000000000000eda3 /system/lib/libopenjdk.so (ZipFile_getNextEntry+6)
#06 pc 000000000052e4b1 /system/framework/arm/boot.oat (offset 0x1cc000) (java.util.zip.ZipFile.getNextEntry [DEDUPED]+112)
#07 pc 00000000005ab795 /system/framework/arm/boot.oat (offset 0x1cc000) (java.util.zip.ZipFile$ZipEntryIterator.next+284)
#08 pc 000000000052c2ef /system/framework/arm/boot.oat (offset 0x1cc000) (java.util.jar.JarFile$JarEntryIterator.next [DEDUPED]+38)
#09 pc 0000000000357e27 /system/framework/arm/boot.oat (offset 0x1cc000) (java.security.cert.PKIXRevocationChecker.clone [DEDUPED]+38)
#10 pc 0000000000095153 /data/app/air.com.company.app-Fw2-AH2cwlekNkuyQK43cA==/oat/arm/base.odex (offset 0x86000)signal 11 (SIGSEGV), code 1 (SEGV_MAPERR) pthread_mutex_lock
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> air.com.company.app <<<
backtrace:
#00 pc 0000000000084020 /system/lib64/libc.so (pthread_mutex_lock)
#01 pc 00000000000aaf08 /system/lib64/libc.so (je_tcache_bin_flush_small+176)
#02 pc 00000000000abe08 /system/lib64/libc.so (tcache_destroy+300)
#03 pc 00000000000abcc8 /system/lib64/libc.so (je_tcache_cleanup+28)
#04 pc 00000000000ac5d0 /system/lib64/libc.so (je_tsd_cleanup+60)
#05 pc 00000000000ad15c /system/lib64/libc.so (je_tsd_cleanup_wrapper+32)
#06 pc 0000000000083c38 /system/lib64/libc.so (pthread_key_clean_all()+120)
#07 pc 00000000000836a8 /system/lib64/libc.so (pthread_exit+72)
#08 pc 000000000008358c /system/lib64/libc.so (__pthread_start(void*)+40)
#09 pc 00000000000241dc /system/lib64/libc.so (__start_thread+68)
*signal 11 (SIGSEGV), code 1 (SEGV_MAPERR) art::gc::collector::ConcurrentCopying::ProcessThreadLocalMarkStacks(bool, art::Closure)**
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> air.com.company.app <<<
backtrace:
#00 pc 00000000001c6e98 /system/lib64/libart.so (art::gc::collector::ConcurrentCopying::ProcessThreadLocalMarkStacks(bool, art::Closure*)+320)
#01 pc 00000000001c6a58 /system/lib64/libart.so (art::gc::collector::ConcurrentCopying::ProcessMarkStackOnce()+968)
#02 pc 00000000001c6674 /system/lib64/libart.so (art::gc::collector::ConcurrentCopying::ProcessMarkStack()+28)
#03 pc 00000000001bfa64 /system/lib64/libart.so (art::gc::collector::ConcurrentCopying::MarkingPhase()+1748)
#04 pc 00000000001bde54 /system/lib64/libart.so (art::gc::collector::ConcurrentCopying::RunPhases()+1340)
#05 pc 00000000001d44ac /system/lib64/libart.so (art::gc::collector::GarbageCollector::Run(art::gc::GcCause, bool)+316)
#06 pc 00000000001f73d4 /system/lib64/libart.so (art::gc::Heap::CollectGarbageInternal(art::gc::collector::GcType, art::gc::GcCause, bool)+3428)
#07 pc 0000000000208d00 /system/lib64/libart.so (art::gc::Heap::ConcurrentGC(art::Thread*, art::gc::GcCause, bool)+128)
#08 pc 000000000020e450 /system/lib64/libart.so (art::gc::Heap::ConcurrentGCTask::Run(art::Thread*)+40)
#09 pc 000000000023136c /system/lib64/libart.so (art::gc::TaskProcessor::RunAllTasks(art::Thread*)+68)
#10 pc 000000000007b56c /system/framework/arm64/boot-core-libart.oat (offset 0x79000) (dalvik.system.VMRuntime.clampGrowthLimit [DEDUPED]+124)
#11 pc 00000000001845d4 /system/framework/arm64/boot-core-libart.oat (offset 0x79000) (java.lang.Daemons$HeapTaskDaemon.runInternal+196)
#12 pc 0000000000111e7c /system/framework/arm64/boot-core-libart.oat (offset 0x79000) (java.lang.Daemons$Daemon.run+76)
#13 pc 000000000025e068 /system/framework/arm64/boot.oat (offset 0x115000) (java.lang.Thread.run+72)
#14 pc 0000000000561f88 /system/lib64/libart.so (art_quick_invoke_stub+584)
#15 pc 00000000000d0140 /system/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+200)
#16 pc 00000000004682e0 /system/lib64/libart.so (art::(anonymous namespace)::InvokeWithArgArray(art::ScopedObjectAccessAlreadyRunnable const&, art::ArtMethod*, art::(anonymous namespace)::ArgArray*, art::JValue*, char const*)+104)
#17 pc 00000000004693a8 /system/lib64/libart.so (art::InvokeVirtualOrInterfaceWithJValues(art::ScopedObjectAccessAlreadyRunnable const&, _jobject*, _jmethodID*, jvalue*)+424)
#18 pc 0000000000494ca0 /system/lib64/libart.so (art::Thread::CreateCallback(void*)+1120)
#19 pc 0000000000084dc0 /system/lib64/libc.so (__pthread_start(void*)+208)
#20 pc 0000000000023a4c /system/lib64/libc.so (__start_thread+68)signal 7 (SIGBUS), code 1 (BUS_ADRALN) libGLESv2_adreno.so
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> air.com.company.app <<<
backtrace:
#00 pc 000000000039c1d8 /vendor/lib64/egl/libGLESv2_adreno.so
#01 pc 00000000001d90d0 /vendor/lib64/egl/libGLESv2_adreno.so
#02 pc 000000000039c0b4 /vendor/lib64/egl/libGLESv2_adreno.so
#03 pc 00000000001f1538 /vendor/lib64/egl/libGLESv2_adreno.so
#04 pc 00000000001c7f5c /vendor/lib64/egl/libGLESv2_adreno.so
#05 pc 0000000000153348 /vendor/lib64/egl/libGLESv2_adreno.so
#06 pc 000000000033c5c4 /data/app/air.com.company.app-vEzQ8-QcG0nPRPKthFtgLg==/lib/arm64/libCore.so
#07 pc 000000000033c710 /data/app/air.com.company.app-vEzQ8-QcG0nPRPKthFtgLg==/lib/arm64/libCore.so
#08 pc 000000000033c77c /data/app/air.com.company.app-vEzQ8-QcG0nPRPKthFtgLg==/lib/arm64/libCore.so
#09 pc 000000000032de94 /data/app/air.com.company.app-vEzQ8-QcG0nPRPKthFtgLg==/lib/arm64/libCore.so
#10 pc 0000000000341160 /data/app/air.com.company.app-vEzQ8-QcG0nPRPKthFtgLg==/lib/arm64/libCore.so
#11 pc 0000000000264840 /data/app/air.com.company.app-vEzQ8-QcG0nPRPKthFtgLg==/lib/arm64/libCore.so
#12 pc 0000000000336d68 /data/app/air.com.company.app-vEzQ8-QcG0nPRPKthFtgLg==/lib/arm64/libCore.so
#13 pc 00000000002bbd10 /data/app/air.com.company.app-vEzQ8-QcG0nPRPKthFtgLg==/lib/arm64/libCore.so
#14 pc 00000000002bc9c4 /data/app/air.com.company.app-vEzQ8-QcG0nPRPKthFtgLg==/lib/arm64/libCore.so
#15 pc 00000000002bb404 /data/app/air.com.company.app-vEzQ8-QcG0nPRPKthFtgLg==/lib/arm64/libCore.so
#16 pc 00000000002bcb08 /data/app/air.com.company.app-vEzQ8-QcG0nPRPKthFtgLg==/lib/arm64/libCore.so
#17 pc 00000000002c058c /data/app/air.com.company.app-vEzQ8-QcG0nPRPKthFtgLg==/lib/arm64/libCore.so
#18 pc 00000000003634b0 /data/app/air.com.company.app-vEzQ8-QcG0nPRPKthFtgLg==/lib/arm64/libCore.so
#19 pc 0000000000363f50 /data/app/air.com.company.app-vEzQ8-QcG0nPRPKthFtgLg==/lib/arm64/libCore.so
#20 pc 0000000000363e6c /data/app/air.com.company.app-vEzQ8-QcG0nPRPKthFtgLg==/lib/arm64/libCore.so
#21 pc 00000000003600f0 /data/app/air.com.company.app-vEzQ8-QcG0nPRPKthFtgLg==/lib/arm64/libCore.so
#22 pc 000000000048f0f0 /data/app/air.com.company.app-vEzQ8-QcG0nPRPKthFtgLg==/lib/arm64/libCore.so
#23 pc 00000000004b0ffc /data/app/air.com.company.app-vEzQ8-QcG0nPRPKthFtgLg==/lib/arm64/libCore.so
#24 pc 00000000004b1be4 /data/app/air.com.company.app-vEzQ8-QcG0nPRPKthFtgLg==/lib/arm64/libCore.so
#25 pc 000000000025fdc0 /data/app/air.com.company.app-vEzQ8-QcG0nPRPKthFtgLg==/lib/arm64/libCore.so
#26 pc 0000000000263104 /data/app/air.com.company.app-vEzQ8-QcG0nPRPKthFtgLg==/lib/arm64/libCore.so
#27 pc 0000000000266828 /data/app/air.com.company.app-vEzQ8-QcG0nPRPKthFtgLg==/lib/arm64/libCore.so
#28 pc 00000000000960bc /data/app/air.com.company.app-vEzQ8-QcG0nPRPKthFtgLg==/oat/arm64/base.odex (art_jni_trampoline+140)
#29 pc 00000000000eca14 /data/app/air.com.company.app-vEzQ8-QcG0nPRPKthFtgLg==/oat/arm64/base.odex (com.adobe.air.customHandler.handleMessage+68)
#30 pc 0000000000736764 /system/framework/arm64/boot-framework.oat (android.os.Handler.dispatchMessage+180)
#31 pc 0000000000739db4 /system/framework/arm64/boot-framework.oat (android.os.Looper.loop+1748)
#32 pc 00000000004c6c24 /system/framework/arm64/boot-framework.oat (android.app.ActivityThread.main+756)
#33 pc 00000000001375b8 /apex/com.android.runtime/lib64/libart.so (art_quick_invoke_static_stub+568)
#34 pc 000000000014600c /apex/com.android.runtime/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+276)
#35 pc 00000000004b0d28 /apex/com.android.runtime/lib64/libart.so (art::(anonymous namespace)::InvokeWithArgArray(art::ScopedObjectAccessAlreadyRunnable const&, art::ArtMethod*, art::(anonymous namespace)::ArgArray*, art::JValue*, char const*)+104)
#36 pc 00000000004b28cc /apex/com.android.runtime/lib64/libart.so (art::InvokeMethod(art::ScopedObjectAccessAlreadyRunnable const&, _jobject*, _jobject*, _jobject*, unsigned long)+1480)
#37 pc 000000000043db64 /apex/com.android.runtime/lib64/libart.so (art::Method_invoke(_JNIEnv*, _jobject*, _jobject*, _jobjectArray*)+52)
#38 pc 00000000000c2b54 /system/framework/arm64/boot.oat (art_jni_trampoline+180)
#39 pc 00000000009b0118 /system/framework/arm64/boot-framework.oat (com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run+136)
#40 pc 00000000009b7cec /system/framework/arm64/boot-framework.oat (com.android.internal.os.ZygoteInit.main+2076)
#41 pc 00000000001375b8 /apex/com.android.runtime/lib64/libart.so (art_quick_invoke_static_stub+568)
#42 pc 000000000014600c /apex/com.android.runtime/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+276)
#43 pc 00000000004b0d28 /apex/com.android.runtime/lib64/libart.so (art::(anonymous namespace)::InvokeWithArgArray(art::ScopedObjectAccessAlreadyRunnable const&, art::ArtMethod*, art::(anonymous namespace)::ArgArray*, art::JValue*, char const*)+104)
#44 pc 00000000004b0918 /apex/com.android.runtime/lib64/libart.so (art::InvokeWithVarArgs(art::ScopedObjectAccessAlreadyRunnable const&, _jobject*, _jmethodID*, std::__va_list)+408)
#45 pc 00000000003bb5a8 /apex/com.android.runtime/lib64/libart.so (art::JNI::CallStaticVoidMethodV(_JNIEnv*, _jclass*, _jmethodID*, std::__va_list)+624)
#46 pc 00000000000bf99c /system/lib64/libandroid_runtime.so (_JNIEnv::CallStaticVoidMethod(_jclass*, _jmethodID*, ...)+116)
#47 pc 00000000000c282c /system/lib64/libandroid_runtime.so (android::AndroidRuntime::start(char const*, android::Vector<android::String8> const&, bool)+776)
#48 pc 00000000000035b0 /system/bin/app_process64 (main+1376)
#49 pc 000000000007e898 /apex/com.android.runtime/lib64/bionic/libc.so (__libc_init+108)
Hi @ajwfrost
I have new data on this topic:
One of our test devices has crashed with several signal 11 and a signal 6, I will email you the log file today. Hopefully this will be useful!
Some new information on our app with lots of files to load: the higher crash rates seems to mostly impact the 64 bit build, it's seems it's 2x worse for 64bit:
Version 5.39.1 is 64bit Version 5.39 is 32bit
Here are screenshots of the crash report lists:
64bit:
32bit:
Dima107
commented
4 years ago Hi. We have similar problems with our app. Have you any recipe how to minimize ANR rate?
neuronix
commented
4 years ago Hi @Dima107 so far, we have not found any workaround. If you have such errors occur on one of your devices, please send the log files to adobe.support@harman.com so that @ajwfrost can investigate. Otherwise, if you have any detailed stack traces in your Developer Console that are not listed here, please add them :)
ajwfrost
commented
4 years ago Thanks for all the inputs! So the main thing still seems to be the strcmp .. wondering whether you'll see an improvement from the change we put into the 33.1.1.86 release but if this is GC-related then that may not help, we need to do some more investigations to be able to reproduce this here.
Wondering too, if anyone is using shared bytearrays between workers, there is an ARMv8-specific problem with that currently. That may explain the difference in overall instabilities between 32-bit and 64-bit..? If you are seeing this, please try to avoid changing the length (or calling clear) on any shared bytearray..
thanks
neuronix
commented
4 years ago Hi @ajwfrost we will release with .1.86 in a week or so, so i'll be able to provide data in 2-3 weeks time.
Hopefully we might have sooner results with our crashing test device, will report ASAP! Did you receive the log file by email? Is it more useful than the stack traces?
Regarding workers and shared bytearrays we do not use them at all as far as I know, The only code we have with bytearrays is when loading binary files asynchronously (ATFs) and the are some in the Starling internals for loading shaders unless i'm mistaken.
Thanks for your support on this issue!
ajwfrost
commented
4 years ago Great @neuronix thanks - yes, received the log file and having the register information for the crash is really helpful; if you're able to pull the tombstone files off the device as well it would be great as they provide more context with memory dumps around the locations of each register..
cheers
marcanw
commented
4 years ago Here is mine
ajwfrost
commented
4 years ago @neuronix so with your debug file we can see what's going wrong when we hit the strcmp issue (and the other native stacks that are similar) and it appears to be because there's a hash table being expanded at the same time as another thread is accessing it. The synchronisation that we added into .86 should then solve this.
There's another issue when it's reading a zip file via the Java APIs, we're not clear on why this may be a problem yet unless Java is trying to be efficient and sharing resources across threads..
@marcanw can you please confirm which sort of sensor events you're using? This is crashing when removing a sensor event listener but the variable that's invalid here shouldn't ever be invalid! Do you have any other details on this one (register values or a tombstone dump)?
thanks
marcanw
commented
4 years ago I checked and I don't use directly a sensor. However I'm using several ane that might use a sensor. I think about Distriqt Application and Camera ane.
marchbold
commented
4 years ago Both of those extensions use the orientation sensor, is that what you mean?
ajwfrost
commented
4 years ago Thanks - yes it could have been the orientation sensor that triggered this, will take a look at where those are set up in case there's a way it could have got a bad pointer..
FYI the issue is happening with a pointer within a sensor event source, which is set up if you add an event listener to a DeviceRotation object..
hardcoremore
commented
4 years ago Wondering too, if anyone is using shared bytearrays between workers, there is an ARMv8-specific problem with that currently. That may explain the difference in overall instabilities between 32-bit and 64-bit..? If you are seeing this, please try to avoid changing the length (or calling clear) on any shared bytearray..
Hi Andrew,
This bug was present in Adobe AIR for ages where in some cases when you clear or set length = 0 to shared byte array between workers the byte array .length property still keeps showing length which means byte array was probably not cleared. Adobe never fixed it.
ajwfrost
commented
4 years ago That's not good! Will check this .. the code we've been looking at for that other crash issue caused by workers changing the length appears to suggest that it should work: the idea is, as soon as one thread (aka Worker) wants to change the length, we get all threads to synchronise and throw out any active references to the byte array: then one thread makes the change: and then they all continue, picking back up the new byte array details as required. It's quite clever!
thanks
neuronix
commented
4 years ago Hi @ajwfrost,
We updated one of our apps with 33.1.1.86 about a week ago and since also updated to .98. Even though the userbase is pretty small, so far, so good for the crashes! We have just about none, a single report which has no proper stack trace.
However, the bad news is since we are also getting ANRs which seem to impact quite an alarming number of users, I have reported the issue seperately here: https://github.com/Gamua/Adobe-Runtime-Support/issues/297
Hopefully we can solve the ANR issue quickly and update larger apps to confirm that the crashes are resolved! Thanks for your support :)
marcanw
commented
4 years ago Still has crash with last AIR 33.1.1.98:
backtrace:
#00 pc 0077193e8fca0000 <unknown>
#01 pc 000000000071ea84 /data/app/air.net.trustmedia.eventpic-0Vh18Oz5fFZ_iYIzsyGilA==/lib/arm64/libCore.so
#02 pc 000000000074b7a4 /data/app/air.net.trustmedia.eventpic-0Vh18Oz5fFZ_iYIzsyGilA==/lib/arm64/libCore.so
#03 pc 0000000000841100 /data/app/air.net.trustmedia.eventpic-0Vh18Oz5fFZ_iYIzsyGilA==/lib/arm64/libCore.so
#04 pc 0000000000838ed0 /data/app/air.net.trustmedia.eventpic-0Vh18Oz5fFZ_iYIzsyGilA==/lib/arm64/libCore.so
#05 pc 0000000000838df0 /data/app/air.net.trustmedia.eventpic-0Vh18Oz5fFZ_iYIzsyGilA==/lib/arm64/libCore.so
#06 pc 0000000000838cec /data/app/air.net.trustmedia.eventpic-0Vh18Oz5fFZ_iYIzsyGilA==/lib/arm64/libCore.so
#07 pc 000000000025ee5c /data/app/air.net.trustmedia.eventpic-0Vh18Oz5fFZ_iYIzsyGilA==/lib/arm64/libCore.so
#08 pc 0000000000262188 /data/app/air.net.trustmedia.eventpic-0Vh18Oz5fFZ_iYIzsyGilA==/lib/arm64/libCore.so
#09 pc 0000000000265a7c /data/app/air.net.trustmedia.eventpic-0Vh18Oz5fFZ_iYIzsyGilA==/lib/arm64/libCore.so
#10 pc 0000000000073d2c /data/app/air.net.trustmedia.eventpic-0Vh18Oz5fFZ_iYIzsyGilA==/oat/arm64/base.odex (offset 0x73000) (com.adobe.air.AndroidActivityWrapper.nativeNotifyPermissionRequestResult [DEDUPED]+140)
#11 pc 00000000000b3c34 /data/app/air.net.trustmedia.eventpic-0Vh18Oz5fFZ_iYIzsyGilA==/oat/arm64/base.odex (offset 0x73000) (com.adobe.air.customHandler.handleMessage+68)
#12 pc 000000000179b324 /system/framework/arm64/boot-framework.oat (offset 0xa43000) (android.os.Handler.dispatchMessage+180)
#13 pc 0000000001f7d908 /system/framework/arm64/boot-framework.oat (offset 0xa43000) (android.os.Looper.loop+1304)
#14 pc 0000000000f27688 /system/framework/arm64/boot-framework.oat (offset 0xa43000) (android.app.ActivityThread.main+664)
#15 pc 000000000055764c /system/lib64/libart.so (art_quick_invoke_static_stub+604)
#16 pc 00000000000cfce8 /system/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+232)
#17 pc 000000000045dd7c /system/lib64/libart.so (art::(anonymous namespace)::InvokeWithArgArray(art::ScopedObjectAccessAlreadyRunnable const&, art::ArtMethod*, art::(anonymous namespace)::ArgArray*, art::JValue*, char const*)+104)
#18 pc 000000000045f7d0 /system/lib64/libart.so (art::InvokeMethod(art::ScopedObjectAccessAlreadyRunnable const&, _jobject*, _jobject*, _jobject*, unsigned long)+1440)
#19 pc 00000000003ef398 /system/lib64/libart.so (art::Method_invoke(_JNIEnv*, _jobject*, _jobject*, _jobjectArray*)+52)
#20 pc 000000000078eed4 /system/framework/arm64/boot-core-oj.oat (offset 0x2dc000) (java.lang.Class.getDeclaredMethodInternal [DEDUPED]+180)
#21 pc 000000000247cc48 /system/framework/arm64/boot-framework.oat (offset 0xa43000) (com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run+136)
#22 pc 0000000002488fe8 /system/framework/arm64/boot-framework.oat (offset 0xa43000) (com.android.internal.os.ZygoteInit.main+2104)
#23 pc 000000000055764c /system/lib64/libart.so (art_quick_invoke_static_stub+604)
#24 pc 00000000000cfce8 /system/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+232)
#25 pc 000000000045dd7c /system/lib64/libart.so (art::(anonymous namespace)::InvokeWithArgArray(art::ScopedObjectAccessAlreadyRunnable const&, art::ArtMethod*, art::(anonymous namespace)::ArgArray*, art::JValue*, char const*)+104)
#26 pc 000000000045d9dc /system/lib64/libart.so (art::InvokeWithVarArgs(art::ScopedObjectAccessAlreadyRunnable const&, _jobject*, _jmethodID*, std::__va_list)+424)
#27 pc 0000000000362cd8 /system/lib64/libart.so (art::JNI::CallStaticVoidMethodV(_JNIEnv*, _jclass*, _jmethodID*, std::__va_list)+652)
#28 pc 00000000000b2b74 /system/lib64/libandroid_runtime.so (_JNIEnv::CallStaticVoidMethod(_jclass*, _jmethodID*, ...)+116)
#29 pc 00000000000b5710 /system/lib64/libandroid_runtime.so (android::AndroidRuntime::start(char const*, android::Vector<android::String8> const&, bool)+924)
#30 pc 0000000000002528 /system/bin/app_process64 (main+2012)
#31 pc 00000000000c9270 /system/lib64/libc.so (__libc_init+88)
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> air.net.trustmedia.eventpic <<<
backtrace:
#00 pc 0000000000285acc /data/app/air.net.trustmedia.eventpic-WkJWs1sopcQ8cyf-jOKGjw==/lib/arm64/libCore.so
#01 pc 000000000057f368 /data/app/air.net.trustmedia.eventpic-WkJWs1sopcQ8cyf-jOKGjw==/lib/arm64/libCore.so
#02 pc 00000000005d5e48 /data/app/air.net.trustmedia.eventpic-WkJWs1sopcQ8cyf-jOKGjw==/lib/arm64/libCore.so
#03 pc 00000000000086b4 <anonymous>Any news about these two last crashes?
ajwfrost
commented
4 years ago Hi @marcanw - the first of those crashes was during garbage collection (well, ZCT) clean-up with a regular expression object - an internal structure had a very odd memory pointer to end us up at that memory address in PC! Not sure what the root cause of that one is - it seems to be memory corruption but programmatically there should be no way for this to happen..
The second issue here is another instance of the problem in the sensor event source destructor. We've checked with Distriqt and it's not the same mechanism that they use (orientation sensor). Would it be possible to get a build of your application with the apk-debug version, so that we can start looking at what's creating the sensor event source?
thanks
ajwfrost
commented
4 years ago Hi
Thanks for the APK. From a quick look, I'm wondering whether it's to do with the class you've got that extends from flash.sensors.Geolocation. There's nothing obviously wrong in there but it looks like this is the class that's being cleaned up when we hit the error.
A couple of suggestions in case these help (but we can look into it from the C++ side too): 1) in 'startGeo', could you use a normal instance method for the permission status handler rather than an anonymous function? 2) it may also be worth not calling the 'removeEventListener' for the permission status, as the code looks to me like it only expects the removal of 'status' or 'update' events..! this part we'll certainly look at, I'm not 100% sure if it's why the crash is happening but the C++ code looks wrong to me here..
You're also using the Accelerometer but this is a simpler class with just the 'update' events listened to.
Anyway - we'll investigate further and try to reproduce it using that sort of gelocation extension..
thanks
marcanw
commented
4 years ago Thx ajwfrost I did as you say. I guess I should check the Google Play Console to see if I see an improvement in the crash logs? Do you want a new build?
neuronix
commented
4 years ago Hi @ajwfrost Thank you very much for your hard work, we now have a crash rate of 0.21% on our biggest app! :partying_face:
mumeka
commented
4 years ago This bug creates most of our crashes, how do you get these detailed crash reports? Release build ARM 32 bit reports.
It is worse with 31.1.1.98 and we are using 32 bit release. Soon will have to use 64 bit release, as I read it can be much worse with 64 bit. How can I get more info on stack traces.
Samsung Galaxy On Max (j7maxlte), 3840MB RAM, Android 8.1
Rapor 1
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> air.com.company.app <<<
backtrace:
#00 pc 0000000000887ec8 /data/app/air.com.company.app-fLHn2VS9w57Y2S4Y5NNVOw==/lib/arm/libCore.so
Samsung Galaxy A20s (a20s), 3584MB RAM, Android 10
Rapor 2
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> air.com.company.app <<<
backtrace:
#00 pc 0000000000887e20 /data/app/air.com.company.app-jndS8Gv_skqMcSdNsULsEA==/lib/arm/libCore.so
Huawei Honor 8A (HWJAT-M), 2048MB RAM, Android 9
Rapor 3
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> air.com.company.app <<<
backtrace:
#00 pc 0000000000887e20 /data/app/air.com.company.app--LmmWrKfbCHEUWFehaSbtg==/lib/arm/libCore.so
Huawei HUAWEI P20 Lite (HWANE), 3840MB RAM, Android 9
Rapor 4
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> air.com.company.app <<<
backtrace:
#00 pc 0000000000887f4c /data/app/air.com.company.app-4yUYfg9c_yRkMbZvHnabvw==/lib/arm/libCore.so
Samsung Galaxy M20 (m20lte), 3840MB RAM, Android 10
Rapor 5
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> air.com.company.app <<<
backtrace:
#00 pc 0000000000887f4c /data/app/air.com.company.app-KJXbtPlCf5tYugIiqVENQQ==/lib/arm/libCore.so
Samsung Galaxy S9 (starlte), 3840MB RAM, Android 10
Rapor 6
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> air.com.company.app <<<
backtrace:
#00 pc 00000000008ac480 /data/app/air.com.company.app-V_1Gxxg3_I8S6i9drYG-dw==/lib/arm/libCore.so
Huawei Y9 Prime 2019 (HWSTK-HF), 3840MB RAM, Android 10
Rapor 7
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> air.com.company.app <<<
backtrace:
#00 pc 00000000008880dc /data/app/air.com.company.app-AMCY05GLqHzPh-bG-VjOSg==/lib/arm/libCore.so
Samsung Galaxy S7 edge (hero2lte), 3584MB RAM, Android 8.0
Rapor 8
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> air.com.company.app <<<
backtrace:
#00 pc 0000000000887f4c /data/app/air.com.company.app-ZTVEZ8KbI1ALxU8o3Ok86A==/lib/arm/libCore.so
Huawei Honor 8A (HWJAT-M), 2048MB RAM, Android 9
Rapor 9
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> air.com.company.app <<<
backtrace:
#00 pc 0000000000887e20 /data/app/air.com.company.app-40rG-RIGWuWVvqJfPU2XXA==/lib/arm/libCore.so
Huawei HUAWEI Y6 Pro 2019 (HWMRD-M1), 2816MB RAM, Android 9
Rapor 10
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> air.com.company.app <<<
backtrace:
#00 pc 0000000000887f4c /data/app/air.com.company.app-iZgvSoBoQfEMBX5rtXITRw==/lib/arm/libCore.so
Samsung Galaxy A20s (a20s), 3584MB RAM, Android 10
Rapor 11
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> air.com.company.app <<<
backtrace:
#00 pc 0000000000887f4c /data/app/air.com.company.app-hfn_DJ0r-vtCg9T-VQu2YQ==/lib/arm/libCore.so
Vestel V3 5580 Dual (Orsa), 3072MB RAM, Android 7.0
Rapor 12
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> air.com.company.app <<<
backtrace:
#00 pc 0000000000887f4c /data/app/air.com.company.app-1/lib/arm/libCore.so
Samsung Galaxy A10s (a10s), 1792MB RAM, Android 9
Rapor 13
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> air.com.company.app <<<
backtrace:
#00 pc 0000000000887e20 /data/app/air.com.company.app-M3JMj0Lv0fTO2cPufX6PVA==/lib/arm/libCore.so
Huawei Mate 10 lite (HWRNE), 3840MB RAM, Android 8.0
Rapor 14
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> air.com.company.app <<<
backtrace:
#00 pc 0000000000887f4c /data/app/air.com.company.app-VCf44PiiBHalZW3bNk1UqA==/lib/arm/libCore.so
Huawei honor 10 Lite (HWHRY-H), 2816MB RAM, Android 10
Rapor 15
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> air.com.company.app <<<
backtrace:
#00 pc 00000000008ac480 /data/app/air.com.company.app-Ci78OF61KTsE1JQFCKvGig==/lib/arm/libCore.so
Samsung Galaxy J7 Pro (j7y17lte), 3072MB RAM, Android 9
Rapor 16
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> air.com.company.app <<<
backtrace:
#00 pc 0000000000887f4c /data/app/air.com.company.app-4j0zsl1EXpeNvspCO7kJHA==/lib/arm/libCore.so
Samsung Galaxy A2 Core (a2corelte), 1024MB RAM, Android 8.1
Rapor 17
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> air.com.company.app <<<
backtrace:
#00 pc 00000000008881a4 /data/app/air.com.company.app-4gPpQaT9i4GDvtYVL7iTxw==/lib/arm/libCore.so
Huawei HUAWEI Y6 Pro 2019 (HWMRD-M1), 2816MB RAM, Android 9
Rapor 18
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> air.com.company.app <<<
backtrace:
#00 pc 0000000000887f4c /data/app/air.com.company.app-mCOgylc3FhSLZtTFrPWLjQ==/lib/arm/libCore.so
Huawei HUAWEI P30 lite (HWMAR), 3840MB RAM, Android 10
Rapor 19
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> air.com.company.app <<<
backtrace:
#00 pc 0000000000887ec8 /data/app/air.com.company.app-rI0CHdkMJBePE0I-OJ53Gw==/lib/arm/libCore.so
Huawei honor 10 Lite (HWHRY-H), 2816MB RAM, Android 10
Rapor 20
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> air.com.company.app <<<
backtrace:
#00 pc 00000000008881a4 /data/app/air.com.company.app-qrS6L9MuSBJdisSmNdIZ4w==/lib/arm/libCore.soMore reports from another slot:
Oppo CPH1909 (CPH1909), 2816MB RAM, Android 8.1
Rapor 1 / 5
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> air.com.company.app <<<
backtrace:
#00 pc 0000000000888008 /data/app/air.com.company.app-XXMQTNKAh23vvHITMztaEA==/lib/arm/libCore.so
#01 pc 00000000d1d7f020 <unknown>
Huawei HUAWEI Y5 lite (HWDRA-MG), 1024MB RAM, Android 8.1
Rapor 2 / 5
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> air.com.company.app <<<
backtrace:
#00 pc 0000000000887ec8 /data/app/air.com.company.app-qUbiYQGK7P-SYhAXaY0qRg==/lib/arm/libCore.so
#01 pc 0000000000000000 <unknown>
VIA_G3, Android 8.1
Rapor 3 / 5
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> air.com.company.app <<<
backtrace:
#00 pc 0000000000887ec8 /data/app/air.com.company.app-cwxjKi6ZT2qeQ4y0YFKb_g==/lib/arm/libCore.so
#01 pc 0000000000000000 <unknown>
TCT (Alcatel) Alcatel 3 (A3A), 2048MB RAM, Android 8.1
Rapor 4 / 5
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> air.com.company.app <<<
backtrace:
#00 pc 0000000000887f4c /data/app/air.com.company.app-y_oTecyLytrT1GSXMsnJOA==/lib/arm/libCore.so
#01 pc 0000000062bfa335 <unknown>
VIA_G3, Android 8.1
Rapor 5 / 5
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> air.com.company.app <<<
backtrace:
#00 pc 0000000000887f4c /data/app/air.com.company.app-CO_0lDcb55pAqiP4jGLbxg==/lib/arm/libCore.so
#01 pc 00000000f16dfe0e <unknown>
Oppo CPH1909 (CPH1909), 2816MB RAM, Android 8.1
Rapor 1 / 1
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> air.com.company.app <<<
backtrace:
#00 pc 0000000000887f4c /data/app/air.com.company.app-PQ727XmCEOpJacoHZbqmcg==/lib/arm/libCore.so
#01 pc 0000000000179b3d [anon:libc_malloc:b2780000]
Oppo CPH1909 (CPH1909), 2816MB RAM, Android 8.1
Rapor 1 / 1
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> air.com.company.app <<<
backtrace:
#00 pc 00000000008881a4 /data/app/air.com.company.app-yxZqYkO5EsSEutsblWgq9g==/lib/arm/libCore.so
#01 pc 000000000003daf0 <anonymous>
Samsung Galaxy A30s (a30s), 3840MB RAM, Android 10
Rapor 1 / 1
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> air.com.company.app <<<
backtrace:
#00 pc 0000000000178bae /data/app/air.com.company.app-5utmfUO9ZJtrb6gf9mrxIA==/lib/arm/libCore.so
#01 pc 00000000002fc1a3 /data/app/air.com.company.app-5utmfUO9ZJtrb6gf9mrxIA==/lib/arm/libCore.so
#02 pc 000000000036d463 /data/app/air.com.company.app-5utmfUO9ZJtrb6gf9mrxIA==/lib/arm/libCore.so
#03 pc 00000000000000db <anonymous>
Redmi Redmi K20 (davinci), 5632MB RAM, Android 10
Rapor 1 / 4
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> air.com.company.app <<<
backtrace:
#00 pc 00000000f6a56450 <unknown>
#01 pc 00000000008a5b7c /data/app/air.com.company.app-HuZ_IA-VQrfuYRJ6tzQOPA==/lib/arm/libCore.so
Samsung Galaxy On Max (j7maxlte), 3840MB RAM, Android 8.1
Rapor 1 / 1
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> air.com.company.app <<<
backtrace:
#00 pc 0000000063675f6a <unknown>
#01 pc 0000000001989831 /data/app/com.android.chrome-JDpEKwgpP_tV3bT9btZs9g==/base.apk (offset 0xbe7000)
Samsung Galaxy Tab E (gtelwifi), 1536MB RAM, Android 4.4
Rapor 1 / 2
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> air.com.company.app <<<
backtrace:
#00 pc 0000000000000002 <unknown>
#01 pc 000000000018d41b /system/lib/libwebviewchromium.so
#02 pc 0000000000194521 /system/lib/libwebviewchromium.so
#03 pc 000000000001dd8c /system/lib/libdvm.so (dvmPlatformInvoke+112)
#04 pc 000000000004e283 /system/lib/libdvm.so (dvmCallJNIMethod(unsigned int const*, JValue*, Method const*, Thread*)+398)
#05 pc 00000000000271a0 /system/lib/libdvm.so
#06 pc 000000000002e17c /system/lib/libdvm.so (dvmMterpStd(Thread*)+76)
#07 pc 000000000002b828 /system/lib/libdvm.so (dvmInterpret(Thread*, Method const*, JValue*)+184)
#08 pc 0000000000060b55 /system/lib/libdvm.so (dvmInvokeMethod(Object*, Method const*, ArrayObject*, ArrayObject*, ClassObject*, bool)+392)
#09 pc 0000000000068af7 /system/lib/libdvm.so
#10 pc 00000000000271a0 /system/lib/libdvm.so
#11 pc 000000000002e17c /system/lib/libdvm.so (dvmMterpStd(Thread*)+76)
#12 pc 000000000002b828 /system/lib/libdvm.so (dvmInterpret(Thread*, Method const*, JValue*)+184)
#13 pc 0000000000060873 /system/lib/libdvm.so (dvmCallMethodV(Thread*, Method const*, Object*, bool, JValue*, std::__va_list)+338)
#14 pc 0000000000049e6b /system/lib/libdvm.so
#15 pc 0000000000051d0f /system/lib/libandroid_runtime.so
#16 pc 0000000000053607 /system/lib/libandroid_runtime.so (android::AndroidRuntime::start(char const*, char const*, bool)+358)
#17 pc 0000000000001063 /system/bin/app_process
#18 pc 000000000000e463 /system/lib/libc.so (__libc_init+50)
#19 pc 0000000000000d80 /system/bin/app_processI can copy hundreds more each day, however none have more details. Please fix this issue.
ajwfrost
commented
4 years ago Hi
Thanks for the details. There are a few in here that aren't AIR, and one odd one which is happening during the disposal of a VertexBuffer3D object, but the vast majority of all those earlier ones are crashes within the openssl library that is used by AIR.
Are you doing activities that would involve any encryption or signature validation etc? Am wondering how we can try to reproduce these..
If you get a reproduction on a device that you have access to, then you may be able to get the 'tombstone' error report - typically if you've got the device set up with developer access then you can just do 'send bug report' to get at all the details of a crashed process including call stacks across the different threads, registry and memory information, etc, which is often invaluable!
We'll look a bit more closely into the integration of openssl though, to see if there's anything we can spot in the function that seems to be the most common culprit..
thanks
Andrew
mumeka
commented
4 years ago What is the first one which is one line I copied 20 times? We can work especially on it. Maybe I can find a way to trigger it. I dont know what it is. Without it app crash rate would go down 0.10% probably, right now it is 2%.
App uses facebook, google admob, application, in app billing ANEs from Distriqt. Also a sound ANE to use native sounds. Actionscript side connects to game server via socket without XML overlay. Also game downloads images from game server via HTTPS connection to show user profile images. So some encryption is here probably.
I want trigger first one in developer mode, what is it?
mumeka
commented
4 years ago Also can an error from Feathers or Starling create such a crash? Is so I can update them to see if things improve.
ajwfrost
commented
4 years ago Yes, that first one is an SHA1 operation within OpenSSL... It shouldn't be related to Starling/Feathers but may be something to do with HTTPS - or particularly secure sockets?
mumeka
commented
4 years ago It looks like app's 5 years old crash ratio fixed positively. Thank you @ajwfrost
ajwfrost
commented
4 years ago Okay thanks for this - so it definitely looks like this is related to the HTTPS access for Facebook/game server. We can try doing some focused testing around this sort of thing to see if we can reproduce..
mumeka
commented
4 years ago Facebook graph api call is like this:
https://graph.facebook.com/4/picture?type=normal&width=200&height=200 https://graph.facebook.com/4/picture?type=square&width=50&height=50
even http redirects to https:
http://graph.facebook.com/4/picture?type=square&width=50&height=50
sometimes Facebook decides to send the image file, triggers download.
In the application, sometimes 50+ profile images needed at same time to show user lists. However while I'm playing with different devices, game does not crash.
Additionally game server uses letsencrypt SSL, I digged into SSL information from Firefox and deepest layer of SSL info seems to use SHA1, however I'm not an expert on this issue.
Michoko92
commented
4 years ago I'm also experiencing those weird crashes, that almost reach the critical acceptable threshold on the Play Store. I'm afraid my app will disappear from search results because of that. I reported this issue a while back here: https://forum.starling-framework.org/d/20888-signal-11-errors . The errors had seemed to decrease for a little while, but they now increased again. Now the errors are slightly different though, and switched from Code 1 to Code 2:
signal 11 (SIGSEGV), code 2 (SEGV_ACCERR)
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> air.com.indiegoes.oraclecards <<<
backtrace:
#00 pc 0000000000887e20 /data/app/air.com.indiegoes.oraclecards-Ug4Yw83k-S0__8h_4kcv0w==/lib/arm/libCore.soThis is the only information I get from the Play Store bug reports.
I use AIR 33.1.1.98. Here is my app manifest:
<android>
<containsVideo>true</containsVideo>
<manifestAdditions>
<![CDATA[
<manifest android:installLocation="auto">
<uses-sdk android:targetSdkVersion="28"/>
<uses-sdk android:minSdkVersion="15"/>
<uses-permission android:name="android.permission.INTERNET"/>
<uses-permission android:name="android.permission.ACCESS_NETWORK_STATE"/>
<uses-permission android:name="com.android.vending.BILLING" />
<uses-permission android:name="android.permission.WAKE_LOCK"/>
<uses-permission android:name="android.permission.DISABLE_KEYGUARD"/>
<uses-configuration android:reqFiveWayNav="false"/>
<supports-screens android:smallScreens="true"
android:normalScreens="true"
android:largeScreens="true"
android:xlargeScreens="true"/>
<uses-feature android:required="false" android:name="android.hardware.touchscreen.multitouch"/>
<application android:enabled="true" android:allowClearUserData="true" android:allowBackup="true" android:roundIcon="@mipmap/ic_launcher_round" >
<meta-data android:name="android.max_aspect" android:value="2.16" />
<activity android:name="com.distriqt.extension.share.permissions.AuthorisationActivity" android:theme="@android:style/Theme.Translucent.NoTitleBar" />
<activity
android:name="com.distriqt.extension.inappbilling.activities.ProductViewActivity"
android:theme="@android:style/Theme.Translucent.NoTitleBar">
</activity>
<!-- GOOGLE PLAY BILLING -->
<activity
android:name="com.android.billingclient.api.ProxyBillingActivity"
android:configChanges="keyboard|keyboardHidden|screenLayout|screenSize|orientation"
android:theme="@android:style/Theme.Translucent.NoTitleBar" />
<provider
android:name="com.distriqt.extension.share.content.ShareFileProvider"
android:authorities="air.com.indiegoes.oraclecards.sharefileprovider"
android:grantUriPermissions="true"
android:exported="false">
<meta-data
android:name="android.support.FILE_PROVIDER_PATHS"
android:resource="@xml/distriqt_share_paths" />
</provider>
</application>
</manifest>
]]>
</manifestAdditions>
</android>
<extensions>
<extensionID>com.distriqt.Core</extensionID>
<extensionID>com.distriqt.InAppBilling</extensionID>
<extensionID>com.distriqt.Share</extensionID>
<extensionID>com.distriqt.Application</extensionID>
<extensionID>com.distriqt.ApplicationRater</extensionID>
<extensionID>com.distriqt.Dialog</extensionID>
<extensionID>com.distriqt.androidsupport.V4</extensionID>
<extensionID>com.distriqt.androidsupport.AppCompatV7</extensionID>
</extensions>I don't use ads, and no StageWebView. I have other apps based on the same engine, but they work fine. The main difference is that the one crashing downloads assets from a web server, and then loads them from local storage. I think I put all proper error handling so it should never crash. Would you have some suggestions to help identify this issue, please? Would HTTPS access to my server be the culprit, as suggested above? Thank you!
ajwfrost
commented
4 years ago Thanks .. so this also looks similar, that crash location is in the same function within the SSL code. Definitely we will look at this .. I had wondered if it's a thread-safety issue, but they're all looking like they're crashing on the main thread (pid: 0 tid: 0). Have added an internal task, it should be fairly quick to update this at least partially - we're planning an update release around August that this should then get into..
Michoko92
commented
4 years ago Oh awesome, I didn't know this little info would allow you to identify so many things, it's encouraging! I just hope my app won't be blacklisted by Google in the meantime. Thank you for your continued support, and hopefully it will solve this long-standing issue. Fingers crossed!
hardcoremore
commented
4 years ago we're planning an update release around August that this should then get into
@ajwfrost Is release at the end of this month still in plan?
ajwfrost
commented
4 years ago @hardcoremore we’be had two showstoppers identified by QA - one is fixed, hoping to solve the other at the start of the week so it’s just a little delayed :-(
Ever since migrating to targetSDK="28" as required by Google, our apps are plagued with these errors, causing up to 9% of sessions with crashes on Android 9. The bug does sometimes occur on other versions but 95% of times, it's Android 9.
With exactly the same ANEs, setup & AIR SDK, we had just about none of these errors using targetSDK=19.
This is a bug that has been around in AIR for years now (not joking) but it has never been so bad. AIR 33 does not fix the issue either.
Some topics where devs have discussed this issue but failed to find a solution:
Unfortunately the stack traces are pretty poor:
signal 11 (SIGSEGV), code 1 (SEGV_MAPERR) strcmp
` pid: 0, tid: 0 >>> air.com.company.application <<<
backtrace:
00 pc 000000000001cb70 /system/lib64/libc.so (strcmp+24)
01 pc 00000000003b8ddc /data/app/air.com.company.application-1/lib/arm64/libCore.so
02 pc 00000000003bc750 /data/app/air.com.company.application-1/lib/arm64/libCore.so
03 pc 00000000002d2a6c /data/app/air.com.company.application-1/lib/arm64/libCore.so
04 pc 00000000002d3b10 /data/app/air.com.company.application-1/lib/arm64/libCore.so
05 pc 00000000002d64e8 /data/app/air.com.company.application-1/lib/arm64/libCore.so
06 pc 00000000002d9250 /data/app/air.com.company.application-1/lib/arm64/libCore.so
07 pc 00000000006f0308 /data/app/air.com.company.application-1/lib/arm64/libCore.so
08 pc 00000000006ef5a8 /data/app/air.com.company.application-1/lib/arm64/libCore.so
09 pc 0000000000271c38 /data/app/air.com.company.application-1/lib/arm64/libCore.so
10 pc 0000000000271ca8 /data/app/air.com.company.application-1/lib/arm64/libCore.so
11 pc 00000000002719dc /data/app/air.com.company.application-1/lib/arm64/libCore.so
12 pc 00000000000677d4 /system/lib64/libc.so (_ZL15__pthread_startPv+52)
13 pc 000000000001edf0 /system/lib64/libc.so (__start_thread+16)`
I'd like to underline the urgence of this matter. 9% crash rate is really far far over the 1% Playstore bad behavior threshold, so this is hurty our apps very badly because Google reduces exposure of apps with poor crash / anr rates. We also have no workaround since we can no longer revert to a lower targetSDK.
Thanks for your support