airsdk / Adobe-Runtime-Support

Report, track and discuss issues in Adobe AIR. Monitored by Adobe - and HARMAN - and maintained by the AIR community.
206 stars 11 forks source link

Signal 11 Crashes #1717

Open SponsorAds opened 2 years ago

SponsorAds commented 2 years ago

We are experiencing a high amount of crashes (about 5% on average) on Android. SDK version does not seem to matter as this is since release about 9 months ago (our older apps are experiencing the same issues).

Every single crash is "signal 11".

Screenshot 2022-02-28 at 11 11 30

We are not using audio or video through AIR. We are using ssl and are downloading assets through https.

Here some traces, maybe they help a bit.

backtrace:
  #00  pc 00000000000f9898  /data/app/~~x-QbghBBvZlRUmEtZOU_aw==/appid-1Hb9IZ7-EV6E9iXWp6U9kA==/split_config.armeabi_v7a.apk!lib/armeabi-v7a/libCore.so (offset 0x1000)
  #00  pc 00000000000f9ccb  /data/app/~~x-QbghBBvZlRUmEtZOU_aw==/appid-1Hb9IZ7-EV6E9iXWp6U9kA==/split_config.armeabi_v7a.apk!lib/armeabi-v7a/libCore.so (offset 0x1000)
  #00  pc 000000000094af1c  /data/app/~~x-QbghBBvZlRUmEtZOU_aw==/appid-1Hb9IZ7-EV6E9iXWp6U9kA==/split_config.armeabi_v7a.apk!lib/armeabi-v7a/libCore.so (offset 0x1000)
backtrace:
  #00  pc 00000000007aff60  /data/app/~~30UAI8qrxquRn72bV-SFGQ==/appid-zhYpllaoH7dj1UXjFH8KwA==/split_config.arm64_v8a.apk!lib/arm64-v8a/libCore.so (offset 0x1000)
  #00  pc 00000000007dcc38  /data/app/~~30UAI8qrxquRn72bV-SFGQ==/appid-zhYpllaoH7dj1UXjFH8KwA==/split_config.arm64_v8a.apk!lib/arm64-v8a/libCore.so (offset 0x1000)
  #00  pc 00000000007bcfb8  /data/app/~~30UAI8qrxquRn72bV-SFGQ==/appid-zhYpllaoH7dj1UXjFH8KwA==/split_config.arm64_v8a.apk!lib/arm64-v8a/libCore.so (offset 0x1000)
  #00  pc 00000000000089a4  <anonymous>
backtrace:
  #00  pc 00000000002893b0  /data/app/~~dSC1DHO6mKEVaJU37BIKIg==/appid-FtL3WSpPu_dlV2k0OIG8MA==/split_config.arm64_v8a.apk!libCore.so
  #00  pc 00000000002899a4  /data/app/~~dSC1DHO6mKEVaJU37BIKIg==/appid-FtL3WSpPu_dlV2k0OIG8MA==/split_config.arm64_v8a.apk!libCore.so
  #00  pc 0000000000d0c1ac  /data/app/~~dSC1DHO6mKEVaJU37BIKIg==/appid-FtL3WSpPu_dlV2k0OIG8MA==/split_config.arm64_v8a.apk!libCore.so
backtrace:
  #00  pc 000000000084c464  /data/app/~~3SRXhGPad48hPP3Bk9lP6w==/appid-CN-I226yfrFCbt8M8KWw-A==/split_config.arm64_v8a.apk!lib/arm64-v8a/libCore.so (offset 0x1000)
  #00  pc 0000000000843310  /data/app/~~3SRXhGPad48hPP3Bk9lP6w==/appid-CN-I226yfrFCbt8M8KWw-A==/split_config.arm64_v8a.apk!lib/arm64-v8a/libCore.so (offset 0x1000)
  #00  pc 0000000000843230  /data/app/~~3SRXhGPad48hPP3Bk9lP6w==/appid-CN-I226yfrFCbt8M8KWw-A==/split_config.arm64_v8a.apk!lib/arm64-v8a/libCore.so (offset 0x1000)
  #00  pc 000000000084312c  /data/app/~~3SRXhGPad48hPP3Bk9lP6w==/appid-CN-I226yfrFCbt8M8KWw-A==/split_config.arm64_v8a.apk!lib/arm64-v8a/libCore.so (offset 0x1000)
  #00  pc 000000000026e1d4  /data/app/~~3SRXhGPad48hPP3Bk9lP6w==/appid-CN-I226yfrFCbt8M8KWw-A==/split_config.arm64_v8a.apk!lib/arm64-v8a/libCore.so (offset 0x1000)
  #00  pc 0000000000271540  /data/app/~~3SRXhGPad48hPP3Bk9lP6w==/appid-CN-I226yfrFCbt8M8KWw-A==/split_config.arm64_v8a.apk!lib/arm64-v8a/libCore.so (offset 0x1000)
  #00  pc 0000000000274d1c  /data/app/~~3SRXhGPad48hPP3Bk9lP6w==/appid-CN-I226yfrFCbt8M8KWw-A==/split_config.arm64_v8a.apk!lib/arm64-v8a/libCore.so (offset 0x1000)
  #00  pc 000000000008bc5c  /data/app/~~3SRXhGPad48hPP3Bk9lP6w==/appid-CN-I226yfrFCbt8M8KWw-A==/oat/arm64/base.odex (art_jni_trampoline+140)
  #00  pc 00000000000ae344  /data/app/~~3SRXhGPad48hPP3Bk9lP6w==/appid-CN-I226yfrFCbt8M8KWw-A==/oat/arm64/base.odex (com.adobe.air.customHandler.handleMessage+68)
  #00  pc 00000000006b67d4  /system/framework/arm64/boot-framework.oat (android.os.Handler.dispatchMessage+180)
  #00  pc 00000000006b9ccc  /system/framework/arm64/boot-framework.oat (android.os.Looper.loop+1516)
  #00  pc 00000000004556b0  /system/framework/arm64/boot-framework.oat (android.app.ActivityThread.main+768)
  #00  pc 00000000001357e8  /apex/com.android.art/lib64/libart.so (art_quick_invoke_static_stub+568)
  #00  pc 00000000001ab804  /apex/com.android.art/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+228)
  #00  pc 0000000000568924  /apex/com.android.art/lib64/libart.so (art::InvokeMethod(art::ScopedObjectAccessAlreadyRunnable const&, _jobject*, _jobject*, _jobject*, unsigned long)+1364)
  #00  pc 00000000004e7030  /apex/com.android.art/lib64/libart.so (art::Method_invoke(_JNIEnv*, _jobject*, _jobject*, _jobjectArray*)+48)
  #00  pc 000000000008a6f4  /apex/com.android.art/javalib/arm64/boot.oat (art_jni_trampoline+180)
  #00  pc 0000000000aebe28  /system/framework/arm64/boot-framework.oat (com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run+136)
  #00  pc 0000000000af5e0c  /system/framework/arm64/boot-framework.oat (com.android.internal.os.ZygoteInit.main+2444)
  #00  pc 00000000001357e8  /apex/com.android.art/lib64/libart.so (art_quick_invoke_static_stub+568)
  #00  pc 00000000001ab804  /apex/com.android.art/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+228)
  #00  pc 0000000000567340  /apex/com.android.art/lib64/libart.so (art::JValue art::InvokeWithVarArgs<art::ArtMethod*>(art::ScopedObjectAccessAlreadyRunnable const&, _jobject*, art::ArtMethod*, std::__va_list)+448)
  #00  pc 0000000000567804  /apex/com.android.art/lib64/libart.so (art::JValue art::InvokeWithVarArgs<_jmethodID*>(art::ScopedObjectAccessAlreadyRunnable const&, _jobject*, _jmethodID*, std::__va_list)+92)
  #00  pc 0000000000449804  /apex/com.android.art/lib64/libart.so (art::JNI<true>::CallStaticVoidMethodV(_JNIEnv*, _jclass*, _jmethodID*, std::__va_list)+652)
  #00  pc 000000000009c48c  /system/lib64/libandroid_runtime.so (_JNIEnv::CallStaticVoidMethod(_jclass*, _jmethodID*, ...)+124)
  #00  pc 00000000000a4480  /system/lib64/libandroid_runtime.so (android::AndroidRuntime::start(char const*, android::Vector<android::String8> const&, bool)+856)
  #00  pc 00000000000035a0  /system/bin/app_process64 (main+1368)
  #00  pc 00000000000854b4  /apex/com.android.runtime/lib64/bionic/libc.so (__libc_init+108)
backtrace:
  #00  pc 000000000004b2c8  <anonymous>
  #00  pc 000000000084c540  /data/app/~~Tq5e0qPkwpXATP8NaAzjyg==/appid-aOVH1R4RzyFZwBeelWbWkA==/split_config.arm64_v8a.apk!lib/arm64-v8a/libCore.so (offset 0x1000)
  #00  pc 0000000000843310  /data/app/~~Tq5e0qPkwpXATP8NaAzjyg==/appid-aOVH1R4RzyFZwBeelWbWkA==/split_config.arm64_v8a.apk!lib/arm64-v8a/libCore.so (offset 0x1000)
  #00  pc 0000000000843230  /data/app/~~Tq5e0qPkwpXATP8NaAzjyg==/appid-aOVH1R4RzyFZwBeelWbWkA==/split_config.arm64_v8a.apk!lib/arm64-v8a/libCore.so (offset 0x1000)
  #00  pc 000000000084312c  /data/app/~~Tq5e0qPkwpXATP8NaAzjyg==/appid-aOVH1R4RzyFZwBeelWbWkA==/split_config.arm64_v8a.apk!lib/arm64-v8a/libCore.so (offset 0x1000)
  #00  pc 000000000026e1d4  /data/app/~~Tq5e0qPkwpXATP8NaAzjyg==/appid-aOVH1R4RzyFZwBeelWbWkA==/split_config.arm64_v8a.apk!lib/arm64-v8a/libCore.so (offset 0x1000)
  #00  pc 0000000000271540  /data/app/~~Tq5e0qPkwpXATP8NaAzjyg==/appid-aOVH1R4RzyFZwBeelWbWkA==/split_config.arm64_v8a.apk!lib/arm64-v8a/libCore.so (offset 0x1000)
  #00  pc 0000000000274d1c  /data/app/~~Tq5e0qPkwpXATP8NaAzjyg==/appid-aOVH1R4RzyFZwBeelWbWkA==/split_config.arm64_v8a.apk!lib/arm64-v8a/libCore.so (offset 0x1000)
  #00  pc 00000000000a5c5c  /data/app/~~Tq5e0qPkwpXATP8NaAzjyg==/appid-aOVH1R4RzyFZwBeelWbWkA==/oat/arm64/base.odex (art_jni_trampoline+140)
  #00  pc 0000000000126ce4  /data/app/~~Tq5e0qPkwpXATP8NaAzjyg==/appid-aOVH1R4RzyFZwBeelWbWkA==/oat/arm64/base.odex (com.adobe.air.customHandler.handleMessage+68)
  #00  pc 0000000000637ce4  /system/framework/arm64/boot-framework.oat (android.os.Handler.dispatchMessage+180)
  #00  pc 000000000063b304  /system/framework/arm64/boot-framework.oat (android.os.Looper.loop+1812)
  #00  pc 00000000003fdf90  /system/framework/arm64/boot-framework.oat (android.app.ActivityThread.main+752)
  #00  pc 00000000001337e8  /apex/com.android.art/lib64/libart.so (art_quick_invoke_static_stub+568)
  #00  pc 00000000001a8a94  /apex/com.android.art/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+228)
  #00  pc 0000000000555738  /apex/com.android.art/lib64/libart.so (art::InvokeMethod(art::ScopedObjectAccessAlreadyRunnable const&, _jobject*, _jobject*, _jobject*, unsigned long)+1364)
  #00  pc 00000000004d4ec4  /apex/com.android.art/lib64/libart.so (art::Method_invoke(_JNIEnv*, _jobject*, _jobject*, _jobjectArray*)+52)
  #00  pc 00000000000896f4  /apex/com.android.art/javalib/arm64/boot.oat (art_jni_trampoline+180)
  #00  pc 00000000008905e8  /system/framework/arm64/boot-framework.oat (com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run+136)
  #00  pc 0000000000898d18  /system/framework/arm64/boot-framework.oat (com.android.internal.os.ZygoteInit.main+2280)
  #00  pc 00000000001337e8  /apex/com.android.art/lib64/libart.so (art_quick_invoke_static_stub+568)
  #00  pc 00000000001a8a94  /apex/com.android.art/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+228)
  #00  pc 0000000000554174  /apex/com.android.art/lib64/libart.so (art::JValue art::InvokeWithVarArgs<art::ArtMethod*>(art::ScopedObjectAccessAlreadyRunnable const&, _jobject*, art::ArtMethod*, std::__va_list)+448)
  #00  pc 0000000000554628  /apex/com.android.art/lib64/libart.so (art::JValue art::InvokeWithVarArgs<_jmethodID*>(art::ScopedObjectAccessAlreadyRunnable const&, _jobject*, _jmethodID*, std::__va_list)+92)
  #00  pc 0000000000438adc  /apex/com.android.art/lib64/libart.so (art::JNI<true>::CallStaticVoidMethodV(_JNIEnv*, _jclass*, _jmethodID*, std::__va_list)+656)
  #00  pc 000000000009a424  /system/lib64/libandroid_runtime.so (_JNIEnv::CallStaticVoidMethod(_jclass*, _jmethodID*, ...)+124)
  #00  pc 00000000000a2260  /system/lib64/libandroid_runtime.so (android::AndroidRuntime::start(char const*, android::Vector<android::String8> const&, bool)+836)
  #00  pc 0000000000003674  /system/bin/app_process64 (main+1580)
  #00  pc 00000000000499e4  /apex/com.android.runtime/lib64/bionic/libc.so (__libc_init+108)
lwmirkk commented 2 years ago

Hi! Have you tryed the last SDK 33.1.1.779?

SponsorAds commented 2 years ago

We currently use 743 as the patch notes don't suggest anything major besides audio issues. If @ajwfrost can confirm it might help I can try it.

I'm currently hesistant pushing too many app updates as our newest app has an episodic release and users get mad on every update without new content. Every app update generates 1 star reviews with "where is episode x". Fun times.

lwmirkk commented 2 years ago

Could be related to: https://github.com/airsdk/Adobe-Runtime-Support/issues/1619

Probably fixed in .779 (https://airsdk.harman.com/release_notes)

ajwfrost commented 2 years ago

Hi

The first and third call stacks there are a null exception happening in the HTTPS certificate handling callback, which we believe were caused by a race condition when setting up the configuration; we've got a change for this (#1619) which went into the .779 build.

The second is just an "equals" operator within some interpreted code, so very tricky to see what may have caused it, but we can see if it's something we can protect against to at least avoid the crash!

The fourth one is a crash within the ZCT "reap" at the end of a frame activity. Strange for this to be crashing, again we can't tell what objects were involved but the crash happens within the reap function itself so we may be able to protect against it. The fifth one is happening within a function that's called from the ZCT "reap" so may be linked, but indicates perhaps that a destructor has gone wrong.

It's possible I suppose that all of these are linked i.e. the problem in the network certificate handling is then causing a knock-on effect in the ZCT reap, but I'm not 100% sure whether that's actually possible... We've not seen many issues with ZCT though (and it's not like this code has changed in the past several years!) so it's more likely some sort of other object destructor isn't working properly. ZCT = Zero Count Table i.e. the easy way to do garbage collection, if there are any objects with zero reference counts then they are added onto the ZCT which is then cleaned up periodically. If one object has no references but has a reference (the only reference) to another object, then the first object is cleaned up and its destructor then causes the second object to be put onto the ZCT. But if something had gone wrong and the second object was already cleaned up, then we'd have an invalid reference in the ZCT i.e. a pointer into already-freed memory which would cause an access error.

I don't suppose you have any ANEs that use native C++ (.so) libraries? These can cause some issues with garbage collection sometimes (or they used to, although it seems Adobe have added a fair amount of protection to avoid this now...)

Just seen the above post added whilst I've been writing this! So yes, one of the issues is likely fixed in build 779 but there are also issues with that build, we've had reports of a consistent crash-on-start-up with the 32-bit ARM version with some text fields on some phones (we've worked out the cause so will be releasing an update that should then bring back some stability, at which point we'll move that to the "downloads" page, currently we're still looking at build 743 as the one that most people should take....)

thanks

SponsorAds commented 2 years ago

List of ANE used, nothing custom or surprising, but a good amount:

<extensionID>com.distriqt.InAppBilling</extensionID>
<extensionID>com.distriqt.facebook.Core</extensionID>
<extensionID>com.distriqt.facebook.Share</extensionID>
<extensionID>com.distriqt.Bolts</extensionID>
<extensionID>com.distriqt.Application</extensionID>
<extensionID>com.distriqt.Volume</extensionID>
<extensionID>com.distriqt.Core</extensionID>
<extensionID>com.distriqt.playservices.GCM</extensionID>
<extensionID>com.distriqt.playservices.Base</extensionID>
<extensionID>com.distriqt.playservices.Ads</extensionID>
<extensionID>com.distriqt.CustomResources</extensionID>
<extensionID>com.distriqt.MediaPlayer</extensionID>
<extensionID>com.distriqt.ZipUtils</extensionID>
<extensionID>com.distriqt.NetworkInfo</extensionID>
<extensionID>com.distriqt.Firebase</extensionID>
<extensionID>com.distriqt.firebase.Performance</extensionID>
<extensionID>com.distriqt.firebase.RemoteConfig</extensionID>
<extensionID>com.google.firebase.core</extensionID>
<extensionID>com.google.protobuflite</extensionID>
<extensionID>androidx.core</extensionID>
<extensionID>androidx.appcompat</extensionID>
<extensionID>androidx.browser</extensionID>
<extensionID>androidx.cardview</extensionID>
<extensionID>androidx.recyclerview</extensionID>
<extensionID>androidx.multidex</extensionID>
<extensionID>com.google.android.material</extensionID>
<extensionID>com.google.android.datatransport</extensionID>
<extensionID>com.google.dagger</extensionID>
<extensionID>com.distriqt.Notifications</extensionID>
<extensionID>com.distriqt.PushNotifications</extensionID>
<extensionID>com.distriqt.playservices.AdsIdentifier</extensionID>
<extensionID>com.distriqt.IDFA</extensionID>
<extensionID>androidx.constraintlayout</extensionID>
<extensionID>androidx.room</extensionID>
<extensionID>androidx.work</extensionID>
<extensionID>androidx.vectordrawable</extensionID>
<extensionID>com.google.code.gson</extensionID>
<extensionID>com.distriqt.Adverts</extensionID>
<extensionID>com.distriqt.admob.FacebookAudience</extensionID>
<extensionID>com.distriqt.admob.IronSource</extensionID>
<extensionID>com.distriqt.admob.AppLovin</extensionID>
<extensionID>com.distriqt.admob.UnityAds</extensionID>
<extensionID>com.android.installreferrer</extensionID>
<extensionID>com.distriqt.playservices.CloudMessaging</extensionID>
<extensionID>com.jetbrains.kotlin</extensionID>
SponsorAds commented 2 years ago

Any movement with them? Or can we do anything to prevent crashes? We updated to .7xx and could shave off 2% crashes (likely all ssl issues). Still at 3-3.5%.

ajwfrost commented 2 years ago

The crashes that were listed above proved somewhat tricky :-( so e.g. there was an exception happening within the "equals" function, but from all the reviews we've done, we can't see how this is crashing unless there's an underlying data problem i.e. where something isn't "null" but is invalid anyway.

Do you know if any of those extensions are using C++? Android extensions are often using Java so don't have direct access to the objects anyway, but with C++ extensions it's possible to do a little more damage..

I'm wondering if this is likely to be something we could attempt to reproduce by putting your application onto a set of devices and running random-press testing on it, or do you have a set sequence of presses that we could go through that might reproduce the problem? If we can get it to reproduce, we can start to add some extra logging and steps to track down where the bad data is coming from.. it's a bit of a needle in a haystack but not necessarily impossible to find..!

thanks

SponsorAds commented 2 years ago

We are using exclusively distriqts ANE. We don't have any such tests ourselves.

I didn't write back that long because I concur with the needdle in a haystack. Currently we are at 8-9% crash rate though and I'd like to pick your @ajwfrost brain for possible causes. Sadly our biggest Signal 11 log has no trace. I am also experiencing high amount of signal 11 crashes on the emulator (sometimes 3 in 5 starts) We are still using .779 (I downgraded, but newer versions have the same results).

backtrace:
  #00  pc 00000000000000b4  <anonymous>
backtrace:
  #00  pc 00000000007ca1f8  /data/app/~~cXe0yqienEaYtk3BlXYcaA==/-mwt5HGSEPkDRAwm2odhwkA==/split_config.arm64_v8a.apk!libCore.so
  #00  pc 00000000007c9eec  /data/app/~~cXe0yqienEaYtk3BlXYcaA==/-mwt5HGSEPkDRAwm2odhwkA==/split_config.arm64_v8a.apk!libCore.so
  #00  pc 00000000007c9b2c  /data/app/~~cXe0yqienEaYtk3BlXYcaA==/-mwt5HGSEPkDRAwm2odhwkA==/split_config.arm64_v8a.apk!libCore.so
  #00  pc 000000000000e344  <anonymous>
backtrace:
  #00  pc 00000000007ca1f8  /data/app/~~WasnN4r0KYxkB5c532-ZxA==/-gdlr6N-8OcLUTpmTNaP1Ug==/split_config.arm64_v8a.apk!lib/arm64-v8a/libCore.so (offset 0x1000)
  #00  pc 00000000007c9eec  /data/app/~~WasnN4r0KYxkB5c532-ZxA==/-gdlr6N-8OcLUTpmTNaP1Ug==/split_config.arm64_v8a.apk!lib/arm64-v8a/libCore.so (offset 0x1000)
  #00  pc 00000000007c9b2c  /data/app/~~WasnN4r0KYxkB5c532-ZxA==/-gdlr6N-8OcLUTpmTNaP1Ug==/split_config.arm64_v8a.apk!lib/arm64-v8a/libCore.so (offset 0x1000)
  #00  pc 000000000000974c  <anonymous>
backtrace:
  #00  pc 00000000001702c8  <anonymous>
  #00  pc 0000000000858da8  /data/app/~~WTmPPvbabCEmmW9xRkXtkg==/-7tPVmgTvlTGW_M2fFKV9gg==/split_config.arm64_v8a.apk!libCore.so
  #00  pc 0000000000850b78  /data/app/~~WTmPPvbabCEmmW9xRkXtkg==/-7tPVmgTvlTGW_M2fFKV9gg==/split_config.arm64_v8a.apk!libCore.so
  #00  pc 0000000000850a98  /data/app/~~WTmPPvbabCEmmW9xRkXtkg==/-7tPVmgTvlTGW_M2fFKV9gg==/split_config.arm64_v8a.apk!libCore.so
  #00  pc 0000000000850994  /data/app/~~WTmPPvbabCEmmW9xRkXtkg==/-7tPVmgTvlTGW_M2fFKV9gg==/split_config.arm64_v8a.apk!libCore.so
  #00  pc 0000000000271750  /data/app/~~WTmPPvbabCEmmW9xRkXtkg==/-7tPVmgTvlTGW_M2fFKV9gg==/split_config.arm64_v8a.apk!libCore.so
  #00  pc 0000000000274abc  /data/app/~~WTmPPvbabCEmmW9xRkXtkg==/-7tPVmgTvlTGW_M2fFKV9gg==/split_config.arm64_v8a.apk!libCore.so
  #00  pc 0000000000278298  /data/app/~~WTmPPvbabCEmmW9xRkXtkg==/-7tPVmgTvlTGW_M2fFKV9gg==/split_config.arm64_v8a.apk!libCore.so
  #00  pc 000000000007ca9c  /data/app/~~WTmPPvbabCEmmW9xRkXtkg==/-7tPVmgTvlTGW_M2fFKV9gg==/oat/arm64/base.odex (art_jni_trampoline+108)
  #00  pc 000000000010d034  /data/app/~~WTmPPvbabCEmmW9xRkXtkg==/-7tPVmgTvlTGW_M2fFKV9gg==/oat/arm64/base.odex (com.adobe.air.customHandler.handleMessage+68)
  #00  pc 00000000007e5a3c  /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot-framework.oat (android.os.Handler.dispatchMessage+188)
  #00  pc 00000000007e8c9c  /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot-framework.oat (android.os.Looper.loopOnce+1036)
  #00  pc 00000000007e87f4  /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot-framework.oat (android.os.Looper.loop+516)
  #00  pc 0000000000564ce0  /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot-framework.oat (android.app.ActivityThread.main+800)
  #00  pc 00000000002ca9e8  /apex/com.android.art/lib64/libart.so (art_quick_invoke_static_stub+568)
  #00  pc 000000000035b5d0  /apex/com.android.art/lib64/libart.so (_jobject* art::InvokeMethod<(art::PointerSize)8>(art::ScopedObjectAccessAlreadyRunnable const&, _jobject*, _jobject*, _jobject*, unsigned long)+608)
  #00  pc 000000000035b348  /apex/com.android.art/lib64/libart.so (art::Method_invoke(_JNIEnv*, _jobject*, _jobject*, _jobjectArray*)+52)
  #00  pc 00000000000b2f74  /apex/com.android.art/javalib/arm64/boot.oat (art_jni_trampoline+132)
  #00  pc 0000000000b3ce5c  /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot-framework.oat (com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run+140)
  #00  pc 0000000000b462b8  /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot-framework.oat (com.android.internal.os.ZygoteInit.main+2376)
  #00  pc 00000000002ca9e8  /apex/com.android.art/lib64/libart.so (art_quick_invoke_static_stub+568)
  #00  pc 000000000044ca04  /apex/com.android.art/lib64/libart.so (art::JValue art::InvokeWithVarArgs<_jmethodID*>(art::ScopedObjectAccessAlreadyRunnable const&, _jobject*, _jmethodID*, std::__va_list)+464)
  #00  pc 000000000062cf30  /apex/com.android.art/lib64/libart.so (art::JNI<true>::CallStaticVoidMethodV(_JNIEnv*, _jclass*, _jmethodID*, std::__va_list)+268)
  #00  pc 00000000000b2ac4  /system/lib64/libandroid_runtime.so (_JNIEnv::CallStaticVoidMethod(_jclass*, _jmethodID*, ...)+120)
  #00  pc 00000000000beb68  /system/lib64/libandroid_runtime.so (android::AndroidRuntime::start(char const*, android::Vector<android::String8> const&, bool)+848)
  #00  pc 00000000000025a0  /system/bin/app_process64 (main+1356)
  #00  pc 000000000004994c  /apex/com.android.runtime/lib64/bionic/libc.so (__libc_init+96)
ajwfrost commented 2 years ago

Hi @spAds2 - thanks for the traces. These are looking very odd .. the first two call stacks (crash at 7ca1f8) are actually where the just-in-time compiler is trying to work out what function to call on a method implemented from an interface (BaseExecMgr::resolveImtSlotFull from https://github.com/adobe/avmplus/blob/master/core/exec-jit.cpp#L731). I don't suppose there is any register information available from a tombstone file or similar, it would be good to know the value of registers 24 (slot) and 21 (cur).. the only thing I'm wondering is whether the loop has continued too long and cur is null, which implies we've not actually got an implementation of the requested method. But that condition should have been caught at verify-time ...

The final trace is another 'ZCT Reap' problem, which is likely a side-effect of something else going on which then perhaps causes a double-delete; it may also be linked to the first issues as the reaping of something that's badly set up like that could be a problem.

If there was any reproducibility we could check this further (e.g. if you're able to reproduce this by automatically launching/random-hammering/closing the application repeatedly) but as it stands, we don't have enough information I'm afraid. We can try adding a null-check and highlighting the condition in the logcat output in case that is the problem with these, but that's about the only suggestion I can make..

thanks

SponsorAds commented 1 year ago

Hi,

after some time, some more symbols.

Currently the state of our newest app looks like this: Screenshot 2023-02-13 at 15 46 59

I took a good amount of time last week to optimize the app to ensure that there are no memory leaks and friends.

None of the crashes are reproducable to us, I can only see weird random signal 11 crashes on emulator startup here and there, which all seem to be memory access/pointer issues. We are preloading a good amount of data from a sqlite db on app start (around 200k entries across a dozen tables).

If you need a compiled version or access to our app, I'd have no problems with that. Would be great to get the crash rate down. I already have a client that switched to Unity for their next project because of these issues.

70% of crashes:

backtrace:
  #00  pc 0x00000000007cf994  /data/app/~~yjQxyUlqBvGbORQs3NF5iA==/appid-YZAiwBTKwPVhsOstZ9QAag==/split_config.arm64_v8a.apk!libCore.so
  #01  pc 0x00000000007fc894  /data/app/~~yjQxyUlqBvGbORQs3NF5iA==/appid-YZAiwBTKwPVhsOstZ9QAag==/split_config.arm64_v8a.apk!libCore.so
  #02  pc 0x00000000007dc9ec  /data/app/~~yjQxyUlqBvGbORQs3NF5iA==/appid-YZAiwBTKwPVhsOstZ9QAag==/split_config.arm64_v8a.apk!libCore.so
  #03  pc 0x000000000000ed5c 

backtrace:
  #00  pc 0x00000000007cf994  /data/app/~~j15nSJG4YHTv7u5hHENLGA==/appid-ASY6bTyzYZmufQKZUToYbA==/split_config.arm64_v8a.apk!libCore.so
  #01  pc 0x00000000007fc894  /data/app/~~j15nSJG4YHTv7u5hHENLGA==/appid-ASY6bTyzYZmufQKZUToYbA==/split_config.arm64_v8a.apk!libCore.so
  #02  pc 0x00000000007dc9ec  /data/app/~~j15nSJG4YHTv7u5hHENLGA==/appid-ASY6bTyzYZmufQKZUToYbA==/split_config.arm64_v8a.apk!libCore.so
  #03  pc 0x000000000000e1e8 

backtrace:
  #00  pc 0x00000000007cf994  /data/app/~~NtP-eY_SFyfFGNS35h_pcg==/appid-pj9VkGXWoLo1Cq0eTSJwQQ==/split_config.arm64_v8a.apk!libCore.so
  #01  pc 0x00000000007fc894  /data/app/~~NtP-eY_SFyfFGNS35h_pcg==/appid-pj9VkGXWoLo1Cq0eTSJwQQ==/split_config.arm64_v8a.apk!libCore.so
  #02  pc 0x00000000007dc9ec  /data/app/~~NtP-eY_SFyfFGNS35h_pcg==/appid-pj9VkGXWoLo1Cq0eTSJwQQ==/split_config.arm64_v8a.apk!libCore.so
  #03  pc 0x000000000000ed20 

backtrace:
  #00  pc 0x00000000007cf994  /data/app/~~hCltrjfUidA568ZWkOJ_tA==/appid-IRWn2VwBQ79ESdoSOa4_1g==/split_config.arm64_v8a.apk!libCore.so
  #01  pc 0x00000000007fc894  /data/app/~~hCltrjfUidA568ZWkOJ_tA==/appid-IRWn2VwBQ79ESdoSOa4_1g==/split_config.arm64_v8a.apk!libCore.so
  #02  pc 0x00000000007dc9ec  /data/app/~~hCltrjfUidA568ZWkOJ_tA==/appid-IRWn2VwBQ79ESdoSOa4_1g==/split_config.arm64_v8a.apk!libCore.so
  #03  pc 0x000000000000e878 

25% of crashes:

backtrace:
  #00  pc 0x000000000000d238

backtrace:
  #00  pc 0x000000000000d238 

backtrace:
  #00  pc 0x000000000000bad8 

backtrace:
  #00  pc 0x0000000000002578 
ajwfrost commented 1 year ago

Which exact version of the AIR SDK is used in those apps please?

SponsorAds commented 1 year ago

Currently 50.0.1.3

ajwfrost commented 1 year ago

The crashes there are happening in avmplus which is (mostly) public so I'll share the thinking here:

_ZN7avmplus11BaseExecMgr9interpGPREPNS_9MethodEnvEiPj
  7dc9ec:       940067d7        bl      7f6948 <_ZN7avmplus11interpBoxedEPNS_9MethodEnvEiPl>

00000000007f6948 <_ZN7avmplus11interpBoxedEPNS_9MethodEnvEiPl>:
  7fc894:       97ff4bb0        bl      7cf754 <_ZN7avmplus7AvmCore6equalsEll>

00000000007cf754 <_ZN7avmplus7AvmCore6equalsEll>:
  7cf994:       f9401508        ldr     x8, [x8,#40]

First entry point shows an interpreted general method being kicked off, it goes into the interpBoxed function and then calls AvmCore::equals. The crash is happening there.. trying to find out exactly where is going to be a bit tricky, but it's not exactly a common occurrence! The fact that you're getting so many crashes happening within that method makes me think something is fundamentally wrong in the app... normally I would think it takes mis-use of the C++ APIs to start making things go this wrong (e.g. hanging pointers to objects that no longer exist but weren't managed by the avmplus core engine), are you able to provide a list of your ANEs or send us an APK to check things through here?

thanks

SponsorAds commented 1 year ago

I can upload you an apk or you can download the store version (air.de.sponsorads.orphans).

APM config:

    "dependencies": [
        {
            "version": "7.0.1",
            "id": "com.distriqt.Core"
        },
        {
            "version": "14.0.0",
            "id": "com.distriqt.InAppBilling"
        },
        {
            "version": "3.1.16",
            "id": "com.distriqt.Volume"
        },
        {
            "version": "6.11.0",
            "id": "com.distriqt.Application"
        },
        {
            "version": "3.0.32",
            "id": "com.distriqt.ZipUtils"
        },
        {
            "version": "2.71828.0",
            "id": "com.distriqt.square.picasso"
        },
        {
            "version": "10.2.1",
            "id": "com.distriqt.facebook.Core"
        },
        {
            "version": "11.1.1",
            "id": "com.distriqt.PushNotifications"
        },
        {
            "version": "11.4.6",
            "id": "com.distriqt.admob.AppLovin"
        },
        {
            "version": "13.7.5",
            "id": "com.distriqt.Adverts"
        },
        {
            "version": "4.0.12",
            "id": "com.distriqt.NetworkInfo"
        },
        {
            "version": "6.11.3",
            "id": "com.distriqt.admob.FacebookAudience"
        },
        {
            "version": "5.1.0",
            "id": "com.distriqt.IDFA"
        },
        {
            "version": "7.2.5",
            "id": "com.distriqt.admob.IronSource"
        },
        {
            "version": "4.2.4",
            "id": "com.distriqt.admob.UnityAds"
        },
        {
            "version": "7.2.3",
            "id": "com.distriqt.firebase.Performance"
        },
        {
            "version": "6.3.1",
            "id": "com.distriqt.Notifications"
        },
        {
            "version": "4.6.0",
            "id": "com.distriqt.MediaPlayer"
        },
        {
            "version": "10.2.1",
            "id": "com.distriqt.facebook.Share"
        },
        {
            "version": "7.2.3",
            "id": "com.distriqt.Firebase"
        }
    ],
ajwfrost commented 1 year ago

Thanks -> so the ANEs look okay, nothing there that uses C++ code. If we can get the APK it may help please: https://transfer.harman.com/requests/AtzsZP9gkt4FV8DyHxKqmO

thanks

SponsorAds commented 1 year ago

I am currently testing it out a bit and I am able to reproduce signal 11 crashes on startup (by simply looping my startup, sometimes takes 100 "starts" for a crash). So starting the app repeatedly may be enough. Of course I can't be sure if that is the crash that causes the 5%.

Every crash I reproduce happens while loading data from sqlite. Works on emulator and device (Samsung S20+).

I uploaded two versions. HauntedDebug, loops through start (no output in app); the other is normal (debuggable) live version.

02-14 11:56:02.597  1495  1495 F libc    : Fatal signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x50 in tid 1495 (nsorads.orphans), pid 1495 (nsorads.orphans)
02-14 11:56:02.696 26353 26353 E crash_dump32: unknown process state: t
02-14 11:56:02.750 26353 26353 I crash_dump32: obtaining output fd from tombstoned, type: kDebuggerdTombstone
02-14 11:56:02.756   741   741 I /system/bin/tombstoned: received crash request for pid 1495
02-14 11:56:02.758 26353 26353 I crash_dump32: performing dump of process 1495 (target tid = 1495)
02-14 11:56:02.772 26353 26353 F DEBUG   : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
02-14 11:56:02.772 26353 26353 F DEBUG   : Build fingerprint: 'samsung/y2seea/y2s:10/QP1A.190711.020/G985FXXU4BTH5:user/release-keys'
02-14 11:56:02.772 26353 26353 F DEBUG   : Revision: '22'
02-14 11:56:02.772 26353 26353 F DEBUG   : ABI: 'arm'
02-14 11:56:02.777 26353 26353 F DEBUG   : Timestamp: 2023-02-14 11:56:02+0100
02-14 11:56:02.777 26353 26353 F DEBUG   : pid: 1495, tid: 1495, name: nsorads.orphans  >>> air.de.sponsorads.orphans <<<
02-14 11:56:02.777 26353 26353 F DEBUG   : uid: 10404
02-14 11:56:02.777 26353 26353 F DEBUG   : signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x50
02-14 11:56:02.777 26353 26353 F DEBUG   : Cause: null pointer dereference
02-14 11:56:02.777 26353 26353 F DEBUG   :     r0  00000000  r1  aa57f978  r2  ac205670  r3  ac2abcb8
02-14 11:56:02.777 26353 26353 F DEBUG   :     r4  ffee2fa8  r5  00000001  r6  9cbd4ac0  r7  ffee2dd8
02-14 11:56:02.777 26353 26353 F DEBUG   :     r8  9cbd4ac0  r9  ac33b5e1  r10 00000000  r11 ffee23f0
02-14 11:56:02.777 26353 26353 F DEBUG   :     ip  aed8c7cb  sp  ffee2378  lr  9dae8e08  pc  a0f2655c
02-14 11:56:02.779   603   603 E audit   : type=1400 audit(1676372162.773:17034): avc:  denied  { getattr } for  pid=727 comm="epic" name="control_profile" dev="sysfs" ino=65048 scontext=u:r:epicd:s0 tcontext=u:object_r:migov_file:s0 tclass=file permissive=0 SEPF_SM-G985F_10_0019 audit_filtered
02-14 11:56:02.779   603   603 E audit   : type=1300 audit(1676372162.773:17034): arch=c00000b7 syscall=56 success=yes exit=7 a0=ffffff9c a1=7b154af81c a2=2 a3=0 items=0 ppid=1 pid=727 auid=4294967295 uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=1000 sgid=1000 fsgid=1000 tty=(none) ses=4294967295 comm="epic" exe="/vendor/bin/epic" subj=u:r:epicd:s0 key=(null)
02-14 11:56:02.779   603   603 E audit   : type=1327 audit(1676372162.773:17034): proctitle="/vendor/bin/epic"
02-14 11:56:02.782   603   603 E audit   : type=1400 audit(1676372162.777:17035): avc:  denied  { getattr } for  pid=727 comm="epic" name="set_margin" dev="sysfs" ino=65047 scontext=u:r:epicd:s0 tcontext=u:object_r:migov_file:s0 tclass=file permissive=0 SEPF_SM-G985F_10_0019 audit_filtered
02-14 11:56:02.895 26353 26353 F DEBUG   : 
02-14 11:56:02.895 26353 26353 F DEBUG   : backtrace:
02-14 11:56:02.895 26353 26353 F DEBUG   :       #00 pc 0000155c  <anonymous:a0f25000>
SponsorAds commented 1 year ago

Let me share a small function in which my logs repeatedly stop after crash.

public function query(query:String, cb:Function = null, log:Boolean = true):Boolean {
  _app.log("DBStatement. query #1");
  this.isReady = false;

  _callback = cb;
  _query = query;
  _log = log;

  _app.log("DBStatement. query #2");

  if(_stmt == null) {
  _stmt = new SQLStatement(); 
  }

  _app.log("DBStatement. query #3");
  _app.log("DBStatement. query #3.1");

  _stmt.addEventListener(SQLEvent.RESULT, queryResultHandler);
  _app.log("DBStatement. query #3.2");
  _stmt.addEventListener(SQLErrorEvent.ERROR, queryErrorHandler);

  _app.log("DBStatement. query #3.3");

  _stmt.sqlConnection = _app.db.con;
  _app.log("DBStatement. query #3.4");
  _stmt.text = query;

  _app.log("DBStatement. query #4");
  _stmt.execute();
  _app.log("DBStatement. query #5");

  return true;
}

Log always stops at _app.log("DBStatement. query #3.1");

The app is reusing/pooling SQLStatement() as much as possible, event handlers are being removed in result handlers and re added on new statements.

SponsorAds commented 1 year ago

30 minutes after removing the reusage and using a fresh SqlStatement() every time, my reproducable crashes are completely gone. I'll keep testing.

Can AIRs sqlite implementation have anything to do with crashes above?

ajwfrost commented 1 year ago

Interesting! Thanks for this -> looks like you've found the culprit (and yes I suspect the AIR build of sqlite has got a problem, maybe an unexpected restriction/bug around the re-use of the object). With that, and the debug build, we'll see if we can reproduce and add extra information in to find out where it's going wrong...

thanks!

ajwfrost commented 1 year ago

@spAds2 just to check here, with the debug version of the app, all I get after a brief splash logo is a set of starling stats on the right hand side of an otherwise black screen. It just sits there for a very long time.. is this the right result, but maybe for me it's just not crashing? This is on a Pixel 6, can you suggest some devices where you've seen is going wrong and we can try others..?

thanks

SponsorAds commented 1 year ago

Yes, it is internally just repeating the start (loading all data from sqlite). You can see it working with the fluctuating frame rate/RAM usage. I tested on AIR emulator and Samsung S20+.

I already uploaded a new version with the change, so I'll see in 2-3 days how crashes go for the users. Does look promising for now with 0 crashes on 5k installs.

ajwfrost commented 1 year ago

Thanks - interesting, I put it onto an S20 and it failed very quickly. Will check that we can replace the runtime library with a debug version to investigate further..

SponsorAds commented 1 year ago

A little update: Screenshot 2023-02-18 at 11 19 47

Would love to get the ANR at least halved too, but I guess that will be another topic.

ajwfrost commented 1 year ago

ANRs will hopefully be cut with our next release that allows the runtime to be put into a background thread, so from Android's perspective this should mean it's not waiting to process user input...

In terms of the crashing in sqlite .. I don't suppose you're able to condense that into a swf file, or provide a way for us to package it up with our own version of the runtime? The newer Android OS versions prevent us from loading our runtime file automatically, and we didn't have any luck in hacking/repacking the APK file...

thanks