airsdk / Adobe-Runtime-Support

Report, track and discuss issues in Adobe AIR. Monitored by Adobe - and HARMAN - and maintained by the AIR community.
200 stars 11 forks source link

[BUG] signal 11 (SIGSEGV), code 1 (SEGV_MAPERR) #167

Open neuronix opened 4 years ago

neuronix commented 4 years ago

Ever since migrating to targetSDK="28" as required by Google, our apps are plagued with these errors, causing up to 9% of sessions with crashes on Android 9. The bug does sometimes occur on other versions but 95% of times, it's Android 9.

With exactly the same ANEs, setup & AIR SDK, we had just about none of these errors using targetSDK=19.

This is a bug that has been around in AIR for years now (not joking) but it has never been so bad. AIR 33 does not fix the issue either.

Some topics where devs have discussed this issue but failed to find a solution:

Unfortunately the stack traces are pretty poor:

signal 11 (SIGSEGV), code 1 (SEGV_MAPERR) strcmp

` pid: 0, tid: 0 >>> air.com.company.application <<<

backtrace:

00 pc 000000000001cb70 /system/lib64/libc.so (strcmp+24)

01 pc 00000000003b8ddc /data/app/air.com.company.application-1/lib/arm64/libCore.so

02 pc 00000000003bc750 /data/app/air.com.company.application-1/lib/arm64/libCore.so

03 pc 00000000002d2a6c /data/app/air.com.company.application-1/lib/arm64/libCore.so

04 pc 00000000002d3b10 /data/app/air.com.company.application-1/lib/arm64/libCore.so

05 pc 00000000002d64e8 /data/app/air.com.company.application-1/lib/arm64/libCore.so

06 pc 00000000002d9250 /data/app/air.com.company.application-1/lib/arm64/libCore.so

07 pc 00000000006f0308 /data/app/air.com.company.application-1/lib/arm64/libCore.so

08 pc 00000000006ef5a8 /data/app/air.com.company.application-1/lib/arm64/libCore.so

09 pc 0000000000271c38 /data/app/air.com.company.application-1/lib/arm64/libCore.so

10 pc 0000000000271ca8 /data/app/air.com.company.application-1/lib/arm64/libCore.so

11 pc 00000000002719dc /data/app/air.com.company.application-1/lib/arm64/libCore.so

12 pc 00000000000677d4 /system/lib64/libc.so (_ZL15__pthread_startPv+52)

13 pc 000000000001edf0 /system/lib64/libc.so (__start_thread+16)`

I'd like to underline the urgence of this matter. 9% crash rate is really far far over the 1% Playstore bad behavior threshold, so this is hurty our apps very badly because Google reduces exposure of apps with poor crash / anr rates. We also have no workaround since we can no longer revert to a lower targetSDK.

Thanks for your support

PippoApps commented 4 years ago

What is going on here is heartwarming. this is what AIR developers have been dreaming for during the last decade. Thank you Andrew, thank you everybody <3

raresn commented 4 years ago

I couldn't agree more with @PippoApps ! Thank you @ajwfrost. Not sure you are aware of the positive impact your work has on our lives.

ajwfrost commented 4 years ago

Thank you @PippoApps, @raresn - not exactly the response I expected after announcing a delay to an update, but very nice for us to hear :-)

hardcoremore commented 4 years ago

@ajwfrost, I really do love using AIR. I like the AS3 Syntax. There are a lot of years of experience of many devs behind AS3 and Adobe never cared that much. Many people here had a lot of issues in communication with Adobe. There were issues that we waited for years to be fixed and many of them didn't get fixed at all.

That's why people are reacting positively for your prompt communication about the issues. Please keep doing that :)

We love AIR and we hope it will see a lot more improvements :)

Michoko92 commented 4 years ago

I've got some good news. I changed all calls in my app from HTTPS to simple HTTP, and published an update to the Play Store on the 29th of June. Since then, I don't see any crash on the Play Store console! So the SSL code definitely seems to be the issue.

Also I wanted to add my voice to the previous comments, and thank you for all the amazing technical support you provide. You're doing a great job, and it is appreciated!

ghost commented 4 years ago

Hi! We migrated from AIR SDK 32 to 33.1 and got a lot of this type issues on Android 9 and 10:

00 pc 0000000000887f4c /data/app/com.belkatechnologies.fe-UXmBNqyaXrSkhf7XSgw9FQ==/lib/arm/libCore.so

What can it be? Thank you!

ventr1x commented 3 years ago

So.. any update on this? We're currently at 4% crash rate with "signal 11" errors. These crashes cause horrible problems like users losing or corrupting save games if it happens in a bad timing. @ajwfrost

Gokulv617 commented 3 years ago

@ajwfrost

signal 11 (SIGSEGV), code 1 (SEGV_MAPERR)

AIR 33.1.1.259 ANDROID

Crash-501-2

Stack trace: Crash - LeTV Le 2 - Android 6.0 (SDK 23) - all 22 crashes were reported by this same device


pid: 0, tid: 0 >>> air.com.company<<<

backtrace:

00 pc 0000000000cf1754 /system/vendor/lib64/libllvm-glnext.so (ShaderObjects::adjustSymbolPointers(char, GLSL_SYMBOL)+80)

00 pc 0000000000cf1dfc /system/vendor/lib64/libllvm-glnext.so (ShaderObjects::loadProgramBinary(CompilerContext, void, unsigned long, QGLC_LINKPROGRAM_RESULT*)+812)

00 pc 0000000000c73e5c /system/vendor/lib64/libllvm-glnext.so (CompilerContext::loadProgramBinary(void, unsigned long, QGLC_LINKPROGRAM_RESULT)+176)

00 pc 0000000000d07888 /system/vendor/lib64/libllvm-glnext.so (QGLCLoadProgramBinary(void, void, unsigned long, QGLC_LINKPROGRAM_RESULT*)+68)

00 pc 00000000001a1790 /system/vendor/lib64/egl/libGLESv2_adreno.so (EsxShaderCompiler::LoadProgramBinaryBlob(EsxContext, EsxProgram, void const, unsigned long, EsxInfoLog)+144)

00 pc 0000000000189da4 /system/vendor/lib64/egl/libGLESv2_adreno.so (EsxProgram::LoadProgramBinary(EsxContext, unsigned int, void const, int)+164)

00 pc 000000000012ac88 /system/vendor/lib64/egl/libGLESv2_adreno.so (EsxContext::GlProgramBinary(unsigned int, unsigned int, void const*, int)+136)

00 pc 0000000000110bc4 /system/vendor/lib64/egl/libGLESv2_adreno.so (glProgramBinary+68)

00 pc 0000000000899774 /system/app/webview/webview.apk (offset 0x8b2000)

00 pc 00000000008455a0 /system/app/webview/webview.apk (offset 0x8b2000)

00 pc 0000000000852698 /system/app/webview/webview.apk (offset 0x8b2000)

00 pc 0000000000826634 /system/app/webview/webview.apk (offset 0x8b2000)

00 pc 0000000000826828 /system/app/webview/webview.apk (offset 0x8b2000)

00 pc 000000000082b664 /system/app/webview/webview.apk (offset 0x8b2000)

00 pc 00000000007e21e0 /system/app/webview/webview.apk (offset 0x8b2000)

00 pc 00000000008333b8 /system/app/webview/webview.apk (offset 0x8b2000)

00 pc 000000000146d650 /system/app/webview/webview.apk (offset 0x8b2000)

00 pc 000000000146d6b8 /system/app/webview/webview.apk (offset 0x8b2000)

00 pc 00000000007e232c /system/app/webview/webview.apk (offset 0x8b2000)

00 pc 0000000001468be4 /system/app/webview/webview.apk (offset 0x8b2000)

00 pc 000000000146ba44 /system/app/webview/webview.apk (offset 0x8b2000)

00 pc 000000000147d8f8 /system/app/webview/webview.apk (offset 0x8b2000)

00 pc 0000000001466080 /system/app/webview/webview.apk (offset 0x8b2000)

00 pc 0000000001465534 /system/app/webview/webview.apk (offset 0x8b2000)

00 pc 0000000000764160 /system/app/webview/webview.apk (offset 0x8b2000)

00 pc 000000000077f080 /system/app/webview/webview.apk (offset 0x8b2000)

00 pc 000000000077f988 /system/app/webview/webview.apk (offset 0x8b2000)

00 pc 000000000077ff24 /system/app/webview/webview.apk (offset 0x8b2000)

00 pc 0000000000780f94 /system/app/webview/webview.apk (offset 0x8b2000)

00 pc 000000000077f594 /system/app/webview/webview.apk (offset 0x8b2000)

00 pc 00000000007909f8 /system/app/webview/webview.apk (offset 0x8b2000)

00 pc 000000000077e3d4 /system/app/webview/webview.apk (offset 0x8b2000)

00 pc 00000000007ad570 /system/app/webview/webview.apk (offset 0x8b2000)

00 pc 00000000007a9220 /system/app/webview/webview.apk (offset 0x8b2000)

00 pc 0000000000068714 /system/lib64/libc.so (__pthread_start(void*)+52)

00 pc 000000000001c604 /system/lib64/libc.so (__start_thread+16)

rdefalco commented 3 years ago

AIR 33.1 build 259 (still haven't tested with build 300)

backtrace:

00 pc 0000000000262318 /data/app/air.it.falcosoft.sultan-M82AmXT4SH-yUyPZgZgemw==/lib/arm64/libCore.so

00 pc 0000000000265c10 /data/app/air.it.falcosoft.sultan-M82AmXT4SH-yUyPZgZgemw==/lib/arm64/libCore.so

00 pc 000000000003fc5c /data/app/air.it.falcosoft.sultan-M82AmXT4SH-yUyPZgZgemw==/oat/arm64/base.odex (art_jni_trampoline+140)

00 pc 00000000020060d4 /memfd:/jit-cache (com.adobe.air.customHandler.handleMessage+68)

00 pc 0000000000755fa4 /system/framework/arm64/boot-framework.oat (android.os.Handler.dispatchMessage+180)

00 pc 0000000000759548 /system/framework/arm64/boot-framework.oat (android.os.Looper.loop+1448)

00 pc 00000000004d6fc4 /system/framework/arm64/boot-framework.oat (android.app.ActivityThread.main+788)

00 pc 00000000001375b8 /apex/com.android.runtime/lib64/libart.so (art_quick_invoke_static_stub+568)

00 pc 000000000014608c /apex/com.android.runtime/lib64/libart.so (art::ArtMethod::Invoke(art::Thread, unsigned int, unsigned int, art::JValue, char const)+276)

00 pc 00000000004ab100 /apex/com.android.runtime/lib64/libart.so (art::(anonymous namespace)::InvokeWithArgArray(art::ScopedObjectAccessAlreadyRunnable const&, art::ArtMethod, art::(anonymous namespace)::ArgArray, art::JValue, char const)+104)

00 pc 00000000004acb28 /apex/com.android.runtime/lib64/libart.so (art::InvokeMethod(art::ScopedObjectAccessAlreadyRunnable const&, _jobject, _jobject, _jobject*, unsigned long)+1476)

00 pc 0000000000438f80 /apex/com.android.runtime/lib64/libart.so (art::Method_invoke(_JNIEnv, _jobject, _jobject, _jobjectArray)+52)

00 pc 00000000000c2d34 /system/framework/arm64/boot.oat (art_jni_trampoline+180)

00 pc 00000000009cf8b8 /system/framework/arm64/boot-framework.oat (com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run+136)

00 pc 00000000009d7e64 /system/framework/arm64/boot-framework.oat (com.android.internal.os.ZygoteInit.main+3124)

00 pc 00000000001375b8 /apex/com.android.runtime/lib64/libart.so (art_quick_invoke_static_stub+568)

00 pc 000000000014608c /apex/com.android.runtime/lib64/libart.so (art::ArtMethod::Invoke(art::Thread, unsigned int, unsigned int, art::JValue, char const)+276)

00 pc 00000000004ab100 /apex/com.android.runtime/lib64/libart.so (art::(anonymous namespace)::InvokeWithArgArray(art::ScopedObjectAccessAlreadyRunnable const&, art::ArtMethod, art::(anonymous namespace)::ArgArray, art::JValue, char const)+104)

00 pc 00000000004aad6c /apex/com.android.runtime/lib64/libart.so (art::InvokeWithVarArgs(art::ScopedObjectAccessAlreadyRunnable const&, _jobject, _jmethodID, std::__va_list)+408)

00 pc 00000000003b77a8 /apex/com.android.runtime/lib64/libart.so (art::JNI::CallStaticVoidMethodV(_JNIEnv, _jclass, _jmethodID*, std::__va_list)+628)

00 pc 00000000000ec560 /system/lib64/libandroid_runtime.so (_JNIEnv::CallStaticVoidMethod(_jclass, _jmethodID, ...)+116)

00 pc 00000000000ef5a0 /system/lib64/libandroid_runtime.so (android::AndroidRuntime::start(char const*, android::Vector const&, bool)+796)

00 pc 00000000000034e0 /system/bin/app_process64 (main+1168)

00 pc 000000000007e798 /apex/com.android.runtime/lib64/bionic/libc.so (__libc_init+108)

cgascons commented 3 years ago

Hi, This is also an issue for us now. In our particular case the stacktrace we have is very short:

signal 11 (SIGSEGV), code 2 (SEGV_ACCERR)
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> air.com.xxx <<<

backtrace:
  #00  pc 000000000007e038  <anonymous>

We released our last update on Dec 7 and had to change the targetSDK to 29. We never had so many crashes like this.

image

I've got some good news. I changed all calls in my app from HTTPS to simple HTTP, and published an update to the Play Store on the 29th of June. Since then, I don't see any crash on the Play Store console! So the SSL code definitely seems to be the issue.

I'm not sure if your comment has gone unnoticed but if this is true it's definitely something I will try to do right away. The bad news for us is, since we are using Gamesparks as our backend service I'm afraid there's no way we will be able to change the URLS that connect to their servers to try to avoid this issue. Any suggestions on how to proceed/what to try? @ajwfrost Any chance this might be the reason for this crashes to happen?

berkayk commented 3 years ago

@cgascons our crash logs are only one line like yours.

I am using https for loading facebook profile pictures. iOS doesn't allow HTTP URLs, but I'll try HTTP URLs for the android version.

Wish we had a proper response from the team. These crashes are really annoying our users.

cgascons commented 3 years ago

Hi @berkayk we managed to lower the crashes by reverting back to AIR 33.1.259

berkayk commented 3 years ago

Thanks, may I ask your ANR rate? Our ANR rate is 1.7% after AIR 33. It was 0.3 at AIR 32.

cgascons commented 3 years ago

Sure, this is how our current chart looks now:

image

According to Android Vitals Overview we are now on 1.17%, we used to be prior the update in ~0.4%, I'm assuming we still need to wait a few more days to flatten the curve

berkayk commented 3 years ago

So your ANR rate is 1.17%, right? The graph shows both the ANRs and crashes' counts. Also, what do you mean by prior the update? :) I'm a little confused. Was your previous update SDK 33?

cgascons commented 3 years ago

So your ANR rate is 1.17%, right?

Yes

Also, what do you mean by prior the update? :)

We updated the game with all our ANEs updated and a new AIR sdk (build 300) and started to increase the ANR & Crash curve, after decreasing it to build 259 it started to look the same as before.

Was your previous update SDK 33?

33.1.259

berkayk commented 3 years ago

Thanks a lot for the clarification. Probably we have something bad that is causing ANRs after SDK 33.

Input dispatching timed out (Waiting because the touched window's input channel is full. Outbound queue length: 1. Wait queue length: 52.

Hoping to find a solution for this.

ventr1x commented 3 years ago

As you can see there is no reaction at all about the issue. Even direct support requests simply stay unanswered and ignored. We are currently considering legal action as support for a SDK that wants >1000€/year from us has to provide support for their product or are liable for damages they cause.. at least in the EU.

The metrics not only cause users to give bad ratings and hurt user activity and sales, they also hinder us from selling two of our games (with a 4m user base). That sale is worth 7 figures for us. Of course users also flood us with support requests, causing us increased support manpower and therefore near 5 figure additional fix costs per month. The entire lack of support and transparency about any issue or roadmap is unacceptable.

PippoApps commented 3 years ago

This is strange. Usually @ajwfrost monitors these threads and provides punctual support.

ventr1x commented 3 years ago

@PippoApps My last request about this issue was in September. Since then no one from Harman wrote anything about this issue. They seem to think it's a ssl issue which I can't confirm. I removed https from one of our apps with no affect at all.

ajwfrost commented 3 years ago

@ventr1x @PippoApps well I get notification emails when an issue is raised or commented on, but I'm currently getting many hundred emails a day as we're also supporting all the companies who decided to ignore Adobe's "Flash Player End of Life" notice from 2017 and who are now scrambling to get support from us to continue using Flash Player beyond 2020...

One problem with these sorts of threads are that there are several different root causes being discussed (sadly). The build 300 had an ARM64 update in the JIT to make shared bytearrays work properly in Workers, but that had a condition that could cause access errors. There are also sometimes issues in both AIR and in Webkit and other apps caused by 'null' activity variables. There was an audio bug that could be a problem if you're using RTPM. And there's the potential for HTTPS to go wrong due to an issue in openssl.

Our next release should be up very shortly to resolve a couple of those major ones, but there will be some things that we can necessarily solve (particularly the webkit things! and I suspect the root cause of those is more in Android than in Webkit..).

ventr1x commented 3 years ago

We're currently migrating browsergames from flash to javascript after the issue being ignored for years. I know that pain, but it also gives me quite the big bonus for my efforts :)

The annoying thing with the issue is the missing stack trace. The one line libCore doesn't help anyone, so people discussing multiple issues at once is not really a surprise.

Example: 2k reports this month alone:

backtrace:
  #00  pc 00000000004acbd0  /data/app/air.*-MOd_ItwjN3wlRV_-OdZIGg==/lib/arm/libCore.so
  #00  pc 00000000004ce271  /data/app/air.*-MOd_ItwjN3wlRV_-OdZIGg==/lib/arm/libCore.so
  #00  pc 00000000004b6451  /data/app/air.*-MOd_ItwjN3wlRV_-OdZIGg==/lib/arm/libCore.so
  #00  pc 0000000000000a17  <anonymous>
berkayk commented 3 years ago

@ventr1x we have the same missing stack trace reports. @ajwfrost Maybe that could also help you discover what could be removing those stack traces.

ajwfrost commented 3 years ago

Missing/corrupted stack traces is a clue to us that the issue is in the JIT-compiled code.. if these go wrong then the system can't find the function return address or the link register gets overwritten perhaps. But the same sort of thing could happen with a buffer overflow error etc. Most crashes within our C++ code would have a normal stack trace; the Java ones are even better as we get filename/line numbers :-)