Segfault in production Android app - facebook::hermes::debugger::EventObserver::breakpointResolved

MFazio23 commented 3 weeks ago

Bug Description

We have a massive (15K) amount of crashes in our production React Native Android app which are shown to end in the breakpointResolved function inside Hermes.

This crash occurs ~90% of the time on Motorola devices, though we've seen it a slight bit on other devices like Samsungs and Pixels.

It is also happening all over our app without a clear pattern as to something in particular causing it.

[X] I have run gradle clean and confirmed this bug does not occur with JSC
[X] The issue is reproducible with the latest version of React Native.

Hermes git revision (if applicable): N/A React Native version: 0.72.1 OS: Android Platform (most likely one of arm64-v8a, armeabi-v7a, x86, x86_64): arm64_v8a

Stack trace

OS Version: Android 12 (S3RDES32.123-37-5-11)
Report Version: 104

Exception Type: Unknown (SIGSEGV)

Application Specific Information:
Segfault

Thread 0 Crashed:
0   split_config.arm64_v8a.apk      0x73f0d32c70        facebook::hermes::debugger::EventObserver::breakpointResolved
1   split_config.arm64_v8a.apk      0x73f0ceebb4        facebook::hermes::debugger::EventObserver::breakpointResolved
2   split_config.arm64_v8a.apk      0x73f0da8a58        facebook::hermes::debugger::EventObserver::breakpointResolved
3   split_config.arm64_v8a.apk      0x73f0da80c8        facebook::hermes::debugger::EventObserver::breakpointResolved
4   split_config.arm64_v8a.apk      0x73f0da720c        facebook::hermes::debugger::EventObserver::breakpointResolved
5   split_config.arm64_v8a.apk      0x73f0daa250        facebook::hermes::debugger::EventObserver::breakpointResolved
6   split_config.arm64_v8a.apk      0x73f0da6fb0        facebook::hermes::debugger::EventObserver::breakpointResolved
7   split_config.arm64_v8a.apk      0x73f0da9804        facebook::hermes::debugger::EventObserver::breakpointResolved
8   split_config.arm64_v8a.apk      0x73f0d15574        facebook::hermes::debugger::EventObserver::breakpointResolved
9   split_config.arm64_v8a.apk      0x73f0d14650        facebook::hermes::debugger::EventObserver::breakpointResolved
10  split_config.arm64_v8a.apk      0x73f0d081c8        facebook::hermes::debugger::EventObserver::breakpointResolved
11  split_config.arm64_v8a.apk      0x73f0d03660        facebook::hermes::debugger::EventObserver::breakpointResolved
12  split_config.arm64_v8a.apk      0x73f0ce7db8        facebook::hermes::debugger::EventObserver::breakpointResolved
13  split_config.arm64_v8a.apk      0x73f0ce6cc0        facebook::hermes::debugger::EventObserver::breakpointResolved
14  split_config.arm64_v8a.apk      0x73f0d92fdc        facebook::hermes::debugger::EventObserver::breakpointResolved
15  split_config.arm64_v8a.apk      0x73f0ce7b54        facebook::hermes::debugger::EventObserver::breakpointResolved
16  split_config.arm64_v8a.apk      0x73f0d02664        facebook::hermes::debugger::EventObserver::breakpointResolved
17  split_config.arm64_v8a.apk      0x73f0d03ec4        facebook::hermes::debugger::EventObserver::breakpointResolved
18  split_config.arm64_v8a.apk      0x73f0d03660        facebook::hermes::debugger::EventObserver::breakpointResolved
19  split_config.arm64_v8a.apk      0x73f0ce7db8        facebook::hermes::debugger::EventObserver::breakpointResolved
20  split_config.arm64_v8a.apk      0x73f0ce6cc0        facebook::hermes::debugger::EventObserver::breakpointResolved
21  split_config.arm64_v8a.apk      0x73f0d92fdc        facebook::hermes::debugger::EventObserver::breakpointResolved
22  split_config.arm64_v8a.apk      0x73f0ce7b54        facebook::hermes::debugger::EventObserver::breakpointResolved
23  split_config.arm64_v8a.apk      0x73f0d02664        facebook::hermes::debugger::EventObserver::breakpointResolved
24  split_config.arm64_v8a.apk      0x73f0d03ec4        facebook::hermes::debugger::EventObserver::breakpointResolved
25  split_config.arm64_v8a.apk      0x73f0d03660        facebook::hermes::debugger::EventObserver::breakpointResolved
26  split_config.arm64_v8a.apk      0x73f0ce7db8        facebook::hermes::debugger::EventObserver::breakpointResolved
27  split_config.arm64_v8a.apk      0x73f0ce776c        facebook::hermes::debugger::EventObserver::breakpointResolved
28  split_config.arm64_v8a.apk      0x73f0cd86cc        facebook::jsi::Array::setValueAtIndex<T>
29  split_config.arm64_v8a.apk      0x73ed783aa8        facebook::jsi::Function::call<T>
30  split_config.arm64_v8a.apk      0x73ed783914        folly::to_ascii_with<T>
31  split_config.arm64_v8a.apk      0x73ed77ab44        std::__ndk1::__invoke_void_return_wrapper<T>::__call<T>
32  split_config.arm64_v8a.apk      0x73ed78043c        facebook::react::JSIExecutor::callFunction
33  split_config.arm64_v8a.apk      0x73f0aaaca0        facebook::react::JsToNativeBridge::callSerializableNativeHook
34  split_config.arm64_v8a.apk      0x73f0aabcb8        facebook::react::JsToNativeBridge::callSerializableNativeHook
35  split_config.arm64_v8a.apk      0x73f0a81144        facebook::react::JMessageQueueThread::~JMessageQueueThread
36  split_config.arm64_v8a.apk      0x7409296e84        facebook::jni::detail::MethodWrapper<T>::dispatch
37  split_config.arm64_v8a.apk      0x7409296dfc        facebook::jni::detail::FunctionWrapper<T>::call
38  base.odex                       0x741e0e0004        <unknown> + 498720440324

EOF

Steps To Reproduce

We have yet to reproduce this on our side and are going off of multiple crash/app health apps (Sentry/Crashlytics) to let us know this is happening. We've tested dev and production builds on lots of devices (including the most common Motorola device in the list) without seeing anything.

The Expected Behavior

The app does not crash.

neildhar commented 3 weeks ago

Hey @MFazio23, looking at the stack, it looks like it hasn't been symbolicated properly, which is why the stack trace doesn't make sense, and so many frames are annotated with that one function.

Take a look at the steps here to get a proper stack trace: https://github.com/facebook/hermes/blob/main/doc/ReactNativeIntegration.md#react-native--071

MFazio23 commented 3 weeks ago

Thanks @neildhar - I did this before but I'll give it another shot because it looks like I got something wrong.

MFazio23 commented 2 weeks ago

We updated our app to use 0.73.6 (with plans for 0.74.x coming up), and the stack traces have changed (see below for an example), but the errors seem to be coming up just as frequently.

The stack trace was symbolicated by Sentry for us automatically, but when I uploaded the debug symbols to Crashlytics before, they were showing the same stack trace as Sentry.

With the doc linked above, I'm still confused by "you can just invoke ndk-stack and [it] will just work" - the ndk-stack command expects a symbol directory, so where does that come from?

OS Version: Android 12 (S3RDES32.123-37-5-11)
Report Version: 104

Exception Type: Unknown (SIGSEGV)

Application Specific Information:
Segfault

Thread 0 Crashed:
0   split_config.arm64_v8a.apk      0x6f8acc8bcc        facebook::hermes::HermesRuntime::~HermesRuntime
1   split_config.arm64_v8a.apk      0x6f8ac953e8        facebook::hermes::HermesRuntime::~HermesRuntime
2   split_config.arm64_v8a.apk      0x6f8ad3c430        facebook::hermes::HermesRuntime::~HermesRuntime
3   split_config.arm64_v8a.apk      0x6f8ad3ba8c        facebook::hermes::HermesRuntime::~HermesRuntime
4   split_config.arm64_v8a.apk      0x6f8ad3ae20        facebook::hermes::HermesRuntime::~HermesRuntime
5   split_config.arm64_v8a.apk      0x6f8ad3dcd0        facebook::hermes::HermesRuntime::~HermesRuntime
6   split_config.arm64_v8a.apk      0x6f8ad3abc8        facebook::hermes::HermesRuntime::~HermesRuntime
7   split_config.arm64_v8a.apk      0x6f8ad3d290        facebook::hermes::HermesRuntime::~HermesRuntime
8   split_config.arm64_v8a.apk      0x6f8ace8294        facebook::hermes::HermesRuntime::~HermesRuntime
9   split_config.arm64_v8a.apk      0x6f8ace5890        facebook::hermes::HermesRuntime::~HermesRuntime
10  split_config.arm64_v8a.apk      0x6f8acafcc0        facebook::hermes::HermesRuntime::~HermesRuntime
11  split_config.arm64_v8a.apk      0x6f8acece40        facebook::hermes::HermesRuntime::~HermesRuntime
12  split_config.arm64_v8a.apk      0x6f8ac8f93c        facebook::hermes::HermesRuntime::~HermesRuntime
13  split_config.arm64_v8a.apk      0x6f8aca0464        facebook::hermes::HermesRuntime::~HermesRuntime
14  split_config.arm64_v8a.apk      0x6f8aca1c9c        facebook::hermes::HermesRuntime::~HermesRuntime
15  split_config.arm64_v8a.apk      0x6f8aca14ac        facebook::hermes::HermesRuntime::~HermesRuntime
16  split_config.arm64_v8a.apk      0x6f8ac8fa4c        facebook::hermes::HermesRuntime::~HermesRuntime
17  split_config.arm64_v8a.apk      0x6f8ad0a854        facebook::hermes::HermesRuntime::~HermesRuntime
18  split_config.arm64_v8a.apk      0x6f8ac8f93c        facebook::hermes::HermesRuntime::~HermesRuntime
19  split_config.arm64_v8a.apk      0x6f8acaa92c        facebook::hermes::HermesRuntime::~HermesRuntime
20  split_config.arm64_v8a.apk      0x6f8aca1f4c        facebook::hermes::HermesRuntime::~HermesRuntime
21  split_config.arm64_v8a.apk      0x6f8aca14ac        facebook::hermes::HermesRuntime::~HermesRuntime
22  split_config.arm64_v8a.apk      0x6f8ac8fa4c        facebook::hermes::HermesRuntime::~HermesRuntime
23  split_config.arm64_v8a.apk      0x6f8ac8eb30        facebook::hermes::HermesRuntime::~HermesRuntime
24  split_config.arm64_v8a.apk      0x6f8ad26b40        facebook::hermes::HermesRuntime::~HermesRuntime
25  split_config.arm64_v8a.apk      0x6f8ac8f93c        facebook::hermes::HermesRuntime::~HermesRuntime
26  split_config.arm64_v8a.apk      0x6f8aca0464        facebook::hermes::HermesRuntime::~HermesRuntime
27  split_config.arm64_v8a.apk      0x6f8aca1c9c        facebook::hermes::HermesRuntime::~HermesRuntime
28  split_config.arm64_v8a.apk      0x6f8aca14ac        facebook::hermes::HermesRuntime::~HermesRuntime
29  split_config.arm64_v8a.apk      0x6f8ac8fa4c        facebook::hermes::HermesRuntime::~HermesRuntime
30  split_config.arm64_v8a.apk      0x6f8ad26dc0        facebook::hermes::HermesRuntime::~HermesRuntime
31  split_config.arm64_v8a.apk      0x6f8ac8f93c        facebook::hermes::HermesRuntime::~HermesRuntime
32  split_config.arm64_v8a.apk      0x6f8ad0a854        facebook::hermes::HermesRuntime::~HermesRuntime
33  split_config.arm64_v8a.apk      0x6f8ac8f93c        facebook::hermes::HermesRuntime::~HermesRuntime
34  split_config.arm64_v8a.apk      0x6f8acaa92c        facebook::hermes::HermesRuntime::~HermesRuntime
35  split_config.arm64_v8a.apk      0x6f8aca1f4c        facebook::hermes::HermesRuntime::~HermesRuntime
36  split_config.arm64_v8a.apk      0x6f8aca14ac        facebook::hermes::HermesRuntime::~HermesRuntime
37  split_config.arm64_v8a.apk      0x6f8ac8fa4c        facebook::hermes::HermesRuntime::~HermesRuntime
38  split_config.arm64_v8a.apk      0x6f8ac8eb30        facebook::hermes::HermesRuntime::~HermesRuntime
39  split_config.arm64_v8a.apk      0x6f8ad26b40        facebook::hermes::HermesRuntime::~HermesRuntime
40  split_config.arm64_v8a.apk      0x6f8ac8f93c        facebook::hermes::HermesRuntime::~HermesRuntime
41  split_config.arm64_v8a.apk      0x6f8aca0464        facebook::hermes::HermesRuntime::~HermesRuntime
42  split_config.arm64_v8a.apk      0x6f8aca1c9c        facebook::hermes::HermesRuntime::~HermesRuntime
43  split_config.arm64_v8a.apk      0x6f8aca14ac        facebook::hermes::HermesRuntime::~HermesRuntime
44  split_config.arm64_v8a.apk      0x6f8ac8fa4c        facebook::hermes::HermesRuntime::~HermesRuntime
45  split_config.arm64_v8a.apk      0x6f8ac8f5e4        facebook::hermes::HermesRuntime::~HermesRuntime
46  split_config.arm64_v8a.apk      0x6f8ac82630        facebook::jsi::Array::setValueAtIndex<T>
47  split_config.arm64_v8a.apk      0x6f8a86edc0        facebook::jsi::Function::call<T>
48  split_config.arm64_v8a.apk      0x6f8a86ec30        folly::to_ascii_with<T>
49  split_config.arm64_v8a.apk      0x6f8a866380        std::__ndk1::__function::__func<T>::operator()
50  split_config.arm64_v8a.apk      0x6f8a86b96c        facebook::react::JSIExecutor::callFunction
51  split_config.arm64_v8a.apk      0x6f8a966a3c        facebook::react::JsToNativeBridge::callSerializableNativeHook
52  split_config.arm64_v8a.apk      0x6f8a9679f4        facebook::react::JsToNativeBridge::callSerializableNativeHook
53  split_config.arm64_v8a.apk      0x6f8a93cc18        facebook::react::JMessageQueueThread::~JMessageQueueThread
54  split_config.arm64_v8a.apk      0x6f9309deb0        facebook::jni::detail::MethodWrapper<T>::dispatch
55  split_config.arm64_v8a.apk      0x6f9309de00        facebook::jni::detail::FunctionWrapper<T>::call
56  base.odex                       0x6fbc55c6c4        <unknown> + 479901107908

neildhar commented 2 weeks ago

you can just invoke ndk-stack and [it] will just work

cc @cortinico who is the expert on the intended workflow here.

That said, you can also download the symbols directly from maven: https://repo1.maven.org/maven2/com/facebook/react/hermes-android/

cortinico commented 2 weeks ago

With the doc linked above, I'm still confused by "you can just invoke ndk-stack and [it] will just work" - the ndk-stack command expects a symbol directory, so where does that come from?

You'll have to point to your app build folder like:

adb logcat | $ANDROID_HOME/ndk/26.1.10909125/ndk-stack -sym android/app/build/intermediates/cmake/debug/obj/arm64-v8a

Also there was a sporadic bug with ndk-stack in NDK 26 (https://github.com/android/ndk/issues/1993) which was resolved in NDK 27, which we sadly haven't bumped yet.

facebook / hermes