React Native Javascript not symbolicated self-hosted backend error

ayush-leap commented 3 weeks ago

OS:

[ ] Windows
[ ] Linux
[x] MacOS

Platform:

[ ] iOS
[x] Android

SDK:

[x] @sentry/react-native (>= 1.0.0)
[ ] react-native-sentry (<= 0.43.2)

SDK version: 5.22.2

react-native version: 0.71.19

Are you using Expo?

[ ] Yes
[x] No

Are you using sentry.io or on-premise?

[ ] sentry.io (SaaS)
[x] on-premise

If you are using sentry.io, please post a link to your issue so we can take a look:

[Link to issue]

Configuration:

(@sentry/react-native)

Sentry.init({
      dsn: Config.REACT_APP_SENTRY_DSN,
      tracesSampleRate: 1,
      sampleRate: 1,
      environment: "development",
      release: `${DeviceInfo.getBundleId()}@${DeviceInfo.getVersion()}+${DeviceInfo.getBuildNumber()}`,
      dist: DeviceInfo.getBuildNumber(),
      attachStacktrace: true,
      enableAutoPerformanceTracing: true,
      enabled: true,
      debug: false,
      integrations: [
        new Sentry.ReactNativeTracing({
          routingInstrumentation,
          enableStallTracking: true,
          enableAppStartTracking: true,
          enableNativeFramesTracking: true
        })
      ]
});

I have the following issue:

We recently migrated from RN 0.69.9 to 0.71.19 and stack traces started showing incorrectly. Currently i'm reporting it only for Android. As per this comment, I tried using metro-symbolicate and in it's output it did mentioned the exact line it happened but Sentry doesn't show the same.

I also tried adding Sentry AGP which wasn't present earlier but it also didn't solve the issue for me.

I'm triggering error as below.

  const onSendButtonPress = () => {
    captureException(new Error("TEST: v0.33.3 Send Error #2"))
  };

Output from npx metro-symbolicate index.android.bundle.map < stack.txt

Error: TEST: v0.33.3 Send Error #2
  at onSendButtonPress(src/screens/CosmosHome/components/TxButtonGroup/index.tsx:87:onSendButtonPress)
  at handleOnPress(src/screens/CosmosHome/components/TxButton/index.tsx:41:handleOnPress)
  at _performTransitionSideEffects(node_modules/react-native/Libraries/Pressability/Pressability.js:758:_performTransitionSideEffects)
  at _receiveSignal(node_modules/react-native/Libraries/Pressability/Pressability.js:695:_receiveSignal)
  at onResponderRelease(node_modules/react-native/Libraries/Pressability/Pressability.js:525:responderEventHandlers.onResponderRelease)
  at apply(native)
  at invokeGuardedCallbackImpl(node_modules/react-native/Libraries/Renderer/implementations/ReactNativeRenderer-prod.js:22:invokeGuardedCallbackImpl)
  at apply(native)
  at invokeGuardedCallback(node_modules/react-native/Libraries/Renderer/implementations/ReactNativeRenderer-prod.js:40:invokeGuardedCallback)
  at apply(native)
  at invokeGuardedCallbackAndCatchFirstError(node_modules/react-native/Libraries/Renderer/implementations/ReactNativeRenderer-prod.js:53:invokeGuardedCallbackAndCatchFirstError)
  at executeDispatch(node_modules/react-native/Libraries/Renderer/implementations/ReactNativeRenderer-prod.js:73:executeDispatch)
  at executeDispatchesAndReleaseTopLevel(node_modules/react-native/Libraries/Renderer/implementations/ReactNativeRenderer-prod.js:1125:executeDispatchesAndReleaseTopLevel)
  at forEach(native)
  at forEachAccumulated(node_modules/react-native/Libraries/Renderer/implementations/ReactNativeRenderer-prod.js:361:forEachAccumulated)
  at anonymous(node_modules/react-native/Libraries/Renderer/implementations/ReactNativeRenderer-prod.js:1160:batchedUpdates$argument_0)
  at batchedUpdatesImpl(node_modules/react-native/Libraries/Renderer/implementations/ReactNativeRenderer-prod.js:8457:batchedUpdatesImpl)
  at batchedUpdates(node_modules/react-native/Libraries/Renderer/implementations/ReactNativeRenderer-prod.js:1106:batchedUpdates)
  at _receiveRootNodeIDEvent(node_modules/react-native/Libraries/Renderer/implementations/ReactNativeRenderer-prod.js:1137:_receiveRootNodeIDEvent)
  at receiveTouches(node_modules/react-native/Libraries/Renderer/implementations/ReactNativeRenderer-prod.js:1209:ReactNativePrivateInterface.RCTEventEmitter.register$argument_0.receiveTouches)
  at apply(native)
  at __callFunction(node_modules/react-native/Libraries/BatchedBridge/MessageQueue.js:427:__callFunction)
  at anonymous(node_modules/react-native/Libraries/BatchedBridge/MessageQueue.js:113:__guard$argument_0)
  at __guard(node_modules/react-native/Libraries/BatchedBridge/MessageQueue.js:368:__guard)
  at callFunctionReturnFlushedQueue(node_modules/react-native/Libraries/BatchedBridge/MessageQueue.js:112:callFunctionReturnFlushedQueue)%

android/app/build.gradle

```gradle apply plugin: "com.android.application" apply plugin: "com.facebook.react" apply plugin: "io.fabric" apply plugin: "com.google.gms.google-services" apply plugin: "io.sentry.android.gradle" import com.android.build.OutputFile ... /** * This is the configuration block to customize your React Native Android app. * By default you don't need to apply any configuration, just uncomment the lines you need. */ react { /* Folders */ // The root of your project, i.e. where "package.json" lives. Default is '..' // root = file("../") // The folder where the react-native NPM package is. Default is ../node_modules/react-native // reactNativeDir = file("../node_modules/react-native") // The folder where the react-native Codegen package is. Default is ../node_modules/react-native-codegen // codegenDir = file("../node_modules/react-native-codegen") // The cli.js file which is the React Native CLI entrypoint. Default is ../node_modules/react-native/cli.js // cliFile = file("../node_modules/react-native/cli.js") /* Variants */ // The list of variants to that are debuggable. For those we're going to // skip the bundling of the JS bundle and the assets. By default is just 'debug'. // If you add flavors like lite, prod, etc. you'll have to list your debuggableVariants. debuggableVariants = ["leapdebug", "seidebug"] /* Bundling */ // A list containing the node command and its flags. Default is just 'node'. nodeExecutableAndArgs = ["node", "--max-old-space-size=8192"] // // The command to run when bundling. By default is 'bundle' // bundleCommand = "ram-bundle" // // The path to the CLI configuration file. Default is empty. // bundleConfig = file(../rn-cli.config.js) // // The name of the generated asset file containing your JS bundle // bundleAssetName = "MyApplication.android.bundle" // // The entry file for bundle generation. Default is 'index.android.js' or 'index.js' // entryFile = file("../js/MyApplication.android.js") // // A list of extra flags to pass to the 'bundle' commands. // See https://github.com/react-native-community/cli/blob/main/docs/commands.md#bundle // extraPackagerArgs = [] /* Hermes Commands */ // The hermes compiler command to run. By default it is 'hermesc' // hermesCommand = "$rootDir/my-custom-hermesc/bin/hermesc" // // The list of flags to pass to the Hermes compiler. By default is "-O", "-output-source-map" // hermesFlags = ["-O", "-output-source-map"] } // leap configuration for sentry AGP - https://docs.sentry.io/platforms/react-native/manual-setup/manual-setup/#enable-sentry-agp sentry { // fixes: An organization ID or slug is required (provide with --org) org = "leapwallet" // fixes: A project ID or slug is required (provide with --project) projectName = "leap-cosmos-mobile" // List the build flavors that should be ignored (e.g. "production"). // https://docs.sentry.io/platforms/android/configuration/gradle/#variant-filtering ignoredFlavors = ["sei", "dev"] // Enables or disables the automatic configuration of Native Symbols // for Sentry. This executes sentry-cli automatically so // you don't need to do it manually. // Default is disabled. uploadNativeSymbols = true // Enables or disables the automatic upload of the app's native source code to Sentry. // This executes sentry-cli with the --include-sources param automatically so // you don't need to do it manually. // This option has an effect only when [uploadNativeSymbols] is enabled. // Default is disabled. includeNativeSources = true // `@sentry/react-native` ships with compatible `sentry-android` // This option would install the latest version that ships with the SDK or SAGP (Sentry Android Gradle Plugin) // which might be incompatible with the React Native SDK // Enable auto-installation of Sentry components (sentry-android SDK and okhttp, timber and fragment integrations). // Default is enabled. autoInstallation { enabled = false } // Generates a JVM (Java, Kotlin, etc.) source bundle and uploads your source code to Sentry. // This enables source context, allowing you to see your source // code as part of your stack traces in Sentry. includeSourceContext = true } // compass configuration for sentry AGP - https://docs.sentry.io/platforms/react-native/manual-setup/manual-setup/#enable-sentry-agp sentry { // fixes: An organization ID or slug is required (provide with --org) org = "leapwallet" // fixes: A project ID or slug is required (provide with --project) projectName = "compass-mobile" // List the build flavors that should be ignored (e.g. "production"). // https://docs.sentry.io/platforms/android/configuration/gradle/#variant-filtering ignoredFlavors = ["leap", "dev"] // Enables or disables the automatic configuration of Native Symbols // for Sentry. This executes sentry-cli automatically so // you don't need to do it manually. // Default is disabled. uploadNativeSymbols = true // Enables or disables the automatic upload of the app's native source code to Sentry. // This executes sentry-cli with the --include-sources param automatically so // you don't need to do it manually. // This option has an effect only when [uploadNativeSymbols] is enabled. // Default is disabled. includeNativeSources = true // `@sentry/react-native` ships with compatible `sentry-android` // This option would install the latest version that ships with the SDK or SAGP (Sentry Android Gradle Plugin) // which might be incompatible with the React Native SDK // Enable auto-installation of Sentry components (sentry-android SDK and okhttp, timber and fragment integrations). // Default is enabled. autoInstallation { enabled = false } // Generates a JVM (Java, Kotlin, etc.) source bundle and uploads your source code to Sentry. // This enables source context, allowing you to see your source // code as part of your stack traces in Sentry. includeSourceContext = true } project.ext.sentryCli = [ logLevel: "debug", flavorAware: true ] apply from: new File(["node", "--print", "require.resolve('@sentry/react-native/package.json')"].execute().text.trim(), "../sentry.gradle") ... /** * Set this to true to Run Proguard on Release builds to minify the Java bytecode. */ def enableProguardInReleaseBuilds = false android { ... flavorDimensions 'default' productFlavors { leap { applicationId "io.leapwallet.cosmos" minSdkVersion rootProject.ext.minSdkVersion targetSdkVersion rootProject.ext.targetSdkVersion resValue "string", "build_config_package", "io.leapwallet.cosmos" versionCode 359 versionName "0.0.3" } dev { applicationId "io.leapwallet.cosmos.dev" minSdkVersion rootProject.ext.minSdkVersion targetSdkVersion rootProject.ext.targetSdkVersion resValue "string", "build_config_package", "io.leapwallet.cosmos" versionCode 359 versionName "0.0.3" } sei { applicationId "io.leapwallet.compass" minSdkVersion rootProject.ext.minSdkVersion targetSdkVersion rootProject.ext.targetSdkVersion resValue "string", "build_config_package", "io.leapwallet.cosmos" versionCode 359 versionName "0.0.3" } } } ```

`build.gradle`

```gradle dependencies { // RN Upgrade helper mentions 7.3.1 but expo modules docs says to update it to 7.4.1. // refer: https://docs.expo.dev/bare/upgrade/?fromSdk=46&toSdk=48#androidbuildgradle classpath("com.android.tools.build:gradle:7.4.1") classpath("com.facebook.react:react-native-gradle-plugin") // NOTE: Do not place your application dependencies here; they belong // in the individual module build.gradle files classpath("com.google.gms:google-services:4.3.15") classpath('io.fabric.tools:gradle:1.28.1') classpath("org.jetbrains.kotlin:kotlin-gradle-plugin:$kotlinVersion") // enable sentry AGP - https://docs.sentry.io/platforms/react-native/manual-setup/manual-setup/#enable-sentry-agp classpath("io.sentry:sentry-android-gradle-plugin:4.2.0") } ```

metro.config.js

```js /** * Metro configuration for React Native * https://github.com/facebook/react-native * * @format */ const { getDefaultConfig } = require("metro-config"); const { withSentryConfig } = require("@sentry/react-native/metro"); module.exports = (async () => { const { resolver: { sourceExts, assetExts } } = await getDefaultConfig(); const config = { transformer: { experimentalImportSupport: false, inlineRequires: true, babelTransformerPath: require.resolve("react-native-svg-transformer") }, resolver: { assetExts: assetExts.filter((ext) => ext !== "svg"), sourceExts: [...sourceExts, "svg", "jsx", "js", "ts", "tsx", "cjs", "mjs"], extraNodeModules: require("node-libs-react-native") } }; // Ensures all Sentry configuration is added to your Metro config // Sentry Metro Serializer which allows you to automatically generate Debug IDs for your applications' bundles. This is crucial for making source maps work properly with Sentry // withSentryConfig will remove any existing customSerializer. Guide for advanced setups can be found here - https://docs.sentry.io/platforms/react-native/manual-setup/metro/#wrap-your-custom-serializer. // Sentry Metro Serializer can't add Debug ID to the Hermes Composed Source Maps. // refer - https://docs.sentry.io/platforms/react-native/manual-setup/metro/ return withSentryConfig(config); })(); ```

Steps to reproduce:

Step 1
Step 2

Actual result:

Expected result:

The stack track should point to the exact line & file the error triggered from.

ayush-leap commented 3 weeks ago

I tried upgrading to v5.30.0 as well, facing same issue.

ayush-leap commented 3 weeks ago

Same issue happening on iOS as well. Tested iOS with RN 0.71.19 + Sentry 5.30.0

Artifact Bundles

![Image](https://github.com/user-attachments/assets/12696f84-3b5c-4b3b-9d5b-d4a149796926)

Sentry Stack Trace

![Image](https://github.com/user-attachments/assets/c87847b8-248b-42de-8966-4ea4ac720022) ![Image](https://github.com/user-attachments/assets/eb59c6e4-70bb-44ff-bf43-16490929b2f1)

`npx metro-symbolicate main.jsbundle.map < stack.txt`

```bash Error: ios TEST: v0.33.3 vote Error 0.0.5 at onVoteButtonPress(src/screens/CosmosHome/components/TxButtonGroup/index.tsx:113:onVoteButtonPress) at handleOnPress(src/screens/CosmosHome/components/TxButton/index.tsx:41:handleOnPress) at _performTransitionSideEffects(node_modules/react-native/Libraries/Pressability/Pressability.js:758:_performTransitionSideEffects) at _receiveSignal(node_modules/react-native/Libraries/Pressability/Pressability.js:695:_receiveSignal) at onResponderRelease(node_modules/react-native/Libraries/Pressability/Pressability.js:525:responderEventHandlers.onResponderRelease) at apply(native) at invokeGuardedCallbackImpl(node_modules/react-native/Libraries/Renderer/implementations/ReactNativeRenderer-prod.js:22:invokeGuardedCallbackImpl) at apply(native) at invokeGuardedCallback(node_modules/react-native/Libraries/Renderer/implementations/ReactNativeRenderer-prod.js:40:invokeGuardedCallback) at apply(native) at invokeGuardedCallbackAndCatchFirstError(node_modules/react-native/Libraries/Renderer/implementations/ReactNativeRenderer-prod.js:53:invokeGuardedCallbackAndCatchFirstError) at executeDispatch(node_modules/react-native/Libraries/Renderer/implementations/ReactNativeRenderer-prod.js:73:executeDispatch) at executeDispatchesAndReleaseTopLevel(node_modules/react-native/Libraries/Renderer/implementations/ReactNativeRenderer-prod.js:1125:executeDispatchesAndReleaseTopLevel) at forEach(native) at forEachAccumulated(node_modules/react-native/Libraries/Renderer/implementations/ReactNativeRenderer-prod.js:361:forEachAccumulated) at anonymous(node_modules/react-native/Libraries/Renderer/implementations/ReactNativeRenderer-prod.js:1160:batchedUpdates$argument_0) at batchedUpdatesImpl(node_modules/react-native/Libraries/Renderer/implementations/ReactNativeRenderer-prod.js:8457:batchedUpdatesImpl) at batchedUpdates(node_modules/react-native/Libraries/Renderer/implementations/ReactNativeRenderer-prod.js:1106:batchedUpdates) at _receiveRootNodeIDEvent(node_modules/react-native/Libraries/Renderer/implementations/ReactNativeRenderer-prod.js:1137:_receiveRootNodeIDEvent) at receiveTouches(node_modules/react-native/Libraries/Renderer/implementations/ReactNativeRenderer-prod.js:1209:ReactNativePrivateInterface.RCTEventEmitter.register$argument_0.receiveTouches) at apply(native) at __callFunction(node_modules/react-native/Libraries/BatchedBridge/MessageQueue.js:427:__callFunction) at anonymous(node_modules/react-native/Libraries/BatchedBridge/MessageQueue.js:113:__guard$argument_0) at __guard(node_modules/react-native/Libraries/BatchedBridge/MessageQueue.js:368:__guard) at callFunctionReturnFlushedQueue(node_modules/react-native/Libraries/BatchedBridge/MessageQueue.js:112:callFunctionReturnFlushedQueue)% ```

Bundle React Native code and images

```bash export NODE_OPTIONS="--max-old-space-size=8192" export SENTRY_PROPERTIES=sentry-leap-release.properties set -e WITH_ENVIRONMENT="../node_modules/react-native/scripts/xcode/with-environment.sh" REACT_NATIVE_XCODE="../node_modules/react-native/scripts/react-native-xcode.sh" SENTRY_XCODE="../node_modules/@sentry/react-native/scripts/sentry-xcode.sh" BUNDLE_REACT_NATIVE="/bin/sh $SENTRY_XCODE $REACT_NATIVE_XCODE" /bin/sh -c "$WITH_ENVIRONMENT \"$BUNDLE_REACT_NATIVE\"" # send runtime JavaScript dependencies in events for React Native apps /bin/sh ../node_modules/@sentry/react-native/scripts/collect-modules.sh ```

Upload Debug Symbols to Sentry

```bash export NODE_OPTIONS=--max-old-space-size=8192 export SENTRY_PROPERTIES=sentry-leap-release.properties /bin/sh ../node_modules/@sentry/react-native/scripts/sentry-xcode-debug-files.sh ```

kahest commented 2 weeks ago

@ayush-leap thank you for the detailed report, we'll investigate and follow up here

krystofwoldrich commented 2 weeks ago

Hi @ayush-leap, thank you for the message,

have you also upgraded the Sentry React Native SDK when changing from react-native@0.69.9 to react-native@0.71.19? Were you using the Sentry Metro Plugin @sentry/react-native/metro which generates Debug IDs?

What version of Sentry Self hosted are you using? Have you check the backend logs, is are there any symbolication errors?

ayush-leap commented 2 weeks ago

have you also upgraded the Sentry React Native SDK when changing from react-native@0.69.9 to react-native@0.71.19?

@krystofwoldrich no, during the RN upgrade we didn't upgrade Sentry as we were already on 5.22.2. after we pushed upgrade live, saw stack traces issue & only then I tried to upgrade to 5.30.0 to check if it resolves the issue but it didn't.

Were you using the Sentry Metro Plugin @sentry/react-native/metro which generates Debug IDs?

yes, we were using sentry's metro plugin previously as well. you can refer the implementation in PR description, it's unchanged.

What version of Sentry Self hosted are you using? Have you check the backend logs, is are there any symbolication errors?

we're using sentry's 24.1.0 at backend. yes, we did see below error log at backend, but team is unsure what it means or ways to resolve it. is this a backend issue? if yes, any tips on resolving this?

backend logs

``` urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='symbolicator', port=3021): Max retries exceeded with url: /symbolicate-js?timeout=5&scope=5 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.10/site-packages/sentry/tasks/symbolication.py", line 249, in _do_symbolicate_event symbolicated_data = symbolication_function(symbolicator, data) File "/usr/local/lib/python3.10/site-packages/sentry/lang/javascript/processing.py", line 240, in process_js_stacktraces response = symbolicator.process_js( File "/usr/local/lib/python3.10/site-packages/sentry/lang/native/symbolicator.py", line 219, in process_js return self._process("symbolicate_js_stacktraces", "symbolicate-js", json=json) File "/usr/local/lib/python3.10/site-packages/sentry/lang/native/symbolicator.py", line 105, in _process json_response = session.create_task(path, **kwargs) File "/usr/local/lib/python3.10/site-packages/sentry/lang/native/symbolicator.py", line 359, in create_task return self._request(method="post", path=path, params=params, **kwargs) File "/usr/local/lib/python3.10/site-packages/sentry/lang/native/symbolicator.py", line 293, in _request response = self.session.request(method, url, timeout=self.timeout + 1, **kwargs) File "/usr/local/lib/python3.10/site-packages/sentry/net/http.py", line 189, in request response = _Session.request(self, *args, **kwargs) File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) File "/usr/local/lib/python3.10/site-packages/requests/adapters.py", line 519, in send raise ConnectionError(e, request=request) ```

krystofwoldrich commented 1 week ago

@ayush-leap Thank you for the details, it looks like the symbolication fails on the backend.

I'll forward this to my colleagues, who can take a look at the backend error.

ayush-leap commented 1 week ago

I'll forward this to my colleagues, who can take a look at the backend error.

@krystofwoldrich appreciate it, waiting for the response.

getsentry / sentry-react-native

React Native Javascript not symbolicated self-hosted backend error #4038