Hermes, sentry source map is showing wrong code in both codepush and production build, but not when running in dev

[kopax-polyconseil]

Environment

• How do you use Sentry?

self-hosted Sentry 21.4.037b2bca

• Which SDK and version?

@sentry/react-native: 3.1.1

Steps to Reproduce

1. git clone https://github.com/pass-culture/pass-culture-app-native.git
   cd pass-culture-app-native
   nvm use
   yarn 

2. Source this file https://github.com/pass-culture/pass-culture-app-native/blob/master/scripts/upload_sourcemaps_to_sentry.sh#L7-L35 with . scripts/upload_sourcemaps_to_sentry.sh, this follow the step to generate sentry source map for react native with hermes
3. upload the source map to sentry using the same script
4. generate an error like in here: https://github.com/pass-culture/pass-culture-app-native/blob/master/src/features/cheatcodes/pages/Navigation/Navigation.tsx#L86-L92

Expected Result

I expect to see the error in sentry with the source maps showing the proper code, like when I run the app in __DEV__,

Actual Result

Production build does show some unexpected source map for the same error :

When I check the minified version, we can clearly see that the error happen in onPress:

This is the same error with Android :

Again we clearly see that the error is in onPress:

[jennmueng]

Discussing this issue via DM on discord.

[marandaneto]

Discussing this issue via DM on discord.

Please post the output here if you find out, so others can learn.

[kopax-polyconseil]

Yes, I will. So far we are still stuck, it seems that @jennmueng can see with the provided event.json and builded .map that we have an issue, but without being able to build the application it seems hard for him to tell why do we have wrong source map.

I have offered some help to build our app, and a peer programming session. I will keep you tuned.

As we have no hint at all why doing what's explained on sentry documentation, if anyone want to jump into this bug resolution and help us troubleshoot, it would still be welcomed.

[delgiudices]

I'm experiencing something similar

[marandaneto]

@kopax-polyconseil please provide a minimal reproducible example on Github, it should be an empty app that has only Sentry and its necessary bits to exemplify the problem.

[marandaneto]

Closing the issue as a part of large repository cleanup, due to it being inactive and/or outdated. Please do not hesitate to ping me if it is still relevant, and I will happily reopen and work on it. Cheers!

[arekm213]

I'm having similar issue to this using "@sentry/react-native": "3.4.0", @kopax-polyconseil were you able to fix it somehow?

[jennmueng]

I'm having similar issue to this using "@sentry/react-native": "3.4.0", @kopax-polyconseil were you able to fix it somehow?

The issue that was happening here was that the sourcemaps themselves were malformed and was not due to a bug on Sentry. Have been able to reproduce it with the provided bundle and sourcemaps but I do not have a reproducible app example so I have no idea what's causing it.

For future reference, here is how you manually symbolicate a React Native stacktrace to debug: https://reactnative.dev/docs/0.64/symbolication

[kopax-polyconseil]

Hello, we have the reproduction in our backlog and it should be priorized in the next month.

Also we did already follow the documentation and have a configuration made from it, you already checked and everything is correct. There is likely an issue with sentry or Hermes source maps. Upgrading our sentry server and cli did not help.

@marandaneto I would only advise to leave this open as we still have the minimal reproduction requested by @jennmueng in our backlog and it is one of the top priority of our team. On Wed, Apr 13, 2022, 11:41 PM Jenn Mueng @.***> wrote:

I'm having similar issue to this using @.***/react-native": "3.4.0", @kopax-polyconseil https://github.com/kopax-polyconseil were you able to fix it somehow?

The issue that was happening here was that the sourcemaps themselves were malformed and was not due to a bug on Sentry. Have been able to reproduce it with the provided bundle and sourcemaps but I do not have a reproducible app example so I have no idea what's causing it.

For future reference, here is how you manually symbolicate a React Native stacktrace to debug: https://reactnative.dev/docs/0.64/symbolication

— Reply to this email directly, view it on GitHub https://github.com/getsentry/sentry-react-native/issues/2087#issuecomment-1098514806, or unsubscribe https://github.com/notifications/unsubscribe-auth/ASQTMPTTABXJR5FHZRT5BGLVE45RJANCNFSM5PH2RRHA . You are receiving this because you were mentioned.Message ID: @.***>

[kopax-polyconseil]

Hello, as stated before, we have prepared a minimal reproduction here : https://github.com/pass-culture/pass-culture-app-native/pull/2917

You can test it straight using this script:

git clone  --single-branch --branch PC-14133 https://github.com/pass-culture/pass-culture-app-native.git sentry-minimal-app 
cd sentry-minimal-app 
nvm use # or `nvm install v16.13`
yarn --force
yarn apk:build
yarn apk:install
. scripts/upload_sourcemaps_to_sentry.sh
create_sourcemaps android index.android.bundle

• Everyone can now build it.
• I can still remove some more code but 99% of the app disappeared

To generate a sentry error :

1. press Tout passer in the upper right corner :
   
2. press the Sentry button
3. The error will be send to sentry with a recognizable ID
4. Find it on sentry

Wrong sourcemaps mapping only happen in production build, if you run in __DEV__ , it will work fine.

[marandaneto]

@kopax-polyconseil This is not a minimal reproduction example but I will try to run it early next week, If I run into troubles because of your environment, I will need an empty-template project with Sentry demonstrating the problem as requested before.

[kopax-polyconseil]

Hello @marandaneto , I have tested in an empty environment and it can run, all the keys are here. Also, I am starting today a latest create react native app with latest dependency and selfhosted then sentry.io project, to check if the problem persist, as you requested before. I hope all of these configurations will help us understand where it is failing soon.

[kopax-polyconseil]

Hi @marandaneto , I just created a minimal reproduction example starting from scratch using the latest tools and hermes and I got the same result on our self hosted instance.

I will redo the test on sentry.io but I doubt result will be different.

Reproduction latest minimal case

git clone   --single-branch --branch master https://github.com/kopax-polyconseil/testsentrynative.git
cd testsentrynative
nvm use # or nvm install 16; nvm use
yarn --force
yarn apk:build

## install apk on device
yarn apk:install

## create sourcemaps
mkdir -p sourcemaps
npx react-native bundle --platform android --dev false --entry-file index.js --bundle-output sourcemaps/index.android.bundle --sourcemap-output sourcemaps/index.android.bundle.packager.map
node_modules/hermes-engine/linux64-bin/hermesc -O -emit-binary -output-source-map -out=sourcemaps/index.android.bundle.hbc sourcemaps/index.android.bundle
node node_modules/react-native/scripts/compose-source-maps.js sourcemaps/index.android.bundle.packager.map sourcemaps/index.android.bundle.hbc.map -o sourcemaps/index.android.bundle.map

## Upload sourcemaps
node_modules/@sentry/cli/bin/sentry-cli releases files 1.000.0-android upload-sourcemaps --dist 1000000-android --strip-prefix /home/dka/workspace/github.com/pass-culture/pass-culture-app-native --rewrite sourcemaps/index.android.bundle sourcemaps/index.android.bundle.map --log-level debug

Result in __DEV__

Result in prod build

[kopax-polyconseil]

For info, I also created a reproduction using sentry.io instead of self hosted:

git clone   --single-branch --branch sentry.io https://github.com/kopax-polyconseil/testsentrynative.git
cd testsentrynative
nvm use # or nvm install 16; nvm use
yarn --force
yarn apk:build

## install apk on device
yarn apk:install

## create sourcemaps
mkdir -p sourcemaps
npx react-native bundle --platform android --dev false --entry-file index.js --bundle-output sourcemaps/index.android.bundle --sourcemap-output sourcemaps/index.android.bundle.packager.map
node_modules/hermes-engine/linux64-bin/hermesc -O -emit-binary -output-source-map -out=sourcemaps/index.android.bundle.hbc sourcemaps/index.android.bundle
node node_modules/react-native/scripts/compose-source-maps.js sourcemaps/index.android.bundle.packager.map sourcemaps/index.android.bundle.hbc.map -o sourcemaps/index.android.bundle.map

## Upload sourcemaps
node_modules/@sentry/cli/bin/sentry-cli releases files 1.000.0-android upload-sourcemaps --dist 1000000-android --strip-prefix /home/dka/workspace/github.com/pass-culture/pass-culture-app-native --rewrite sourcemaps/index.android.bundle sourcemaps/index.android.bundle.map --log-level debug

This reproduction crash upon startup when builded (dev works ok)

[marandaneto]

@kopax-polyconseil I will only check the source maps issue, the crash on start is likely your environment since it works for all the other cases, so please double-check it and fix the issue, otherwise, I won't be able to run your example.

[kopax-polyconseil]

so please double-check it and fix the issue, otherwise, I won't be able to run your example.

@marandaneto I did 3 reproductions:

1. A reduced pass Culture application with our self-hosted Sentry.
2. A new react-native application with our self-hosted Sentry.
3. A new react-native application with not a self-hosted Sentry (sentry.io).

If (3) can't run, it is still the same source code as (2), and (1) and (2) are similar in term of problem. So I don't understand why you can't check the issue. Especially that it seems the issue can be tested without Sentry using cli tools, ex:

npx source-map-cli resolve sourcemaps/index.android.bundle.map  1 1777848
# Or you can use our sentry-cli v2 now. 
sentry-cli sourcemaps resolve index.android.bundle.map -c 1 -l 1777848 

will show what sentry shows

So the issue are in the source maps generation, I presume it must be in one of the commands here:

npx react-native bundle --platform android --dev false --entry-file index.js --bundle-output sourcemaps/index.android.bundle --sourcemap-output sourcemaps/index.android.bundle.packager.map
node_modules/hermes-engine/linux64-bin/hermesc -O -emit-binary -output-source-map -out=sourcemaps/index.android.bundle.hbc sourcemaps/index.android.bundle
node node_modules/react-native/scripts/compose-source-maps.js sourcemaps/index.android.bundle.packager.map sourcemaps/index.android.bundle.hbc.map -o sourcemaps/index.android.bundle.map

We can give you access to our self hosted sentry. And let me know if you really want me to fix the (3) reproduction in order to look into the problem despite it seems not to be necessary.

[kopax-polyconseil]

So after checkin the 3rd reproduction it is now working, I needed to change the build apk commande from ./gradlew assemble to ./gradlew assembleRelease

Here is a sentry.io error which also have unfortunately the same problem as the self hosted : https://sentry.io/organizations/pass-cultureu/issues/3243092397/?project=6212953&query=is%3Aunresolved+V09ST&statsPeriod=14d

[marandaneto]

@jennmueng can you double check that, please?

[jennmueng]

@kopax-polyconseil @marandaneto I apologize for the delay, I finally finished my finals.

Anyways, I've reproduced the issue with the bad source maps, however, I found that if I use the source maps that are automatically generated by our react native build scripts those work perfectly. Upon further investigation I found that the bundle output from the build script is correct, however when you manually run the bundle command, the output is not the same.

I will need some time to investigate this further.

[owjsub]

I was able to get sourcemaps working for a manually built jsbundle w/ hermes enabled for Android used for Code Push with these steps

1. When building the jsbundle manually w/ hermes for Android, I used the exact same build and sourcemap steps for hermes as seen in node_modules/react-native/react.gradle. The steps are generally the same as seen in sentry docs for manually generating sourcemaps for hermes with a few differences around file naming conventions. (However, I had avoid the step that moves and renames the index.android.bundle.hbc to index.android.bundle. If I tried to upload this renamed index.android.bundle to sentry, it showed up as 0 bytes for file size in my release artifacts and I got a "Source code was not found" error in my events. I had to upload the original index.android.bundle that was built from the react-native bundle command and the index.android.bundle.map created in the compose-source-maps.js script.)

2. Each time I tested a change to generating sourcemaps, I created a unique dist such as android-test1, then I recompiled the app and regenerated+uploaded the new sourcemaps to test if my source maps were working. Otherwise, even though I replaced a sourcemap for a release / dist, an older cached copy of the source map was still in effect for my events.

Environment

react-native: 0.68.1
@sentry/react-native: 3.2.13
@sentry/cli: 1.72.0

[kopax-polyconseil]

Ho @jennmueng and @owjsub thanks for your investigation. We use codepush so it's a necessity for us to build and upload out sourcemap manually.

I understand the 2nd point but I really don't get what you did to make it work on your 1st point.

Why did it work ? Is this a solution ? Would you mind sharing a reproduction or should I try on my own to get the same result ?

What are the next steps ? Thanks again !

[owjsub]

This is my approach to manually building the jsbundle w/ hermes for Android. Source maps was working for me with this:

PLATFORM=android
BUILD_ENV=dev
BUNDLE_FILE_NAME=index.android.bundle
SENTRY_DIST=android-test1
SENTRY_RELEASE=test@1.0.0

mkdir -p "./build/${PLATFORM}/${BUILD_ENV}"
mkdir -p "./sourcemap/${PLATFORM}/${BUILD_ENV}"

npx react-native bundle \
  --config './metro.config.js' \
  --platform $PLATFORM \
  --entry-file './index.js' \
  --bundle-output "./build/${PLATFORM}/${BUILD_ENV}/${BUNDLE_FILE_NAME}" \
  --sourcemap-output "./sourcemap/${PLATFORM}/${BUILD_ENV}/${BUNDLE_FILE_NAME}.map" \
  --assets-dest "./build/${PLATFORM}/${BUILD_ENV}" \
  --reset-cache \
  --dev false \
  --minify false

#
# Hermes Sourcemap logic for Android
# ref: node_modules/react-native/react.gradle
#
mv "./sourcemap/${PLATFORM}/${BUILD_ENV}/${BUNDLE_FILE_NAME}.map" \
  "./sourcemap/${PLATFORM}/${BUILD_ENV}/${BUNDLE_FILE_NAME}.packager.map"

# make sure to choose the correct folder depending on what OS you're building the bundle on. Use 'osx-bin' for macOS.
node_modules/hermes-engine/osx-bin/hermesc -O -emit-binary -output-source-map \
  -out="./sourcemap/${PLATFORM}/${BUILD_ENV}/${BUNDLE_FILE_NAME}.hbc" \
  "./build/${PLATFORM}/${BUILD_ENV}/${BUNDLE_FILE_NAME}"

echo "Move: sourcemap/${PLATFORM}/${BUILD_ENV}/${BUNDLE_FILE_NAME}.hbc.map -> sourcemap/${PLATFORM}/${BUILD_ENV}/${BUNDLE_FILE_NAME}.compiler.map"
mv "./sourcemap/${PLATFORM}/${BUILD_ENV}/${BUNDLE_FILE_NAME}.hbc.map" \
  "./sourcemap/${PLATFORM}/${BUILD_ENV}/${BUNDLE_FILE_NAME}.compiler.map"

node node_modules/react-native/scripts/compose-source-maps.js \
  "./sourcemap/${PLATFORM}/${BUILD_ENV}/${BUNDLE_FILE_NAME}.packager.map" \
  "./sourcemap/${PLATFORM}/${BUILD_ENV}/${BUNDLE_FILE_NAME}.compiler.map" \
  -o "./sourcemap/${PLATFORM}/${BUILD_ENV}/${BUNDLE_FILE_NAME}.map"

echo "Source map hermes transform complete."

node_modules/@sentry/cli/bin/sentry-cli \
  --auth-token $SENTRY_AUTH_TOKEN \
  releases \
  --org testorg \
  --project testproject \
  files "$SENTRY_RELEASE" \
  upload-sourcemaps \
  --strip-prefix "$PWD" \
  --dist $SENTRY_DIST \
  ./build/${PLATFORM}/${BUILD_ENV}/${BUNDLE_FILE_NAME} \
  ./sourcemap/${PLATFORM}/${BUILD_ENV}/${BUNDLE_FILE_NAME}.map

[jennmueng]

@kopax-polyconseil Ok, after further testing and trying out @owjsub's process I have found the solution to get the sourcemaps on your reproduction to work.

1. You want to pass --minify false to the first bundle script, without it, after composing it seems to generate malformed source maps you can see this in action by running the manual symbolicator with the composed source maps with and without modification in the first bundle step.
2. You want to upload the index.android.bundle.hbc as your bundle, the artifact on Sentry dashboard showing 0 Bytes is actually correct. Make sure you rename this to index.android.bundle before uploading though.
3. You also want to upload the artifacts to a brand new release/dist combo, as Sentry seems to have a bug where it still uses the original sourcemaps even though they're deleted/rewritten cc @marandaneto @bruno-garcia

I'll have to further test this on more apps to see if these facts are universal to other apps too before I will update the documentation accordingly. Interesting that @owjsub says theirs works using the original bundle and not the .hbc...could be related to the bug I mentioned in point 3 ?

[marandaneto]

@jennmueng point 3 is likely a caching issue, if you are uploading to the same release and version and testing right away, that's likely the case, ideally, when testing, always test with a new release and/or version, otherwise we've to wait for the caching to be invalidated, not sure how long that takes.

@jennmueng we can add it to the https://docs.sentry.io/platforms/react-native/troubleshooting/ page as a possible solution for this specific problem.

Another solution is actually having a new section (hermes + codepush) in the codepush page since this comes often, having the code snippet that works with a template app is already good enough, if we have to tweak the scripts later for more complex apps, is also fine.

[kopax-polyconseil]

Thanks, I am not sure what I should try first, but I think I will start with @jennmueng recommendation.

I think (3) is not a problem for us, we have 3 release and multiple code push:

1. testing: happen on merge master and we use hash from ci always to identify the dist
2. staging: happen once per tag and we do not use a hash but a different env
3. produciton: happen once per tag and we do not use a hash but a different env

We also have some codepush, but we happen the +codePushId on each codepush to the dist.

@marandaneto by clearing the cache, what cache exactly are you referring to?

[kopax-polyconseil]

I have tried to follow (1) and add --minify false and --reset-cache, (2) was already configured like this in our source map builder script and (3), but I now get this:

I also tried to test the symbolicate command:

$ adb logcat -d | npx metro-symbolicate android/app/build/generated/sourcemaps/react/apptesting/release/index.android.bundle.map 
TypeError: Line must be greater than or equal to 1, got 0

Did I understood you both wrong ?

You want to upload the index.android.bundle.hbc as your bundle, the artifact on Sentry dashboard showing 0 Bytes is actually correct. Make sure you rename this to index.android.bundle before uploading though.

Perhaps (2) is not properly configured ? Can you elaborate on this ? isn't it what is happening in here ? Or should I just skip this command and use mv?

[marandaneto]

@kopax-polyconseil the caching on the backend system, you can't do that right now, it's an issue that we are working on. Every time you run a new test that uploads source maps, please use a new release and a new dist, otherwise, there's the chance that the backend uses the cached source maps, and not the latest uploaded.

[kopax-polyconseil]

please use a new release and a new dist

What happen if I only update a new dist ? Will the combo still use the cached version ? How long is the cached version lasting ?

[kopax-polyconseil]

I have tried to do:

$ ll sourcemaps/
total 27064
drwxr-xr-x 27 dka 10000     4096 May 13 11:08 ..
-rw-r--r--  1 dka 10000 10352667 May 13 11:09 index.android.bundle.packager.map
-rw-r--r--  1 dka 10000  5451626 May 13 11:09 index.android.bundle
-rw-r--r--  1 dka 10000  2598064 May 13 11:09 index.android.bundle.hbc
-rw-r--r--  1 dka 10000  2024482 May 13 11:09 index.android.bundle.compiler.map
-rw-r--r--  1 dka 10000  7269525 May 13 11:10 index.android.bundle.map
drwxr-xr-x  2 dka 10000     4096 May 13 11:10 .

$ mv sourcemaps/index.android.bundle sourcemaps/index.android.bundle.original
$ mv sourcemaps/index.android.bundle.hbc sourcemaps/index.android.bundle

Then uploading sourcemaps with:

$ node_modules/@sentry/cli/bin/sentry-cli releases files "${RELEASE}" \
>     upload-sourcemaps \
>     --dist "${DIST}" \
>     --strip-prefix "${PWD}" \
>     "${SOURCEMAPS_DIR}/${SOURCEMAPS_NAME}" "${SOURCEMAPS_DIR}/${SOURCEMAPS_NAME}.map"
> Found 1 release file
> Found 1 release file
> Analyzing 2 sources
> Analyzing completed in 0.155s
> Rewriting sources
> Rewriting completed in 0.167s
> Adding source map references
> Bundling files for upload... 
> Bundling completed in 0.279s
> Optimizing completed in 0.005s
> Uploading completed in 1.013s
> Uploaded release files to Sentry
> Processing completed in 0.105s
> File upload complete (processing pending on server)
> Organization: sentry
> Project: pass-culture-app-native-minimalist
> Release: 2.183.2-android
> Dist: 20183002-android

Result in release with 0kb:

But sourcemaps aren't working, I can't uncollapse the code on sentry.

However this show proper line with error:

$ npx source-map-cli resolve sourcemaps/index.android.bundle.map  1 1777848

Maps to /home/dka/workspace/github.com/pass-culture/pass-culture-app-native/src/features/cheatcodes/pages/Navigation/index.tsx:38:47 

              eventMonitoring.captureException(new Error(message))

[kopax-polyconseil]

I just managed to get this thanks to your indication:

This is my PR with fix:

https://github.com/pass-culture/pass-culture-app-native/pull/2993

We haven't tested iOS yet, codepush, and all our env dist/release pair. I'll check again when back on vacation on the 30th. :)

Thanks so much for all those effort in investigating the issue :mechanical_arm:

[kopax-polyconseil]

Tested in our app with a code push and still not working but different behavior:

I will test with a new release/dist pair without code push

[kopax-polyconseil]

So after testing, it appears that codepush and iOS aren't working (both as in my screenshot)

It only work on Android without code push.

[owjsub]

Regarding the renamed hbc file (which appears in sentry artifacts as 0 bytes), I tested this in a brand new release and the symbol maps worked as expected, but it's not clear whether this only works because I previously uploaded the original index.android.bundle in a previous release and maybe sentry is still using a cached copy of that original bundle from the previous release?

For now, I will stick with uploading the original index.android.bundle and not upload the renamed hbc that appears as 0 bytes. I was getting inconsistent behavior with the renamed hbc anyways. If anyone could offer more guidance on this, that would be awesome!

[jennmueng]

@kopax-polyconseil For iOS, I just tried on your reproduction, you will want to upload the original bundle.

I am not able to test with code push right now, but I would assume it would use the original bundle as well.

This is very unexpected behavior and is not consistent with how it was previously...I will get to the bottom of this.

[kopax-polyconseil]

Hello @jennmueng , I have tried to follow your recommendation and to use the original bundle, now that we use --minify false, we are able to see the sentry source map, but not from the source code, instead it shows some compiled code, it is still better than nothing but is this expected ?

I will try now using code push on both iOS and android.

[kopax-polyconseil]

iOS code push does not work well, without changing the source code, it should be the same as the not code push build for iOS.

[kopax-polyconseil]

Android code push using the original bundle, same problem as for iOS code push

[kopax-polyconseil]

Using hbc bundle, both iOS and Android code push show no source code, exemple :

If I use a custom dist and release with not hbc bundle, it does not work, I presume the dist as to be the same as for the hard builded version:

What are the next steps ? are there any action taken by sentry team regarding this issue ?

[jennmueng]

Hello @jennmueng , I have tried to follow your recommendation and to use the original bundle, now that we use --minify false, we are able to see the sentry source map, but not from the source code, instead it shows some compiled code, it is still better than nothing but is this expected ?

I will try now using code push on both iOS and android.

I've been running into similar issues, I will get back to you.

[marandaneto]

@jennmueng I've upgraded our RN version to 0.67.4 Should we test it using either this one or 0.68.x? do you have any updates? thanks.

[marandaneto]

Closing in favor of https://github.com/getsentry/sentry-react-native/issues/2244#issuecomment-1160045688 This problem exists even if there's no codepush.

[kopax-polyconseil]

Hi @marandaneto , I can see that you closed the issue in favor of #2244

Our team was waiting for you to finish the troubleshooting regarding hard build: wrong line number + compiled source instead of source code

We were also expecting a resolution regarding code push, currently we are not able to manually upload source maps for code push.

I feel those are two different issue than the one from #2244, where codepush seems to work for him, but is in the same state as our iOS build.

Can we keep this open for our organization tracking ? We are a bit stuck with configuring Sentry sourcemaps (iOS hard + code push) and we where watching this issue regularly.

[marandaneto]

This is blocked due to the linked issues above, the problem isn't with CodePush but rather with Hermes, so please be patient and follow the other issue to be on top of this.