search

getsentry / sentry

Developer-first error tracking and performance monitoring
https://sentry.io
Other
39.27k stars 4.21k forks source link

Sourcemaps works and then suddenly doesn't #23937

Closed cxspxr closed 1 year ago

cxspxr commented 3 years ago

Important Details

We have tried to configure sourcemaps multiple times in our project. And the story is that actually the solutions found in your guide, on medium, stackoverflow, and that we applied worked for a while and then they suddenly broke.

https://toolbox.12v.biz/packs/js/index-e89897c5421c54275a15.js https://toolbox.12v.biz/packs/js/index-e89897c5421c54275a15.js.map

Web archive in case we will re-build the project (but you can still visit the website and see the bundle name in sources with corresponding .map) https://web.archive.org/web/20210218082222/https://toolbox.12v.biz/packs/js/index-e89897c5421c54275a15.js https://web.archive.org/web/20210218082318/https://toolbox.12v.biz/packs/js/index-e89897c5421c54275a15.js.map

Both artifacts we have uploaded to sentry as ~/packs/js... and it shows that they're successfully uploaded and we can find these files in the release.

https://sourcemaps.io/ showing that we don't have //# sourceMappingUrl declaration but you can clearly see we have it (not in webarchive for some matter, maybe because webarchive put their comment there instead)

How are you running Sentry?

Saas (sentry.io)

Description

Sourcemaps doesn't work after some time.

Steps to Reproduce

Not sure how you would reproduce it, I still think that it something with our configuration, the only thing I really don't get is that how it works for a while and then doesn't.

config/webpack/production.js

process.env.NODE_ENV = process.env.NODE_ENV || "production"

const environment = require("./environment")
const SentryCliPlugin = require("@sentry/webpack-plugin")

if (process.env.NODE_ENV === "production") {
  environment.config.merge({ devtool: "source-map" })

  environment.plugins.prepend(
    "Sentry",
    new SentryCliPlugin({
      include: "./public/",
      ignore: ["node_modules"],
      authToken: process.env.SENTRY_AUTH_TOKEN,
      org: "our-org",
      project: "our-org",
      deploy: {
        env: process.env.ENVIRONMENT_IDENTIFIER,
      },
      stripPrefix: ["public"],
      setCommits: {
        repo: "our/repo",
        commit: process.env.SOURCE_VERSION,
      },
    })
  )
}

module.exports = environment.toWebpackConfig()

config/webpack/environment.js

const webpack = require("webpack")
const { environment } = require("@rails/webpacker")
const typescript = require("./loaders/typescript")
const GitRevisionPlugin = require("git-revision-webpack-plugin")

environment.loaders.append("typescript", typescript)

environment.plugins.prepend(
  "Define",
  new webpack.DefinePlugin({
    // On Heroku git is not available, but SOURCE_VERSION is provided
    GIT_VERSION: JSON.stringify(process.env.SOURCE_VERSION || new GitRevisionPlugin().commithash()),
    BUILD_TIME: JSON.stringify(new Date().toISOString()),
  })
)

environment.plugins.prepend(
  "Provide",
  new webpack.ProvidePlugin({
    $: "jquery",
    jQuery: "jquery",
    jquery: "jquery",
    "window.jQuery": "jquery",
  })
)

environment.config.externals = ["cloudinary"]

// Add support for CKEditor 5.
const CKEditorWebpackPlugin = require("@ckeditor/ckeditor5-dev-webpack-plugin")

environment.plugins.prepend(
  "CKEditor",
  new CKEditorWebpackPlugin({
    language: "en",
  })
)

// Define custom loaders for CKEditor's SVG and CSS files.
environment.loaders.append("CKEditorSVGLoader", require("./loaders/ckeditor/svg"))
environment.loaders.append("CKEditorCSSLoader", require("./loaders/ckeditor/css"))

// Tell the standard CSS and file loaders to ignore CKEditor's CSS and SVG files. We have our own loaders for those.
environment.loaders.get("css").exclude = /(\.module\.[a-z]+$)|(ckeditor5-[^/\\]+[/\\]theme[/\\].+\.css)/
environment.loaders.get("file").exclude = /ckeditor5-[^/\\]+[/\\]theme[/\\]icons[/\\][^/\\]+\.svg$/

module.exports = environment

What you expected to happen

It should work (and it did for a week) and it should map the sources to the minified webpack bundles correctly as it really does in browser when showing some stacktraces etc.

untitaker commented 3 years ago

reassigned because I don't know sourcemaps

HazAT commented 3 years ago

@cxspxr Just to double-check, you are not doing any deploys in the meantime of your JS bundles, right? You are also not updating release or dist when calling Sentry.init?

Speaking of, can you post how you initialize the Sentry SDK?

cxspxr commented 3 years ago

No, we're not doing any extra deploys, just that via webpack plugin sourcemaps.io tells us that bundle is incorrect and sourceMappingUrl is missing which is not true

app/javascript/packs/index.tsx

if (!window.location.host.includes("localhost") && !window.location.host.includes("192.168")) {
  Sentry.init({
    dsn: process.env.SENTRY_DSN,
    environment: process.env.SERVER_ENV || "development",
    beforeSend: (event, hint) => {
      if (!hint) return event
      const error = hint.originalException as any

      if (error && error.stack && error.stack.match(/syncfusion/)) {
        event.fingerprint = ["syncfusion", error.message]
      } else if (error && error.message === "Network Error") {
        event.fingerprint = ["network-error"]
      }

      return event
    },
  })
}
HazAT commented 3 years ago

@cxspxr Can you also share a link to an issue please?

cxspxr commented 3 years ago

Yes, sure: https://sentry.io/organizations/12vbiz/issues/2188684763/?project=1356811&referrer=slack

Oh, and now I noticed that the issue with working sourcemaps has been merged into the one where sourcemaps don't work. May it be the cause? But still I am not sure, it would be super great if you could ensure if everything is fine because it's 3rd time we tried to configure sourcemaps and it worked all 3 times with different approaches and after a while we still received not-mapped code issues.

The last issue that has been merged to that one contains the correct trace with sourcemaps working

github-actions[bot] commented 3 years ago

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you label it Status: Accepted, I will leave it alone ... forever!

"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀

kamilogorek commented 3 years ago

@cxspxr based on the issue you linked above:

I can see you have this option set, but that's the issue, so maybe there was some webpack misconfiguration. You can run your uploads with SENTRY_LOG_LEVEL=debug to see what command is exactly used for uploads (it'll confirm to you whether prefixes are passed correctly to cli).

cxspxr commented 3 years ago

Still no success. But sourcemaps do work in Chrome, I doubt it's webpack misconfiguration then

Not sure, tried to upload public path only, not stripping a prefix - the same issue.

st0012 commented 3 years ago

@cxspxr can you also share these information:

I'll take a look from the webpacker gem/Rails perspective

kamilogorek commented 3 years ago

@cxspxr can you share a link to the latest event that should but doesn't work?

cxspxr commented 3 years ago

@st0012

config/webpacker.yml

# Note: You must restart bin/webpack-dev-server for changes to take effect

default: &default
  source_path: app/javascript
  source_entry_path: packs
  public_output_path: packs
  cache_path: tmp/cache/webpacker
  check_yarn_integrity: false

  # Additional paths webpack should lookup modules
  # ['app/assets', 'engine/foo/app/assets']
  resolved_paths: []

  # Reload manifest.json on all requests so we reload latest compiled packs
  cache_manifest: false

  webpack_compile_output: true

  # Extract and emit a css file
  extract_css: false

  static_assets_extensions:
    - .jpg
    - .jpeg
    - .png
    - .gif
    - .tiff
    - .ico
    - .svg
    - .eot
    - .otf
    - .ttf
    - .woff
    - .woff2

  extensions:
    - .mjs
    - .tsx
    - .ts
    - .jsx
    - .js
    - .sass
    - .scss
    - .css
    - .module.sass
    - .module.scss
    - .module.css
    - .png
    - .svg
    - .gif
    - .jpeg
    - .jpg

development:
  <<: *default
  compile: true

  # Verifies that versions and hashed value of the package contents in the project's package.json
  check_yarn_integrity: true

  # Reference: https://webpack.js.org/configuration/dev-server/
  dev_server:
    https: false
    host: localhost
    port: 3035
    public: localhost:3035
    hmr: false
    # Inline should be set to true if using HMR
    inline: true
    overlay: true
    compress: true
    disable_host_check: true
    use_local_ip: false
    quiet: false
    headers:
      "Access-Control-Allow-Origin": "*"
    watch_options:
      ignored: "**/node_modules/**"

test:
  <<: *default
  compile: true

  # Compile test packs to a separate directory
  public_output_path: packs-test

production:
  <<: *default

  # Production depends on precompilation of packs prior to booting for performance.
  compile: false

  # Extract and emit a css file
  extract_css: true

  # Cache manifest.json for performance
  cache_manifest: true

Rails: 6.0.3.5 Webpacker: 5.1

cxspxr commented 3 years ago

@kamilogorek https://sentry.io/organizations/12vbiz/issues/2175965872/?environment=production&project=1356811&query=is%3Aunresolved

but virtually any on the front-end, they are all not mapped.

st0012 commented 3 years ago

Because in this comment @kamilogorek mentioned that the uploaded sourcemap contains the extra ~/public prefix that's causing the issue, I decided to reproduce the same result locally and see what could be the cause of that.

So I added webpacker to sentry-ruby SDK's example app in the webpacker-example branch (PR: https://github.com/getsentry/sentry-ruby/pull/1355). I also tried adopting the exact same webpacker configs as @cxspxr's application has.

But I can always get the source maps uploaded correctly

source maps uploaded correctly and have the stacktrace linked correctly (report link)

correctly linked stacktrace

The only way I can add the public prefix to the source map names is to add the urlPrefix option like:

  new SentryCliPlugin({
    include: "./public/",
    ignore: ["node_modules"],
    authToken: process.env.SENTRY_AUTH_TOKEN,
    org: "sentry-sdks",
    project: "sentry-ruby",    
    stripPrefix: ["public"],
    urlPrefix: "~/public" // <----- This line
  })

You can see that it now has the public prefix.

source maps with public prefix

Not sure if it's possible that urlPrefix is injected accidentally? I checked the source code of sentry-cli and sentry-webpacker-plugin but didn't find anything like a ENV to do that. @kamilogorek I hope this would help though.

(PR for the webpacker example https://github.com/getsentry/sentry-ruby/pull/1355)

cxspxr commented 3 years ago

For now I will proceed with 

    new SentryCliPlugin({
      include: "./public/packs/js/",
      ignore: ["node_modules"],
      authToken: process.env.SENTRY_AUTH_TOKEN,
      org: "12vbiz",
      project: "12vbiz",
      deploy: {
        env: process.env.ENVIRONMENT_IDENTIFIER,
      },
      urlPrefix: "~/packs/js",
      setCommits: {
        repo: "tonydehnke/12vBiz",
        commit: process.env.SOURCE_VERSION,
      },
    })

Because this seems to work for now. But previous configs worked as well for some time and then they eventually failed to map the sources after some time with new releases coming.

cxspxr commented 3 years ago

So again, config above did work for some time/some issues, some issues have been mapped, but for majority - it doesn't map MAPPED: https://sentry.io/organizations/12vbiz/issues/2291237732/?project=1356811&query=is%3Aunresolved https://sentry.io/organizations/12vbiz/issues/2172063029/?project=1356811&query=is%3Aunresolved https://sentry.io/organizations/12vbiz/issues/2073891396/?project=1356811&query=is%3Aunresolved https://sentry.io/organizations/12vbiz/issues/2291238086/?project=1356811&query=is%3Aunresolved

NOT MAPPED: https://sentry.io/organizations/12vbiz/issues/2291180045/?project=1356811&referrer=slack https://sentry.io/organizations/12vbiz/issues/2289961069/?project=1356811&query=is%3Aunresolved https://sentry.io/organizations/12vbiz/issues/2288291909/?project=1356811&query=is%3Aunresolved

Issues occurred on the same day, with the very same config, some being mapped, some not

cxspxr commented 3 years ago

But now files are uploaded without public in their path for sure, it should work

kamilogorek commented 3 years ago

Artifacts from your 2nd example are just broken, that's why it's not mapped. No idea why, and how your setup works, but this is the main issue.

Let's compare 2 examples.

Working

Issue: https://sentry.io/organizations/12vbiz/issues/2291237732/?project=1356811 Release: https://sentry.io/settings/12vbiz/projects/12vbiz/source-maps/c0401df7054790c410dbc566c587ea3c4722d79b/ File: ~/packs/js/index-c001f75d00abb164813d.js.map Frame position: 2:3194127

npx source-map-cli resolve index-c001f75d00abb164813d.js.map 2 3194127   
Maps to webpack:///app/javascript/components/CheckForm.tsx:192:64 (map)

          const checkItemElements = form.values[itemsFieldName].map((item, index) =>
                                                                ^

Broken

Issue: https://sentry.io/organizations/12vbiz/issues/2291180045/?project=1356811 Release: https://sentry.io/settings/12vbiz/projects/12vbiz/source-maps/ee75130aeee03214c60bbba6a1becc7231e9676d/ File: ~/packs/js/index-94f87e3b61a070045f4b.js.map Frame position: 1:3019812

npx source-map-cli resolve index-94f87e3b61a070045f4b.js.map 1 3019812
Could not resolve mapping:  Mapping not found

However, when we move one line lower, to line 2 it works just fine:

MMaps to webpack:///node_modules/@syncfusion/ej2-schedule/src/schedule/renderer/vertical-view.js:395:86 (offsetHeight)

      currentTimeEle.style.top = formatUnit(currentTimeEle.offsetTop - currentTimeEle.offsetHeight / 2);
                                                                                      ^

It appears that for some reason, your code is sometimes correctly detecting line number 2, and sometimes it skips the initial comment line /*! For license information please see index-94f87e3b61a070045f4b.js.LICENSE.txt */ and ignores it completely reporting as frames come from there.

We cannot solve it on the serverside, nor in the SDK, as we don't control input given to us.

tonydehnke commented 3 years ago

Is there a Sentry implementation specialist that we can hire to help resolve this?

kamilogorek commented 3 years ago

@tonydehnke feel free to join our Discord https://discord.com/invite/Ww9hbqr or explain your issue in detail either here or through DM/mail and I'll see what we can do.

tonydehnke commented 3 years ago

@kamilogorek - I'm a non-tech founder, so I can't really add much more than what has been shown above, unfortunately. Hence looking for someone that has experience with it to help get things working.

BYK commented 3 years ago

@tonydehnke you may want to look elsewhere then as GitHub issues are for reporting bug or feature requests. We cannot offer help to fix your setup or advertise this place to get help for you.

tonydehnke commented 3 years ago

Thanks @BYK - I understand. If you have any thoughts on where else would be good I would appreciate any suggestions.

BYK commented 3 years ago

@tonydehnke - I would have said the forums but haven't seen folks getting this kind of help there much. Maybe StackOverflow or just hiring someone from those one-off contracting sites?

github-actions[bot] commented 3 years ago

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you label it Status: Backlog or Status: In Progress, I will leave it alone ... forever!

"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀

BillSchumacher commented 3 years ago

We were having a similar problem, which turned out to be devs running the production version while testing fixes from feature branches. Which uploaded different source maps than what would have mapped to the deployed application.

Might be your issue or might not, figured I would throw it out there.

getsantry[bot] commented 1 year ago

Routing to @getsentry/telemetry-experience for triage, due by (vie). ⏲️

ashwoods commented 1 year ago

@tonydehnke Want to reach out to me (kyrix) on discord https://discord.gg/sentry if this issue still persists?

ashwoods commented 1 year ago

Closed due to inactivity, and hopefully improved with debug id sourcemaps: https://github.com/getsentry/team-webplatform-meta/issues/17