search

getsentry / sentry-cli

A command line utility to work with Sentry.
https://docs.sentry.io/cli/
BSD 3-Clause "New" or "Revised" License
896 stars 222 forks source link

Uploading DIF from Windows fails with "Invalid checksum" eror #1906

Open petrmanek opened 8 months ago

petrmanek commented 8 months ago

Environment

Steps to Reproduce

  1. Open Visual Studio 2019 Developer Command Prompt
  2. Build a C++ project using cmake with CMAKE_BUILD_TYPE set to RelWithDebInfo
  3. Run: sentry-cli upload-dif -o %SENTRY_ORGANIZATION% -p %SENTRY_PROJECT% --log-level=debug .

Expected Result

All debug information files should be uploaded to Sentry.

Actual Result

The sentry-cli call fails with exit code 1. The following message is printed:

error: Invalid checksum

This runs on a Windows VM. Please note that I run similar jobs on macOS and Linux VMs on the same computing cluster, which also call sentry-cli upload-dif with similar inputs at some point. Neither of these ever fail because of Sentry, only the Windows job does.

Logs

Here are (sanitized) logs from a typical run: sentry_fail.log

szokeasaurusrex commented 7 months ago

Hi @petrmanek, thank you for reporting this issue. Have you ever been able to run this command successfully on Windows? If yes, what was the most recent version of the Sentry CLI where running the command succeeded?

petrmanek commented 7 months ago

Have you ever been able to run this command successfully on Windows?

Yes, in fact I was running this in my CI/CD pipeline as recently as 2 weeks ago. The only thing that changed was time and some features of my software. I recently switched my C++ standard from C++17 to C++20. Could that have anything to do with it?

If yes, what was the most recent version of the Sentry CLI where running the command succeeded?

Everything worked with v2.12.0 until about 2 weeks ago, when the error started appearing. Since then I tried updating to the latest version (v2.25.3 as if writing), which made no difference.

szokeasaurusrex commented 7 months ago

I recently switched my C++ standard from C++17 to C++20. Could that have anything to do with it?

Perhaps, if making that change could have changed the debug information file format. You can use the sentry-cli debug-files check command to check whether the debug information files are usable. More information about the debug information file formats we accept is available here.

Please try running the sentry-cli debug-files check command against the files which are causing the issue, and let me know what the output is.

petrmanek commented 7 months ago

Perhaps, if making that change could have changed the debug information file format

I will try that. A job is currently running so I should have results in approximately an hour.

In the meantime, looking at logs I do not see anything out of ordinary. Mostly DLLs (x86_64 library) and PDBs (x86_64 debug companion) and a single EXE (x86_64 executable). Correct me if I am wrong, but if sentry-cli identifies these files correctly and displays UPLOADED next to their name, it implies that they are considered useful, right?

petrmanek commented 7 months ago

I have a follow up: sentry dif check seems to recognize my files just fine even after switching to C++20. The following would be a typical output:

20240122_16h08m10s_grim

szokeasaurusrex commented 7 months ago

Correct me if I am wrong, but if sentry-cli identifies these files correctly and displays UPLOADED next to their name, it implies that they are considered useful, right?

The code that the check command calls to determine whether the file is useful does not appear to get called from the upload code, so it could be possible that a file might not be considered usable even if it can get uploaded properly. Although I am not sure about that, since the code is pretty complex. But seeing that the files pass the dif check, there is probably a different problem.

Have you checked whether you are able to see the uploaded debug files in Sentry?

petrmanek commented 7 months ago

Apologies for a delayed response.

Have you checked whether you are able to see the uploaded debug files in Sentry?

Yes, they do and there appear to be no processing issues either.

The following log snippet of sentry-cli:

[...]
  UPLOADED 3875da5b-901f-4639-8fa2-5d1883c454fe-1 (plugins\app\csvfile.dll; x86_64 library)
  UPLOADED 7d509770-d1b8-4423-bf65-be6d87dc6162-1 (lib\scn.dll; x86_64 library)
  UPLOADED f4f2beaa-2b80-4bed-95e9-cf7b2e5e91c4-1 (share\debug\app\dacscan.pdb; x86_64 debug companion)
  UPLOADED 3ba9b8a0-3a72-4da4-9d78-080af7e842ee-1 (share\debug\app\mmfile.pdb; x86_64 debug companion)
  UPLOADED 179b21bb-a7d4-44a2-a06d-3b9275daf552-1 (share\debug\app\fitpix.pdb; x86_64 debug companion)
  UPLOADED 9bf1a6ce-5f6c-4040-bba8-bd66084e3f9d-1 (bin\crashpad_wer.pdb; x86_64 debug companion)
  UPLOADED 2337ece1-3804-47cb-86cf-377af91bcd00-1 (plugins\app\livehist.dll; x86_64 library)
[...]

...nicely corresponds with what I see on sentry.io:

20240124_11h30m26s_grim

szokeasaurusrex commented 7 months ago

@petrmanek Just want to make sure I am understanding everything correctly: my understanding is that besides the error message showing up in the logs, everything appears to be working in Sentry as expected? Or have you noticed anything wrong in Sentry (such as an error message stating that debug files are missing when you view an error event)?

petrmanek commented 7 months ago

besides the error message showing up in the logs, everything appears to be working in Sentry as expected?

That is correct, as far as I can tell. DIFs appear to be identified as useful, uploaded and visible online. Only the process for some reason exits with a failing exit code and prints the error: Invalid checksum message before dying.

Or have you noticed anything wrong in Sentry (such as an error message stating that debug files are missing when you view an error event)?

I have been on a lookout for those. Over the last two weeks, I have not noticed any such incident (or anything similar for that matter).

szokeasaurusrex commented 7 months ago

@petrmanek That is good to hear, glad that everything appears to work as expected, besides the error message.

Since our team is currently busy with other projects, and since the only symptom appears to be the error message in the logs, it might take us some time before we are able to fully investigate this issue.

However, if you begin noticing any other problems as a result of this issue (such as DIFs not being uploaded to Sentry, or a missing debug file error), please let us know right away! We would, of course, prioritize investigating this issue if it begins to negatively impact your experience using Sentry.

petrmanek commented 7 months ago

Since our team is currently busy with other projects, and since the only symptom appears to be the error message in the logs, it might take us some time before we are able to fully investigate this issue.

As you wish. I will modify my CI's deployment script for Windows to allow for non-zero exit code to be returned by sentry-cli at this stage for as long as this issue remains open. Please let me know if you would like to run any tests.

However, if you begin noticing any other problems as a result of this issue (such as DIFs not being uploaded to Sentry, or a missing debug file error), please let us know right away! We would, of course, prioritize investigating this issue if it begins to negatively impact your experience using Sentry.

I will do that if/when I notice anything serious. Realistically speaking, however, my pipeline has 50+ jobs, so I am unlikely to spot anything unless I start seeing unsymbolicated crashes at sentry.io.

szokeasaurusrex commented 7 months ago

Ok thank you, and I appreciate your understanding! Also, if anyone else experiences this issue, please leave a comment, so we can understand how common this problem is.

Mordil commented 3 months ago

We just started getting this error on our macOS GitHub Actions runner. We're not seeing releases & debug info in Sentry. The last release created from our CI was on May 24th, the last successful build we had.

We're using the Fastlane plugin, with the Sentry CLI installed via Homebrew. (version 2.31.2)

We started getting this error before we updated, as we were on a slightly older version of the CLI.

We haven't had any changes to our system before we started seeing this error.

We updated our Fastlane script to the following

  sentry_debug_files_upload(
    auth_token: SENTRY_AUTH_TOKEN,
    org_slug: 'dev-peek',
    project_slug: 'peekpro-inhouse',
    wait: false,
    log_level: 'trace'
  )

Other than just changing our script to allow failures from this - is there another remedy?

CI debug logs 
[10:48:49]: ---------------------------------------
[10:48:49]: --- Step: sentry_debug_files_upload ---
[10:48:49]: ---------------------------------------
[10:48:49]: Using sentry-cli 2.31.2
[10:48:49]: Starting sentry-cli...
[10:48:49]: DEBUG   2024-05-28 10:48:49.609613 -07:00 sentry-cli version: 2.31.2, platform: "darwin", architecture: "arm64"
[10:48:49]: INFO    2024-05-28 10:48:49.610240 -07:00 sentry-cli was invoked with the following command line: "/Users/administrator/actions-runner/_work/nomad/nomad/vendor/bundle/ruby/2.6.0/gems/fastlane-plugin-sentry-1.22.1/bin/sentry-cli-Darwin-universal" "debug-files" "upload" "." "--wait"
[10:48:49]: INFO    2024-05-28 10:48:49.610420 -07:00 Issuing a command for Organization: dev-peek Project: peekpro-inhouse
[10:48:49]: DEBUG   2024-05-28 10:48:49.611011 -07:00 request GET https://sentry.io/api/0/organizations/dev-peek/chunk-upload/
[10:48:49]: DEBUG   2024-05-28 10:48:49.611029 -07:00 using token authentication
[10:48:49]: DEBUG   2024-05-28 10:48:49.611045 -07:00 retry number 0, max retries: 0
[10:48:49]: DEBUG   2024-05-28 10:48:49.646666 -07:00 > GET /api/0/organizations/dev-peek/chunk-upload/ HTTP/1.1
[10:48:49]: DEBUG   2024-05-28 10:48:49.646678 -07:00 > Host: sentry.io
[10:48:49]: DEBUG   2024-05-28 10:48:49.646680 -07:00 > Accept: */*
[10:48:49]: DEBUG   2024-05-28 10:48:49.646682 -07:00 > Connection: TE
[10:48:49]: DEBUG   2024-05-28 10:48:49.646684 -07:00 > TE: gzip
[10:48:49]: DEBUG   2024-05-28 10:48:49.646686 -07:00 > User-Agent: sentry-cli/2.31.2
[10:48:49]: DEBUG   2024-05-28 10:48:49.647350 -07:00 > Authorization: ***
[10:48:49]: DEBUG   2024-05-28 10:48:49.784833 -07:00 < HTTP/1.1 200 OK
[10:48:49]: DEBUG   2024-05-28 10:48:49.784851 -07:00 < server: nginx
[10:48:49]: DEBUG   2024-05-28 10:48:49.784860 -07:00 < date: Tue, 28 May 2024 17:48:49 GMT
[10:48:49]: DEBUG   2024-05-28 10:48:49.784868 -07:00 < content-type: application/json
[10:48:49]: DEBUG   2024-05-28 10:48:49.784876 -07:00 < vary: Accept-Encoding,Accept-Language, Cookie
[10:48:49]: DEBUG   2024-05-28 10:48:49.784882 -07:00 < allow: GET, POST, HEAD, OPTIONS
[10:48:49]: DEBUG   2024-05-28 10:48:49.784889 -07:00 < access-control-allow-methods: GET, POST, HEAD, OPTIONS
[10:48:49]: DEBUG   2024-05-28 10:48:49.784901 -07:00 < access-control-allow-headers: X-Sentry-Auth, X-Requested-With, Origin, Accept, Content-Type, Authentication, Authorization, Content-Encoding, sentry-trace, baggage, X-CSRFToken
[10:48:49]: DEBUG   2024-05-28 10:48:49.784912 -07:00 < access-control-expose-headers: X-Sentry-Error, X-Sentry-Direct-Hit, X-Hits, X-Max-Hits, Endpoint, Retry-After, Link
[10:48:49]: DEBUG   2024-05-28 10:48:49.784920 -07:00 < access-control-allow-origin: *
[10:48:49]: DEBUG   2024-05-28 10:48:49.784927 -07:00 < x-sentry-rate-limit-remaining: 39
[10:48:49]: DEBUG   2024-05-28 10:48:49.784933 -07:00 < x-sentry-rate-limit-limit: 40
[10:48:49]: DEBUG   2024-05-28 10:48:49.784941 -07:00 < x-sentry-rate-limit-reset: 1716918530
[10:48:49]: DEBUG   2024-05-28 10:48:49.784948 -07:00 < x-sentry-rate-limit-concurrentremaining: 24
[10:48:49]: DEBUG   2024-05-28 10:48:49.784954 -07:00 < x-sentry-rate-limit-concurrentlimit: 25
[10:48:49]: DEBUG   2024-05-28 10:48:49.784961 -07:00 < content-language: en
[10:48:49]: DEBUG   2024-05-28 10:48:49.784967 -07:00 < x-frame-options: deny
[10:48:49]: DEBUG   2024-05-28 10:48:49.784975 -07:00 < x-content-type-options: nosniff
[10:48:49]: DEBUG   2024-05-28 10:48:49.784981 -07:00 < x-xss-protection: 1; mode=block
[10:48:49]: DEBUG   2024-05-28 10:48:49.784995 -07:00 < content-security-policy: script-src 'self' 'unsafe-inline' 'report-sample' s1.sentry-cdn.com js.sentry-cdn.com browser.sentry-cdn.com statuspage-production.s3.amazonaws.com static.zdassets.com aui-cdn.atlassian.com connect-cdn.atl-paas.net js.stripe.com 'strict-dynamic' cdn.pendo.io data.pendo.io pendo-io-static.storage.googleapis.com pendo-static-5634074999128064.storage.googleapis.com; connect-src 'self' *.algolia.net *.algolianet.com *.algolia.io sentry.io *.sentry.io s1.sentry-cdn.com o1.ingest.sentry.io api2.amplitude.com app.pendo.io data.pendo.io reload.getsentry.net t687h3m0nh65.statuspage.io sentry.zendesk.com ekr.zdassets.com maps.googleapis.com; worker-src blob:; frame-ancestors 'self' *.sentry.io; img-src * blob: data:; style-src * 'unsafe-inline'; base-uri 'none'; frame-src app.pendo.io demo.arcade.software js.stripe.com sentry.io; default-src 'none'; media-src *; object-src 'none'; font-src * data:; report-uri https://o1.ingest.sentry.io/ap
[10:48:49]: DEBUG   2024-05-28 10:48:49.785006 -07:00 < x-envoy-attempt-count: 1
[10:48:49]: DEBUG   2024-05-28 10:48:49.785012 -07:00 < x-envoy-upstream-service-time: 84
[10:48:49]: DEBUG   2024-05-28 10:48:49.785018 -07:00 < x-served-by: getsentry-web-rpc-production-5775747db-fw2hw
[10:48:49]: DEBUG   2024-05-28 10:48:49.785025 -07:00 < x-sentry-proxy-url: http://10.2.0.67:8999/api/0/organizations/dev-peek/chunk-upload/
[10:48:49]: DEBUG   2024-05-28 10:48:49.785032 -07:00 < x-served-by: getsentry-control-web-default-common-production-57c99dccbdwxrss
[10:48:49]: DEBUG   2024-05-28 10:48:49.785039 -07:00 < x-served-by: frontend-default-6ffdc95f4b-5kpw7
[10:48:49]: DEBUG   2024-05-28 10:48:49.785045 -07:00 < strict-transport-security: max-age=31536000; includeSubDomains; preload
[10:48:49]: DEBUG   2024-05-28 10:48:49.785052 -07:00 < via: 1.1 google
[10:48:49]: DEBUG   2024-05-28 10:48:49.785058 -07:00 < Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
[10:48:49]: DEBUG   2024-05-28 10:48:49.785065 -07:00 < Transfer-Encoding: chunked
[10:48:49]: DEBUG   2024-05-28 10:48:49.786121 -07:00 response status: 200
[10:48:49]: DEBUG   2024-05-28 10:48:49.786133 -07:00 body: {"url":"https://us.sentry.io/api/0/organizations/dev-peek/chunk-upload/","chunkSize":8388608,"chunksPerRequest":64,"maxFileSize":2147483648,"maxRequestSize":33554432,"concurrency":8,"hashAlgorithm":"sha1","compression":["gzip"],"accept":["debug_files","release_files","pdbs","sources","bcsymbolmaps","il2cpp","portablepdbs","artifact_bundles","artifact_bundles_v2"]}
[10:48:57]: error: Invalid checksum
[10:48:57]: DEBUG   2024-05-28 10:48:57.578688 -07:00 skipping update nagger because session is not attended
petrmanek commented 3 months ago

Just to update this: we are still seeing this on Windows, there has been no change since the issue was reported half a year ago.

szokeasaurusrex commented 3 months ago

@Mordil Please open a separate issue, since the problem you are describing seems to be different from the problem @petrmanek is experiencing.

@petrmanek only observed the problem on Windows and besides the error showing in the logs, everything in Sentry works properly.

On the other hand, from what you have written @Mordil, it sounds like you are noticing something wrong in Sentry, since you don't see releases and debug info in Sentry. You are also observing this problem on Mac, rather than Windows.

Please provide as much detail as possible when opening your issue; this will help me to reproduce your issue and to identify a fix/workaround.

Mordil commented 3 months ago

@szokeasaurusrex Apologies, we were looking at the wrong organization when were verifying if release symbols were being uploaded or not.

We've confirmed that the correct release information is being sent to the correct organization in Sentry, and our issue is only the same as Petr reported, simply on macOS instead of just Windows.

We're going to update our CI script to allow this subprocess failure, though we'd ideally like to see a fix for this.

szokeasaurusrex commented 3 months ago

Okay @Mordil, glad to hear that everything is working as expected in Sentry, but sorry to hear that this problem is affecting you as well.

Would you be able to share a debug symbol file that reproduces this error (or the source code and steps to recreate such a debug file)? If your project is open source, you could just share the code/debug files that you noticed the problems with. Otherwise, if you have been able to create a minimal reproduction that you can share with me, it would greatly help me with debugging this problem.

bjorn commented 2 months ago

Since recently I'm seeing this issue on my Linux builds, though I'm not sure if it's just the same error message or whether the entire behavior is the same. I have the impression the entire upload fails in my case.

An example build running into this issue can be seen at https://github.com/mapeditor/tiled/actions/runs/9592646514/job/26451582147 (see the "Upload symbols and sources to Sentry" step):

Sucessfully installed sentry-cli 2.32.1
Done!
  INFO    2024-06-20 06:32:47.682623978 +00:00 sentry-cli was invoked with the following command line: "sentry-cli" "upload-dif" "--log-level=info" "--include-sources" "src" "."
  INFO    2024-06-20 06:32:47.682699229 +00:00 Issuing a command for Organization: mapeditor Project: tiled
error: Invalid checksum

The last succesfull build is at https://github.com/mapeditor/tiled/actions/runs/9562961181/job/26360475510.

The problem actually started happening after updating from Sentry 0.6.7 to Sentry 0.7.6. In both cases sentry-cli 2.32.1 was used.

szokeasaurusrex commented 2 months ago

@bjorn, this might be the same problem.

The problem actually started happening after updating from Sentry 0.6.7 to Sentry 0.7.6

Which SDK version numbers are you referring to here?

bjorn commented 2 months ago

Which SDK version numbers are you referring to here?

Whoops, that would be the version of the Sentry Native SDK at https://github.com/getsentry/sentry-native.

By downgrading the sentry-native version I've now confirmed that this uploading issue does not occur on 0.6.7 but does happen since the 0.7.0 version (this version made crashpad the the default, but I'm passing -DSENTRY_BACKEND=breakpad).

szokeasaurusrex commented 2 months ago

Okay, thank you for that information @bjorn, this is very helpful! One more question to ensure your issue is the same as what @petrmanek and @Mordil reported: despite the error message, are the debug files still showing up in Sentry?

Mordil commented 2 months ago

For the record, with additional testing, debug symbols were NOT being sent to Sentry and we identified the issue with something from one of our dependencies: https://github.com/filestack/filestack-ios/issues/80

bjorn commented 2 months ago

@szokeasaurusrex Yeah when this error occurs for me I don't see the debug files showing up in Sentry. I guess I'm not running into the same issue as @Mordil, since for me the only difference is whether I use Sentry Native 0.6.7 or Sentry Native 0.7.x.

For now I think I've been able to resolve this issue by only uploading the .debug files and not all .o files (see https://github.com/mapeditor/tiled/pull/3987, which changes from passing repository root to passing the AppDir directory which contains only the installed files). I'm not sure if the .o files add any information.

szokeasaurusrex commented 2 months ago

@bjorn Are you still seeing full stack traces in Sentry after https://github.com/mapeditor/tiled/pull/3987? If yes, then probably you don't need the .o files you were uploading before

bjorn commented 2 months ago

Are you still seeing full stack traces in Sentry after mapeditor/tiled#3987?

I will only know for sure once somebody manages to crash the Tiled 1.11 AppImage, but that's only a matter of time unfortunately. :-)