Open Snack-X opened 7 months ago
@Snack-X thank you for the very detailed issue report!
It appears that we did make some changes to the compiler optimizations from version 2.21.2 to 2.21.3. However, we reverted one of the changes in 2.23.0 (commit 59fcf92) because it was causing problems when Sentry CLI was run in Xcode Cloud.
Have you tried running sentry-cli
≥2.23.0 in rootless Docker yet? If not, please try again with sentry-cli
≥2.23.0, since it is possible that the fix from 59fcf92 will fix your problem, too. Otherwise, if you still have trouble with sentry-cli
≥2.23.0, please let me know, and I will investigate further.
Thanks for the response.
That version was found with bisection from 2.0.0 to 2.28.0, so I think I've already tried >= 2.23.0. But I'll check it again just in case.
Okay, please let me know once you have tried ≥2.23.0 again and have confirmed whether the issue is still occurring in those versions
I can confirm both 2.23.0 and latest (2.28.0) still have same issue.
# curl -sL https://sentry.io/get-cli/ | SENTRY_CLI_VERSION=2.23.0 bash
Installed sentry-cli 2.23.0
Done!
# sentry-cli --version
sentry-cli 2.23.0
# sentry-cli info
Sentry Server: https://sentry.io
Default Organization: -
Default Project: -
Authentication Info:
Method: Unauthorized
(failure on authentication: API request failed)
# curl -sL https://sentry.io/get-cli/ | SENTRY_CLI_VERSION=2.28.0 bash
Installed sentry-cli 2.28.0
Done!
# sentry-cli --version
sentry-cli 2.28.0
# sentry-cli info
Sentry Server: https://sentry.io
Default Organization: -
Default Project: -
Authentication Info:
Method: Unauthorized
(failure on authentication: API request failed)
I've also found these issues which be related to this one, since sentry-cli is built with musl.
Looks like musl has problem with DNS resolving.
Thank you for the information and for linking those issues! I will investigate further to see whether we can fix this somehow, or whether this is something that needs to be fixed in musl
Thanks @Snack-X for debugging and linking to issue https://github.com/getsentry/sentry-cli/issues/1843, I can confirm my issue is the same :) The error message for me is thrown in a Gitlab pipeline, using the docker:dind-rootless image to build my docker image. I have pinned my sentry-cli version to 2.21.2
currently.
@Snack-X, I tried to reproduce your bug using a clean install of Ubuntu 22.04 LTS running the latest Docker, installed in rootless mode (following the Docker Docs instructions). However, everything appears to be working correctly on my end with sentry-cli
version 2.28.0
. Below are the specific commands I ran, along with the corresponding outputs:
root@3661a4367ba7:/# sentry-cli --version
sentry-cli 2.28.0
root@3661a4367ba7:/# sentry-cli info
Sentry Server: https://sentry.io
Default Organization: -
Default Project: -
Authentication Info:
Method: Unauthorized
root@3661a4367ba7:/# sentry-cli info --auth-token=sntrys_[redacted]
Sentry Server: https://sentry.io
Default Organization: -
Default Project: -
Authentication Info:
Method: Auth Token
Scopes:
- org:ci
Perhaps, you could try passing an auth token with --auth-token
to see if you observe similar output to what I observed in the last command I ran? If you still are getting an error, when passing the --auth-token
, would you please be able to provide me with more information about your specific setup, so I can try to reproduce the issue?
In my situation, I'm building a docker image in a Gitlab CI pipeline.
Moving from
docker:25.0.3-dind-rootless@sha256:a67a4f149cd62d8c2023778ddd65e82e3895b21c10b8a5a19fd18b3dd1c3c96a
to
docker:25.0.3-dind@sha256:915cd1624f521b6337f135075f712c8fb14c0b151595c6144d7ce05d2f257869
resolves my issue in https://github.com/getsentry/sentry-cli/issues/1843.
@szokeasaurusrex the docker:25.0.3-dind
image has alpine
as base and not ubuntu
. Maybe thats the difference why you cannot reproduce with Ubuntu 22.04 LTS
?
Hi @darthf1, I tried to run the docker:25.0.3-dind-rootless
image, but was getting errors on my system. I was, however, able to test against the alpine
Docker image, but everything appeared to work as expected (I observed the same results as in my previous comment).
the docker:25.0.3-dind image has alpine as base and not ubuntu. Maybe thats the difference why you cannot reproduce with Ubuntu 22.04 LTS?
Ubuntu 22.04 LTS is the operating system of the machine where I have installed the Docker daemon, not the operating system of the Docker containers. So, I don't think this explains the difference in behavior I observed compared with what you and @Snack-X are observing.
Unfortunately, I will only be able to help with resolving this issue if I am able to reproduce your problem, and in order to do that, I will need a detailed list of the specific commands you ran to set up a Docker container where you observe this bug, and a list of commands to run in the Docker container to observe the result.
I'm affected by this issue (or a similar one), intermittently failing to resolve sentry.io while running Docker builds in a particular hosted environment:
[my-app] my-app:build: Error: [sentry-debug-id-upload-plugin] Command failed: /usr/src/app/node_modules/.pnpm/@sentry+cli-linux-arm64@2.30.2/node_modules/@sentry/cli-linux-arm64/bin/sentry-cli releases finalize dab5a9dd58166b08213e432181608e7e
[my-app] my-app:build: error: API request failed
[my-app] my-app:build: caused by: [6] Couldn't resolve host name (Could not resolve host: sentry.io)
I'm actually using @sentry/vite-plugin
, but the problem goes away when I pin the underlying @sentry/cli
version like so:
"pnpm": {
"overrides": {
"@sentry/cli": "2.21.2"
}
}
The environment is BalenaCloud's build server.
One question, as I try to create a repro: is there a good way to exercise @sentry/cli
's DNS access, without supplying account credentials? It'd be really helpful if I could run something like sentry-cli --test-api-connection
in a public repo without worrying about a token.
Update - I've now seen it happen with 2.21.2 as well:
[my-app] my-app:build: Error: [sentry-debug-id-upload-plugin] Command failed: /usr/src/app/node_modules/.pnpm/@sentry+cli@2.21.2/node_modules/@sentry/cli/sentry-cli releases finalize 7a200f797cbcd93e356691aaaf1353fc
[my-app] my-app:build: error: API request failed
[my-app] my-app:build: caused by: [6] Couldn't resolve host name (Could not resolve host: sentry.io)
@jrr You can run sentry-cli update
. This command pings the Sentry server to determine the latest sentry-cli
release, but it does not require authentication.
Please let me know if you have a reproduction; as I wrote in my previous comment, I was struggling to reproduce this issue, and will only be able to fix it once I can reproduce the problem.
I switched upstream images and the problem seems to have gone away. In summary:
Using FROM balenalib/raspberrypi4-64-node:18-bookworm-run
, I'd get intermittent Could not resolve host: sentry.io
.
I haven't seen it since switching to FROM node:20-bookworm-slim
.
I had never seen network problems with other services, and I never saw it when running the docker build locally. So I think there's something particular about the combination of 1) upstream image, 2) the hosted docker environment, and 3) sentry-cli's network access.
Since it seems to be resolved for us, I'm going to let this go. If you have a similar problem, try experimenting with your upstream image/layers.
Good to hear, @jrr! I am going to leave this issue open for now, however, since I am still waiting for more information from @Snack-X on how to reproduce the original issue.
Perhaps, you could try passing an auth token with --auth-token
# curl -sL https://sentry.io/get-cli/ | SENTRY_CLI_VERSION=2.21.2 bash
(snipped)
Installed sentry-cli 2.21.2
Done!
# sentry-cli info --auth-token=(redacted)
Sentry Server: https://sentry.io
Default Organization: -
Default Project: -
Authentication Info:
Method: Auth Token
User: (redacted)
(snipped)
# curl -sL https://sentry.io/get-cli/ | SENTRY_CLI_VERSION=2.21.3 bash
(snipped)
Installed sentry-cli 2.21.3
Done!
# sentry-cli info --auth-token=(redacted)
Sentry Server: https://sentry.io
Default Organization: -
Default Project: -
Authentication Info:
Method: Unauthorized
(failure on authentication: API request failed)
Finally I've got around to create reproducible environment under AWS with Terraform.
My original assumption was it has something to do with IPv6. That is the reason why Terraform configuration above contains two separate VPC for both IPv4 only and dualstack VPC. It does not matter as it turns out.
Hope this helps.
Thank you @Snack-X, I will look into this as soon as possible!
This issue has gone three weeks without activity. In another week, I will close it.
But! If you comment or otherwise update it, I will reset the clock, and if you label it Status: Backlog
or Status: In Progress
, I will leave it alone ... forever!
"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀
Don't close it, bot! This is still a thing.
@szokeasaurusrex any progress with @Snack-X 's repro?
For my project, I thought they were gone, but I'm still getting intermittent network failures during my docker builds. I have the following in my Dockerfile:
RUN nslookup sentry.io
RUN pnpx @sentry/cli update
RUN pnpm turbo --filter my-app build
nslookup always works, but the other two may fail. Update's failure looks like this:
[my-app] error: Could not get the latest release version.
the build failure looks like this:
my-app:build: Error: [sentry-debug-id-upload-plugin] Command failed: /usr/src/app/node_modules/.pnpm/@sentry+cli-linux-arm64@2.31.1/node_modules/@sentry/cli-linux-arm64/bin/sentry-cli releases new c7a298d8a186788b044c3272e4bca8bb
my-app:build: error: API request failed
my-app:build: caused by: [6] Couldn't resolve host name (Could not resolve host: sentry.io)
my-app:build:
my-app:build: Add --log-level=[info|debug] or export SENTRY_LOG_LEVEL=[info|debug] to see more output.
my-app:build: Please attach the full debug log to all bug reports.
my-app:build:
my-app:build: at genericNodeError (node:internal/errors:984:15)
my-app:build: at wrappedFn (node:internal/errors:538:14)
my-app:build: at ChildProcess.exithandler (node:child_process:422:12)
my-app:build: at ChildProcess.emit (node:events:519:28)
my-app:build: at maybeClose (node:internal/child_process:1105:16)
my-app:build: at Socket.<anonymous> (node:internal/child_process:457:11)
my-app:build: at Socket.emit (node:events:519:28)
my-app:build: at Pipe.<anonymous> (node:net:338:12) {
my-app:build: code: 'PLUGIN_ERROR',
my-app:build: killed: false,
my-app:build: signal: null,
my-app:build: cmd: '/usr/src/app/node_modules/.pnpm/@sentry+cli-linux-arm64@2.31.1/node_modules/@sentry/cli-linux-arm64/bin/sentry-cli releases new c7a298d8a186788b044c3272e4bca8bb',
my-app:build: pluginCode: 1,
my-app:build: plugin: 'sentry-debug-id-upload-plugin',
my-app:build: hook: 'writeBundle'
my-app:build: }
I've added SENTRY_LOG_LEVEL=debug and will report back if that reveals anything interesting from future failures.
@jrr I have been busy with other tasks and so I still have yet to be able to try @Snack-X's reproduction.
However, the issue you are experiencing is likely separate. I can see you are running RUN pnpx @sentry/cli update
. However, since you have installed sentry-cli
via a package manager, you should only update sentry-cli
via the package manager, and not be running the RUN pnpx @sentry/cli update
command. The update
command is only intended for users who manually download and install the CLI, e.g. via the curl -sL https://sentry.io/get-cli/ | bash
script.
@szokeasaurusrex I don't actually need to update it. That line exists solely for testing connectivity, as you advised earlier in the thread: https://github.com/getsentry/sentry-cli/issues/1929#issuecomment-2014642070
I'm hoping it fails for the same reason as the vite plugin, and may shed more light on what's goin on.
I have a possibly-interesting log from a failure, with SENTRY_LOG_LEVEL=debug
on:
[foo-app] Step 35/52 : RUN nslookup sentry.io
[foo-app] ---> Running in 0606272e1dfc
[foo-app] Server: 1.1.1.1
[foo-app] Address: 1.1.1.1#53
[foo-app] Non-authoritative answer:
[foo-app] Name: sentry.io
[foo-app] Address: 35.186.247.156
[foo-app] Removing intermediate container 0606272e1dfc
[foo-app] ---> f205de02a1d9
Looks like this host is actually release-registry.services.sentry.io
With vite-plugin/2.16.1 and sentry-cli/2.31.1
I'm not fully understanding what's going on here. I see evidence of two requests (GET /chunk-upload/
and PUT /releases/
). Is the second of those succeeding? ("HTTP/1.1 200 OK" in the log).
I also noticed "max retries: 0" in there. Can the vite plugin or CLI be configured to allow retries? (or perhaps some env configuration for the underlying HTTP client?)
And here's one where @sentry/cli update fails, though it doesn't log much detail:
@jrr Taking a look, it seems like max retries
is always hardcoded to zero, so I am not sure why this option even exists.
Is the CLI always failing in the same way for you? Based on the error messages (e.g. Couldn't resolve host name (Could not resolve host: sentry.io)
), this behavior seems like it could be most easily explained by some kind of (possibly transient) network issue with your setup.
Digging into some of the related issues and their links, it sounds like there are some significant issues with musl's DNS resolver. In particular, it uses DNS over UDP, not TCP (?!) (comment, link)
As a background refresher, TCP has reliability and retries built-in as part of the protocol. UDP does not. So when you use UDP, it's on you to implement reliability/retries at a higher layer. And it seems like neither musl nor sentry-cli is doing this.
In my opinion, long-term, sentry-cli ought to use a network stack with DNS over TCP. In the short term, can you put some retries around it? (@szokeasaurusrex)
Hi @jrr, thank you for providing this information, as well as the comment and the link – this information is very helpful!
The only thing I wonder is, assuming that the underlying problem is with musl, what could have changed between 2.21.2
and 2.21.3
that caused this issue to appear?
Here's another reference with some more details on musl's DNS behavior: https://wiki.musl-libc.org/functional-differences-from-glibc.html#Name_Resolver/DNS
So it looks like there's more than just the UDP thing: there are some other behavior differences between musl and others that may come into play.
Also I see that DNS over TCP was added to musl recently. @szokeasaurusrex do you know how to track down what version of musl goes into a given sentry-cli version?
Also I see that DNS over TCP was added to musl recently. @szokeasaurusrex do you know how to track down what version of musl goes into a given sentry-cli version?
@jrr I am unsure; a codebase-wide search for "musl" does not yield any results containing a version number
Other than f226a6e4506699b1bcd624e09c1507844003b515, I don't see any other commits related to musl. Although it doesn't make sense to me, but I'm not an expert in this area.
Previous docker image, getsentry/rust-musl-cross (repo) appears to be built in 2023-04-something with latest musl version, which would be 1.2.3
(released 2022-04-07).
2.21.3
was released in 2023-11-09, and I think implies it was built with latest build of new docker image, messense/rust-musl-cross (repo), at that time.
Although latest musl version at that time is 1.2.4
(released in 2023-05-01) which could have caused this issue, that build also appears to use 1.2.3, what?
I have no idea.
Good finds, @Snack-X . Summarizing my understanding:
getsentry/rust-musl-cross
Docker image for this (docker hub, git repo)messense/rust-musl-cross
docker hub and git repo)messense/rust-musl-cross
is on musl 1.2.3 from 2022 (ref)A couple ideas for how Sentry can get get musl's DNS improvements into sentry-cli binaries:
Yes, thank you both for this information! I wonder whether simply reverting https://github.com/getsentry/sentry-cli/commit/f226a6e4506699b1bcd624e09c1507844003b515 could solve this problem – perhaps we could try this before we implement one of the other solutions you proposed, @jrr, so that we can verify that this commit was indeed the cause?
I am unaware of the reasoning behind the change in https://github.com/getsentry/sentry-cli/commit/f226a6e4506699b1bcd624e09c1507844003b515 though, so I will first confirm with @loewenheim (who authored the change) whether there is any reason we should not revert that commit.
I think pushing for a musl update in the upstream image (PR by @jrr here) is a good idea.
Thank you all for your work in investigating this issue!
I have been experiencing the exact same symptoms when running GitLab CI jobs within a docker container (based on debian slim) that runs sentry-cli via the npm package https://www.npmjs.com/package/@sentry/cli
Would we be able to experience the problem described by jrr in https://github.com/getsentry/sentry-cli/issues/1929#issuecomment-2161124224 under these circumstances?
I had spent a lot of time looking into docker DNS resolution, the DNS set up on the build runner server, EDNS etc.etc. but is the fundamental issue how sentry-cli binary itself is performing the DNS resolution?
Happy to test a solution and report back if so 🫡
@joekeilty-oub From my understanding of the situation, it seems like it could be possible that the cause of your problem is the same since you are running the CLI inside a Linux Docker image. Let's see whether we are able to get this PR merged and whether that change ends up solving your issue
@szokeasaurusrex Is this issue related to the error thrown when using sentry-cli in a Docker container? I've got it installed as a dependency as part of @sentry/nextjs
but because it's a macOS binary trying to run on Alpine Linux, it fails. Is there a solution to this if it's not related?
0.775 Creating an optimized production build ...
10.20 Failed to compile.
10.20
10.20 Sentry CLI Plugin: Command failed: /app/node_modules/@sentry/cli/sentry-cli releases new IALP9dRCjJokMal0t774p
10.20 Cannot run macOS (Mach-O) executable in isolated machine: Exec format error
10.20
10.20 Sentry CLI Plugin: Command failed: /app/node_modules/@sentry/cli/sentry-cli releases new IALP9dRCjJokMal0t774p
10.20 Cannot run macOS (Mach-O) executable in isolated machine: Exec format error
10.20
10.20
10.20 > Build failed because of webpack errors
------
Dockerfile:15
--------------------
13 | COPY . .
14 | # Build your Next.js app
15 | >>> RUN npm run build
16 |
17 | # Stage 2: Running the app
--------------------
ERROR: failed to solve: process "/bin/sh -c npm run build" did not complete successfully: exit code: 1
@mzavattaro as far as I can tell, this looks like a separate issue. I don't recognize any of the error messages, and to me it looks like quite possibly the problem is occurring before Sentry CLI even gets called. Although, it's hard for me to tell since I'm not familiar with your particular setup.
Environment
sentry-cli >= 2.21.3 + rootless docker
Steps to Reproduce
python:3.12
) and start new container (docker run --rm -it python:3.12 bash
)curl -sL https://sentry.io/get-cli/ | SENTRY_CLI_VERSION=2.21.3 bash
)sentry-cli info
Expected Result
No error
Actual Result
(failure on authentication: API request failed)
Logs
While updating our nuxt web app's sentry module(
@nuxtjs/sentry
), we've also updated@sentry/webpack-plugin
from1.x.x
to2.x.x
.After this change, we started to see our CI/CD pipeline failing to publish releases and sourcemaps. The only error message we could see was
error: API request failed
caused by: [6] Couldn't resolve host name (Could not resolve host: sentry.io)
.Few hours of tests and bisection narrowed down to combination of rootless docker (used in our Actions runner) and sentry-cli >= 2.21.3. Standard docker installation works just fine, and older version of sentry-cli also has no problem -- which is the fix we chose.
We tested multiple configurations of OS and architectures, which all had same result:
I think some low level change made between 2.21.2 and 2.21.3 affected how DNS resolution works and created some kind of incompatibility.
It could be an issue of docker, it could be both, I don't know at this point. I hope this information is helpful.