Closed LowArmour closed 2 months ago
szokeasaurusrex
commented
2 months ago Hi @LowArmour, seems very strange. When did this problem start? Have you ever been able to run this command successfully on the server where you are currently getting the 403?
LowArmour
commented
2 months ago I did further digging, and found out the bug only applies to AlmaLinux 9 OS.
I have spun up, 3 new servers, one Fedora 40 (brand new), one AlmaLinux 9 (brand new), one CentOS Steam 9 (brand new).
Steps to reproduce:
BANG, you get your 403 forbidden error.
As you can conclude from above, the other servers (Fedora and CentOS Steam 9) get a clean status 200 OK.
I wanted to also try Rocky Linux. Let me know if you would like that.
AlmaLinux 9 is important because its one of the popular alternatives to CentOS, being a spiritual successor to the latter.
szokeasaurusrex
commented
2 months ago Hi @LowArmour, thank you for the additional details. Just to confirm, have you ever been able to successfully run this command with AlmaLinux 9, or has this always been broken? Do you get an error when running every Sentry CLI command that performs HTTP requests to Sentry, or are there any that succeed? Another command you could try is sentry-cli info.
This issue looks a bit to me like the Sentry server might be denying the API request for whatever reason. Would you be able to try manually sending a request to Sentry using curl from the AlmaLinux server? If that also fails, then that would indicate a problem in the server, rather than Sentry CLI.
LowArmour
commented
2 months ago I have never been able to run the command successfully on AlmaLinux 9.
As an update, I spun up another CentOS server, but with the same IP as the old AlmaLinux 9 and now I get the same error on CentOS. I only installed nvm and node.
I tried sentry-cli info and it also gives 403 forbidden. On the other test servers it works.
I tried manually sending a request to Sentry via curl. I get the same 403 forbidden.
What I suspect is that there might be some ip-related issue? Some of them work, some might be banned by sentry? I am getting the servers from Hetzner (Nuremberg, Germany and US East datacenters).
LowArmour
commented
2 months ago Update: it definitely seems IP related.
I have just changed the IP's on the servers that got 403 forbidden, until I got 200 OK.
Most probably sentry banned a lot of IP's owned by the Hetzner provider?
What is curious is that I kept just changing IP's and half of them work, half of them are getting 403 forbidden.
I don't think it is sentry-cli related, but can you redirect this issue to the appropriate github issues page?
szokeasaurusrex
commented
2 months ago @LowArmour Thanks for the information. In that case, I am going to transfer this to the getsentry/sentry repo, hopefully someone there can help or redirect to the appropriate place!
getsantry[bot]
commented
2 months ago Assigning to @getsentry/support for routing ⏲️
szokeasaurusrex
commented
2 months ago Hey @LowArmour, I discussed this issue with our security team, and according to them, the most likely explanation for why you are getting 403 errors is that your IP might be being blocked by Google (Sentry is hosted on GCP). They said this unfortunately is somewhat common with Hetzner IP addresses, since they often end up on IP address denylists.
If your IP address is being blocked by Google, there is not really anything we can do to help you here. We would suggest requesting a different IP address from Hetzner or switching to a different hosting provider if the problem persists.
If you would like to share your IP address (the one receiving 403 errors) with us, we can search our server logs to see if we have received any requests from the IP. If your IP shows up in our logs, then we might be able to make some changes to allow your IP; otherwise, if we don't see your IP, that would mean that Google is blocking it before your request reaches our infrastructure. If you are uncomfortable posting your IP publicly, you can email it to me at [redacted].
LowArmour
commented
2 months ago Hey, thank you for your answer. It is insightful and really helpful.
I have changed the IP of my production server until I got a good one, as I could not afford longer downtimes.
Thank you so much for your help!
IanWoodard
commented
2 months ago Glad you got this resolved, closing
CLI Version
2.32.2
Operating System and Architecture
Operating System Version
Fedora 38 (the server that gets 208), Almalinux 9 (the server that gets 403 forbidden)
Link to reproduction repository
No response
CLI Command
SENTRY_LOG_LEVEL=debug "/root/xxxxx/node_modules/@sentry/cli-linux-x64/bin/sentry-cli" "releases" "new" "60e801e2c6b94572b927a9048d45d8df" --project=xxxxx
Exact Reproduction Steps
The command that gives the error is: SENTRY_LOG_LEVEL=debug "/root/xxxxx/node_modules/@sentry/cli-linux-x64/bin/sentry-cli" "releases" "new" "60e801e2c6b94572b927a9048d45d8df" --project=xxxxx
This is actually a command from sentry-vite-plugin, which I extracted in order to isolate the error.
On one server, it gets 208 and succeeds, on the other (the production server) it gets 403 forbidden.
The .sentryclirc file is the same.
Expected Results
I expect both commands to work in both environments. This command is part of the build procedure under sentry-vite-plugin and because I get 403 forbidden on my production server, my production server is currently down :(.
Actual Results
error: API request failed caused by: sentry reported an error: unknown error (http status: 403)
Logs
BELOW THE SERVER THAT GIVES 403 FORBIDDEN:
BELOW THE SERVER THAT SUCCEEDS WITH STATUS 208