AccessDeniedException: 403 We're sorry, but this service is not available in your location

[TriPSs]

We are getting the following error AccessDeniedException: 403 We're sorry, but this service is not available in your location when running any gsutil command on our production server in Germany.

How can we solve this?

[plantshark]

We got the same issue, one day to the other it just stopped working. Since this happened on the weekend I highly doubt that we accidently changed anything that caused this.

Command: /usr/bin/gsutil cp /home/user/file.gz gs://bucket-name or any other gsutil command

Output: ResumableUploadAbortException: 403 We're sorry, but this service is not available in your location

gsutil version: 5.17 boto version: 2.49.0 python version: 3.9.12 (main, Apr 30 2022, 03:04:12) [Clang 12.0.1 ] OS: Linux 3.10.0-1160.81.1.el7.x86_64

EDIT: Our bucket location is eu (multiple regions in European Union)

[TriPSs]

@plantshark after contacting Google support a while ago they said that our IPV6 was mis labeled to be in a location that was not supported, we simply updated our IP and the issue was resolved.

Maybe you also had a IP change over the weekend?

[plantshark]

Our IPv4/IPv6 adresses didn't change over the weekend, they are still located in germany. We have the same setup on multiple servers where we sync files in the google cloud. It worked for all 8 servers for the last few years, nothing changed over the weekend, no reboot happened and nobody changed configs. Therefore I can only suspect that google changed something.

I've tried reading data from a different google cloud project and other buckets. No gsutil commands from this specific server worked.

[TriPSs]

Weird, what provider are you on? As our server was also located in Germany.

[plantshark]

What's weird, I tried uninstalling or updating gsutil/gcloud on our server but when trying to fetch the package from the repo we get an [Errno 14] HTTPS Error 403 - Forbidden. Trying to download the package from another server from the same provider works as expected. Could it be that this server has been blocked by google somehow?

Our servers are hosted by hetzner.

[TriPSs]

Then I would give it a try by re-adding a new IPV6 as our German server had this issue and was also on Hetzner :)

[plantshark]

Sorry I don't have a broad knowledge regarding IPv6 adresses, can you please point me in the right direction where I can add a new IPv6 in the hetzner interface? Thanks for the great help by the way 😅

[TriPSs]

1. Login into Hetzner
2. Go to your server
3. Turn if off (Required to change the IPs)
4. Go to "Networking"
5. Remove the IPV6
6. I think you will get a new one, I can't try this atm as i'm not able to shut the server down 😅

Hopes this helps!

[plantshark]

For everyone else having this problem, we disabled our IPv6 adress since we don't need it anyways. We followed this guide using sysctl: https://www.thegeekdiary.com/centos-rhel-7-how-to-disable-ipv6/

After doing this it worked again.

[Preisschild]

For everyone else having this problem, we disabled our IPv6 adress since we don't need it anyways. We followed this guide using sysctl: thegeekdiary.com/centos-rhel-7-how-to-disable-ipv6

This might work in some cases, but not in all.

In your case, the IPv6 was blocked, but your IPv4 probably wasn't.

In my case, the IPv4 address which was assigned to my HCloud VM is blocked. Rotating the IPv4 address or adding an IPv6 address should fix this issue in all cases.

The real question is why GCP is blocking so many Hetzner IPs.

[tanji]

Disabling IPv6 is a a bit overkill. By default, the linux DNS system call getaddrinfo uses IPv6 over IPv4, but you can change precedence by adding this line in /etc/gai.conf:

precedence ::ffff:0:0/96  100

[kdrv3xc]

let's breathe some life in this topic: this is the error i'm getting: [primes@7072696d6375]:>rclone copy crDroidAndroid-14.0-20240306-salami-v10.2.zip gcbusket:7072696d6573/14.0-redux 2024/03/07 03:14:26 ERROR : Attempt 1/3 failed with 1 errors and: googleapi: Error 403: We're sorry, but this service is not available in your location, forbidden 2024/03/07 03:14:26 ERROR : Attempt 2/3 failed with 1 errors and: googleapi: Error 403: We're sorry, but this service is not available in your location, forbidden 2024/03/07 03:14:26 ERROR : Attempt 3/3 failed with 1 errors and: googleapi: Error 403: We're sorry, but this service is not available in your location, forbidden 2024/03/07 03:14:26 Failed to copy: googleapi: Error 403: We're sorry, but this service is not available in your location, forbidden and

[primes@7072696d6375]:>ip addr                                                                                                                                                         ~[3:21]
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP group default qlen 1000
    link/ether 02:00:17:00:31:df brd ff:ff:ff:ff:ff:ff
    altname enp0s3
    inet 10.0.101.121/24 metric 100 brd 10.0.101.255 scope global dynamic ens3
       valid_lft 55424sec preferred_lft 55424sec
    inet6 fe80::17ff:fe00:31df/64 scope link 
       valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:c6:7f:87:bf brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever

```and i woudn't think an link-local ipv6 would inrerfere. though correct if i'm mistaken. though it works fine,if using a different 
device, 

so penny for your thoughts?

[andy-pi]

I'm having the same issue, are there any better solutions that avoiding Hetzner?

[Preisschild]

Avoiding Google Services :)

They are the ones responsible for the block.

[rehashedsalt]

Can be awful hard sometimes when, for example, official Kubernetes build artifacts are hosted on storage.googleapis.com. Just got bit by this when trying to kops a cluster into existence.

[andy-pi]

Well I just deleted the server, got a new IP, and it works ... for now ...