search

GoogleCloudPlatform / buildpacks

Builders and buildpacks designed to run on Google Cloud's container platforms
Apache License 2.0
968 stars 144 forks source link

Deployments failing: Regions.Get and Zones.Get are already migrated to regional or zonal esf-proxy/Mixer #457

Open elyobo opened 18 hours ago

elyobo commented 18 hours ago

Describe the bug

Several of our previously working deployments failed to deploy overnight, all failing with errors like

ERROR: (gcloud.app.deploy) PERMISSION_DENIED: Request to https://compute.googleapis.com/compute/v1/projects/projects/***/regions/australia-southeast1?key failed, details: Regions.Get and Zones.Get are already migrated to regional or zonal esf-proxy/Mixer. See go/gce-location-apis-r13n. For more information please contact GCE API Frontend oncall https://oncall.corp.google.com/cloud-cluster-frontend-dev. This command is authenticated as some-account@***.[iam.gserviceaccount.com](http://iam.gserviceaccount.com/) using the credentials in /home/runner/work/pulse/pulse/gha-creds-0849da17eb00b746.json, specified by the [auth/credential_file_override] property.

Deployment is via Github Actions, with the gcloud CLI installed using google-github-actions/setup-gcloud@v2 and then the applications deployed e.g. via gcloud app deploy *.yaml.

A sample app.yaml looks like this, not much to it.

runtime: nodejs
env: flex

runtime_config:
  operating_system: 'ubuntu22'

automatic_scaling:
  min_num_instances: 1
  max_num_instances: 4
  target_concurrent_requests: 80
  cpu_utilization:
    target_utilization: 0.8

resources:
  cpu: 1
  memory_gb: 2
  disk_size_gb: 10

network:
  session_affinity: true

liveness_check:
  path: '/liveness-check'
  check_interval_sec: 60
  timeout_sec: 10
  failure_threshold: 2
  success_threshold: 2

readiness_check:
  path: '/readiness-check'
  timeout_sec: 1
  check_interval_sec: 2
  failure_threshold: 10
  success_threshold: 1
  app_start_timeout_sec: 600

build_env_variables:
  GOOGLE_NODE_RUN_SCRIPTS: ''

If you have a general question about how to use buildpacks, we encourage you to open a discussion topic instead

Additional context How are you using GCP buildpacks?

Did this used to work? (Yes/No) Yes. This started failing across multiple projects that have scheduled deployments overnight.

What language is your project primarily written in? (Python/Java/Node.js/Go/etc.)

Node.js

Steps To Reproduce

gcloud app deploy

Expected behavior

The application should deploy.

Actual behavior

An odd error, see issue description at top.

elyobo commented 18 hours ago

The most promising lead I've found on this this thread here; if the problem is that Google has added more new roles, how can we stay on top of these changes? This isn't the first time this "previously working deployment stopped working because the roles required were changed" type of issue has come up (e.g. see https://github.com/google-github-actions/setup-gcloud/issues/191 which also bit us), however there is e.g. nothing in the release notes here where I had been previously advised to check for breaking changes (not actually a viable way of being notified about breaking changes anyway...)

elyobo commented 18 hours ago

Another update - deployment jobs appear to have just started working again with no changes to roles. Confirming on other projects now.