GKE Auto pilot cluster keeps being recreated on each ur

[vojkny]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

$ terraform -v
Terraform v1.3.7
on darwin_amd64
+ provider registry.terraform.io/hashicorp/google v4.49.0
+ provider registry.terraform.io/hashicorp/google-beta v4.49.0

Affected Resource(s)

• google_container_cluster

Terraform Configuration Files

• https://gist.github.com/knyttl/de77dcab27bf1a5d7c935f27db260f2e

Debug Output

The full output is here:

https://gist.github.com/knyttl/f915d5c201342b38e113c104a7ca7db6

But the crucial part is:

          ~ oauth_scopes      = [ # forces replacement
              + "https://www.googleapis.com/auth/bigquery",
              + "https://www.googleapis.com/auth/devstorage.read_write",
              + "https://www.googleapis.com/auth/logging.write",
              + "https://www.googleapis.com/auth/monitoring",
              + "https://www.googleapis.com/auth/pubsub",
              - "https://www.googleapis.com/auth/userinfo.email",
                # (1 unchanged element hidden)
            ]

Expected Behavior

It seems the oauth_scopes are not set at all:

# gcloud container clusters describe aaa --region=europe-west1

nodeConfig:
  …
  oauthScopes:
  - https://www.googleapis.com/auth/userinfo.email
  - https://www.googleapis.com/auth/cloud-platform

So every terraform run tries to recreate fully the cluster.

References

• b/298500574

[edwardmedia]

@knyttl GKE Auto pilot has special rules for access-scopes. Can you review its doc below? Does it make sense to you?

https://cloud.google.com/kubernetes-engine/docs/how-to/access-scopes

[vojkny]

I have read those and re-read them and these special rules shouldn't be an issue. All the roles I wanted to add should be allowed. Also, in case these were not allowed, it would be better to die than then trying to recreate the cluster infinitely.

[edwardmedia]

@knyttl where did you find below statement? Choosing Auto Pilot, there are many features pre-configured. Here is the comparison between Auto Pilot and Regular GKE. You may see its behavior of setup is different from the regular one.

All the roles I wanted to add should be allowed

[vojkny]

My case is this:

And I see no reason why I would not be able to add:

             + "https://www.googleapis.com/auth/bigquery",
             + "https://www.googleapis.com/auth/devstorage.read_write",
             + "https://www.googleapis.com/auth/logging.write",
             + "https://www.googleapis.com/auth/monitoring",
             + "https://www.googleapis.com/auth/pubsub",

There is nothing mentioned this would be impossible?

[edwardmedia]

@melinath can you take a look at this?

[jeperetz]

I failed to repro this issue with the following config:

resource "google_container_cluster" "primary" {
  name     = "terraform-autopilot"
  location = "us-central1"

  enable_autopilot = true

  release_channel {
    channel = "RAPID"
  }

  cluster_autoscaling {
    auto_provisioning_defaults {
      oauth_scopes = [
        "https://www.googleapis.com/auth/bigquery",
        "https://www.googleapis.com/auth/devstorage.read_write",
        "https://www.googleapis.com/auth/logging.write",
        "https://www.googleapis.com/auth/monitoring",
        "https://www.googleapis.com/auth/pubsub",
        "https://www.googleapis.com/auth/userinfo.email",
      ]
    }
  }
}

@vojkny could you please re-share the config you used? Otherwise this issue may be obsolete.

[vojkny]

sorry, the configuration is here: https://gist.github.com/vojkny/de77dcab27bf1a5d7c935f27db260f2e

[soorena776]

@vojkny can you please verify if this is still an issue?

[vojkny]

Hi we abandoned this topic.