hashicorp / terraform-provider-google

Terraform Provider for Google Cloud Platform
https://registry.terraform.io/providers/hashicorp/google/latest/docs
Mozilla Public License 2.0
2.33k stars 1.73k forks source link

GKE Auto pilot cluster keeps being recreated on each ur #13542

Open vojkny opened 1 year ago

vojkny commented 1 year ago

Community Note

Terraform Version

$ terraform -v
Terraform v1.3.7
on darwin_amd64
+ provider registry.terraform.io/hashicorp/google v4.49.0
+ provider registry.terraform.io/hashicorp/google-beta v4.49.0

Affected Resource(s)

Terraform Configuration Files

Debug Output

The full output is here:

https://gist.github.com/knyttl/f915d5c201342b38e113c104a7ca7db6

But the crucial part is:

          ~ oauth_scopes      = [ # forces replacement
              + "https://www.googleapis.com/auth/bigquery",
              + "https://www.googleapis.com/auth/devstorage.read_write",
              + "https://www.googleapis.com/auth/logging.write",
              + "https://www.googleapis.com/auth/monitoring",
              + "https://www.googleapis.com/auth/pubsub",
              - "https://www.googleapis.com/auth/userinfo.email",
                # (1 unchanged element hidden)
            ]

Expected Behavior

It seems the oauth_scopes are not set at all:

# gcloud container clusters describe aaa --region=europe-west1

nodeConfig:
  …
  oauthScopes:
  - https://www.googleapis.com/auth/userinfo.email
  - https://www.googleapis.com/auth/cloud-platform

So every terraform run tries to recreate fully the cluster.

References

edwardmedia commented 1 year ago

@knyttl GKE Auto pilot has special rules for access-scopes. Can you review its doc below? Does it make sense to you?

https://cloud.google.com/kubernetes-engine/docs/how-to/access-scopes

vojkny commented 1 year ago

I have read those and re-read them and these special rules shouldn't be an issue. All the roles I wanted to add should be allowed. Also, in case these were not allowed, it would be better to die than then trying to recreate the cluster infinitely.

edwardmedia commented 1 year ago

@knyttl where did you find below statement? Choosing Auto Pilot, there are many features pre-configured. Here is the comparison between Auto Pilot and Regular GKE. You may see its behavior of setup is different from the regular one.

All the roles I wanted to add should be allowed
vojkny commented 1 year ago

My case is this:

Screenshot 2023-02-07 at 18 40 57

And I see no reason why I would not be able to add:

             + "https://www.googleapis.com/auth/bigquery",
             + "https://www.googleapis.com/auth/devstorage.read_write",
             + "https://www.googleapis.com/auth/logging.write",
             + "https://www.googleapis.com/auth/monitoring",
             + "https://www.googleapis.com/auth/pubsub",

There is nothing mentioned this would be impossible?

edwardmedia commented 1 year ago

@melinath can you take a look at this?

jeperetz commented 1 year ago

I failed to repro this issue with the following config:

resource "google_container_cluster" "primary" {
  name     = "terraform-autopilot"
  location = "us-central1"

  enable_autopilot = true

  release_channel {
    channel = "RAPID"
  }

  cluster_autoscaling {
    auto_provisioning_defaults {
      oauth_scopes = [
        "https://www.googleapis.com/auth/bigquery",
        "https://www.googleapis.com/auth/devstorage.read_write",
        "https://www.googleapis.com/auth/logging.write",
        "https://www.googleapis.com/auth/monitoring",
        "https://www.googleapis.com/auth/pubsub",
        "https://www.googleapis.com/auth/userinfo.email",
      ]
    }
  }
}

@vojkny could you please re-share the config you used? Otherwise this issue may be obsolete.

vojkny commented 1 year ago

sorry, the configuration is here: https://gist.github.com/vojkny/de77dcab27bf1a5d7c935f27db260f2e

soorena776 commented 5 months ago

@vojkny can you please verify if this is still an issue?

vojkny commented 5 months ago

Hi we abandoned this topic.