`google_container_cluster` fails with an API error on 6.11.0

SphericalKat commented 3 days ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
If you are interested in working on this issue or have submitted a pull request, please leave a comment.
If an issue is assigned to a user, that user is claiming responsibility for the issue.
Customers working with a Google Technical Account Manager or Customer Engineer can ask them to reach out internally to expedite investigation and resolution of this issue.

Terraform Version & Provider Version(s)

Terraform v1.4.4 on x86_64

provider registry.terraform.io/hashicorp/google v6.11.0
provider registry.terraform.io/hashicorp/google-beta v6.11.0

Affected Resource(s)

google_container_cluster

Terraform Configuration

resource "google_compute_network" "vpc" {
  name                    = "tokyo"
  description             = "tokyo"
  auto_create_subnetworks = false
}

resource "google_compute_subnetwork" "subnet" {
  name                     = "tokyo"
  ip_cidr_range            = "10.0.0.0/16"
  region                   = "asia-northeast1"
  private_ip_google_access = true
  network                  = google_compute_network.vpc.id
}

resource "google_container_cluster" "cluster" {
  name             = "gke-cluster"
  location         = "asia-northeast1"
  enable_autopilot = true

  network    = google_compute_network.vpc.id
  subnetwork = google_compute_subnetwork.subnet.id

  private_cluster_config {
    enable_private_endpoint = false
    enable_private_nodes    = true

    master_ipv4_cidr_block = "10.5.0.0/28"
  }

  ip_allocation_policy {
    cluster_ipv4_cidr_block  = "10.6.0.0/16"
    services_ipv4_cidr_block = "10.7.0.0/16"
    stack_type               = "IPV4"
  }

  master_authorized_networks_config {
    cidr_blocks {
      cidr_block   = "10.0.0.0/8"
      display_name = "private"
    }

    cidr_blocks {
      cidr_block   = "0.0.0.0/0"
      display_name = "all"
    }
  }

  binary_authorization {
    evaluation_mode = "DISABLED"
  }

  addons_config {
    horizontal_pod_autoscaling {
      disabled = false
    }
    http_load_balancing {
      disabled = false
    }
  }

  maintenance_policy {
    recurring_window {
      start_time = "2021-04-06T15:00:00Z"
      end_time   = "2021-04-06T21:00:00Z"
      recurrence = "FREQ=WEEKLY;BYDAY=MO,TU,WE,TH,FR,SA,SU"
    }
  }

  release_channel {
    channel = "REGULAR"
  }
}

Debug Output

No response

Expected Behavior

The apply should have passed, because there is no change to the resource.

Actual Behavior

The terraform apply fails with the following API error:

╷
│ Error: googleapi: Error 400: Must specify a field to update.
│ Details:
│ [
│   {
│     "@type": "type.googleapis.com/google.rpc.RequestInfo",
│     "requestId": "0x284c6237e333a9c"
│   }
│ ]
│ , badRequest
│ 
│   with google_container_cluster.cluster,
│   on kubernetes.tf line 1, in resource "google_container_cluster" "cluster":
│    1: resource "google_container_cluster" "cluster" {
│ 
╵

Note that the exact same configuration works when we pin the provider version to 6.10.0

Steps to reproduce

Use version 6.11.0 of the google provider
terraform apply

Important Factoids

No response

References

No response

b/378725361

rorynickolls-skyral commented 3 days ago

It looks like this may have been introduced by https://github.com/hashicorp/terraform-provider-google/pull/20105 in the latest release.

melinath commented 2 days ago

@SphericalKat can you share a plan? It would be helpful to see what fields Terraform thinks need to be updated, since there were a few different changes to this resource in this release.

melinath commented 2 days ago

I'm not able to reproduce this issue with the supplied config - if I apply on 6.10.0 and then upgrade to 6.11.0, no changes are detected. There must be some specific condition that triggers this error.

SphericalKat commented 2 days ago

@melinath sure, here's the planned changes. I've included the other resources marked for a change in case they were relevant too. (note that the plan went through successfully)

edit: I'd like to add that the new changes were applied directly with 6.11.0, and previous changes were applied with older versions because the provider wasn't pinned to a specific version using required_providers.

Initializing provider plugins...
- Finding latest version of hashicorp/google-beta...
- Finding latest version of hashicorp/random...
- Finding latest version of hashicorp/google...
- Installing hashicorp/google-beta v6.11.0...
- Installed hashicorp/google-beta v6.11.0 (signed by HashiCorp)
- Installing hashicorp/random v3.6.3...
- Installed hashicorp/random v3.6.3 (signed by HashiCorp)
- Installing hashicorp/google v6.11.0...
- Installed hashicorp/google v6.11.0 (signed by HashiCorp)

.
.
.

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # google_cloud_scheduler_job.batch-manuscripts will be updated in-place
  ~ resource "google_cloud_scheduler_job" "batch-manuscripts" {
        id               = "projects/<redacted>/locations/asia-northeast1/jobs/batch-manuscripts"
        name             = "batch-manuscripts"
        # (8 unchanged attributes hidden)

      ~ http_target {
          ~ headers     = {
              + "User-Agent" = "Google-Cloud-Scheduler"
            }
            # (2 unchanged attributes hidden)
        }
    }

  # google_cloud_scheduler_job.warmup will be updated in-place
  ~ resource "google_cloud_scheduler_job" "warmup" {
        id               = "projects/<redacted>/locations/asia-northeast1/jobs/api-warmup"
        name             = "api-warmup"
        # (8 unchanged attributes hidden)

      ~ http_target {
          ~ headers     = {
              + "User-Agent" = "Google-Cloud-Scheduler"
            }
            # (2 unchanged attributes hidden)
        }
    }

  # google_container_cluster.cluster will be updated in-place
  ~ resource "google_container_cluster" "cluster" {
        id                                       = "projects/<redacted>/locations/asia-northeast1/clusters/gke-cluster"
        name                                     = "gke-cluster"
        # (31 unchanged attributes hidden)

      - user_managed_keys_config {
          - service_account_signing_keys      = [] -> null
          - service_account_verification_keys = [] -> null
        }

        # (34 unchanged blocks hidden)
    }

Plan: 0 to add, 3 to change, 0 to destroy.

ferrastas commented 2 days ago

I'm also facing the same issue, I do not have the raw text as we use TFE, but here's a screenshot of the changes

josephweinerfulfilld commented 2 days ago

Facing exactly the same issue as @ferrastas and @melinath . This is a blocking issue for us when using 6.11.0. We additionally were not pinning to using an exact version. Looking to workaround by using a previous version.

melinath commented 2 days ago

@ferrastas @josephweinerfulfilld @SphericalKat is there any chance y'all had previously been managing the user managed encryption keys feature outside of Terraform? (It doesn't look like it, since the values are all empty, but possibly setting it up and removing it later would change what the API returns vs. never setting it in the first place.)

melinath commented 2 days ago

I'm able to reproduce the error when trying to update the fields in place, but I'm not able to reproduce the behavior of having a diff on the field despite making no changes.

There's a note in the related tests that these fields can only be set on create, but there is also update support present, so I'm not sure if that note might be outdated. ~@ahmedtd FYI as the PR author in case you can provide insight.~ (Update: confirmed that these fields are not currently updatable in place.)

We will need to fix the update issue by marking the field as immutable (ForceNew) but that would trigger delete & recreate for anyone experiencing the diff behavior.

ryanfulfilld commented 2 days ago

@melinath (replying for @josephweinerfulfilld): We have not been using the user managed encryption keys outside of Terraform. We ran into this error in a newly set-up environment, so I'm pretty confident that there shouldn't have been any manual configuration relics.

ferrastas commented 2 days ago

@melinath not possible, we faced this issue on an almost fresh new project. If it helps, it was created using provider version 6.10, this same plan was working yesterday, just before the 6.11 release. In fact, pinning to 6.10 the plan is not showing changes on the cluster resource (and succeeding with the apply)

ahmedtd commented 2 days ago

The current behavior of these fields is:

For each field, you have to set it or not during cluster create.
If the field is set at cluster create, you can never update the field to be unset (though you can update the value).
If the field is unset at cluster create, you can never set it.

In the future, we are going to support setting / unsetting these fields, but it will never be truly freely settable, since there needs to be a CA root rotation operation in order to transition from set to unset or vice-versa.

melinath commented 2 days ago

@ferrastas @josephweinerfulfilld @SphericalKat @ryanfulfilld thanks! Is there any chance that one of you could share the API response you're getting for the resource's state refresh? You can get these logged by running TF_LOG=DEBUG terraform apply. Please feel free to redact anything except the user managed keys portion (if present).

melinath commented 2 days ago

I have a tentative fix up at:

google: https://github.com/hashicorp/terraform-provider-google/tree/release-6.11.1 google-beta: https://github.com/hashicorp/terraform-provider-google-beta/tree/release-6.11.1

If someone could try some manual testing to see if this resolves the issue that would be great, since I can't reproduce locally. The API responses would also be super helpful. Thanks!

melinath commented 2 days ago

I'm working on an update to the branches - per @ahmedtd's comment above, it should be possible to update in place as long as you're not enabling/disabling the functionality (which folks on this thread aren't).

melinath commented 2 days ago

Looks like the update functionality doesn't currently work as expected so moving forward with removing update support & marking the field as immutable for now.

melinath commented 2 days ago

Hey all - we've released 6.11.1, which should resolve this issue; however, we still haven't been able to reproduce it. Please try out the new release and respond here to let me know whether it fixed the problem. If not, please share an API response as described above. If anyone impacted is working with a Google Technical Account Manager or Customer Engineer, please ask them to reach out internally to help us with further investigation.

joemiller commented 2 days ago

@melinath I ran a plan on one of the clusters exhibiting this behavior and the plan for 6.11.1 was "no changes" whereas on 6.11.0 it wanted to update the user_managed_keys_config field. Seems good.

SphericalKat commented 2 days ago

@melinath thanks for the quick fix and sorry about my slow response, it was nighttime here :P

I'll check out 6.11.1 and see if the problem still occurs

Edit: It works! Thanks, everyone who contributed!

hashicorp / terraform-provider-google