insecure_kubelet_readonly_port_enabled does not work on the cluster level via google_container_cluster

[twingate-blee]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to a user, that user is claiming responsibility for the issue.
• Customers working with a Google Technical Account Manager or Customer Engineer can ask them to reach out internally to expedite investigation and resolution of this issue.

Terraform Version & Provider Version(s)

Terraform v1.9.2 on darwin_arm64

• provider registry.terraform.io/carlpett/sops v1.1.1
• provider registry.terraform.io/hashicorp/google v6.3.0
• provider registry.terraform.io/hashicorp/google-beta v6.4.0
• provider registry.terraform.io/hashicorp/null v3.2.3
• provider registry.terraform.io/hashicorp/random v3.6.3
• provider registry.terraform.io/hashicorp/tls v4.0.6
• provider registry.terraform.io/mrolla/circleci v0.6.1
• provider registry.terraform.io/opsgenie/opsgenie v0.6.37
• provider registry.terraform.io/twingate/twingate v3.0.11
• provider registry.terraform.io/vancluever/acme v2.26.0

Affected Resource(s)

google_container_cluster

Terraform Configuration

resource "google_container_cluster" "app_cluster" {
  ...
  remove_default_node_pool    = true
  ...
  node_config {
    kubelet_config {
      cpu_manager_policy                     = "none"
      insecure_kubelet_readonly_port_enabled = "FALSE"
    }
  }
  ...

Debug Output

No response

Expected Behavior

Terraform should not error and insecureKubeletReadonlyPortEnabled should be set to False

gcloud container clusters describe <name> \
    --location=<location> \
    --flatten=nodePoolDefaults.nodeConfigDefaults \
    --format="value(nodeKubeletConfig)"
insecureKubeletReadonlyPortEnabled=False

Actual Behavior

│ Error: googleapi: Error 400: Node pool "default-pool" not found on update.
│ Details:
│ [
│   {
│     "@type": "type.googleapis.com/google.rpc.RequestInfo",
│     "requestId": "0x46b14d914594c365"
│   }
│ ]
│ , badRequest

We do not use the "default-pool". Node pools created using google_container_node_pool

Steps to reproduce

1. terraform apply

Important Factoids

No response

References

No response

[wyardley]

Not a Google employee, but I did add this feature. The setting can be set in a bunch of places. While I agree this behavior is confusing and not ideal, see the notes in the provider docs about node_config.

Generally, this field should not be used at the same time as a google_container_node_pool or a node_pool block; this configuration manages the default node pool, which isn't recommended to be used with Terraform

Also, looking at your example, you're looking for nodeConfigDefaults, which is in the node_pool_defaults block, not the node_config.kubelet_config block.

So, if you're using remove_default_node_pool = true, you will need to set it for each pool separately. This may not line up with your exact use case, but basically:

resource "google_container_cluster" "app_cluster" {
  ...
  remove_default_node_pool = true

  node_pool_defaults {
    node_config_defaults {
      insecure_kubelet_readonly_port_enabled = "FALSE"
    }
  }
}

resource "google_container_node_pool" "your_node_pool" {
  name = "your-node-pool"
  cluster    = google_container_cluster.app_cluster.id
  node_count = 1

  node_config {
    kubelet_config {
      insecure_kubelet_readonly_port_enabled = "FALSE"
    }
  }
}

HTH

[wyardley]

All that said, it might be a good idea (if it's possible) for the provider team to add some kind of provider level validation / erroring if someone tries to set node_config when remove_default_node_pool is set, but I suspect that removing it should resolve the issue.

[twingate-blee]

Thank you. I was trying to set the insecure_kubelet_readonly_port_enabled at the cluster level as suggested in: https://cloud.google.com/kubernetes-engine/docs/how-to/disable-kubelet-readonly-port "We recommend that you set the read-only port setting at the cluster level in all cases."

I will remove any kubelet_config from the cluster level.

[wyardley]

I think they’re suggesting setting it on any node pools, as well as in the defaults. But there are a lot of different places it can be set.

If you use the CLI to check the values as in their howto, and they all have the right setting, you should be good.