Open billyfoss opened 1 year ago
Note: when trying to apply the plan
- linux_node_config {
- sysctls = {} -> null
}
I get
Error: googleapi: Error 400: At least one of ['node_version', 'image_type', 'updated_node_pool', 'locations', 'workload_metadata_config', 'upgrade_settings', 'kubelet_config', 'linux_node_config', 'tags', 'taints', 'labels', 'node_network_config', 'gcfs_config', 'gvnic', 'confidential_nodes', 'logging_config', 'fast_socket', 'resource_labels] must be specified., badRequest
Had same problem Managed to workaround it for now by adding:
linux_node_config {
sysctls = {}
}
to the node_config
block
I had to remove:
linux_node_config {
sysctls = {}
}
in order to be able to deploy a new nodepool.
otherwise:
panic: interface conversion: interface {} is nil, not map[string]interface {}
Hi @billyfoss - I'm having some trouble reproducing this issue, could you please post an example of how you're calling the beta-private-cluster module?
Getting the same when setting linux_node_config.sysctls
in google_container_node_pool
.
module.nodepool-1[0].google_container_node_pool.nodepool: Modifying... [id=projects/k1-k8s-1-ggya/locations/europe-west1/clusters/k1-k8s-1-ggya-45a0/nodePools/nodepool-1]
β·
β Error: Plugin did not respond
β
β with module.nodepool-1[0].google_container_node_pool.nodepool,
β on fabric/modules/gke-nodepool/main.tf line 70, in resource "google_container_node_pool" "nodepool":
β 70: resource "google_container_node_pool" "nodepool" {
β
β The plugin encountered an error, and failed to respond to the plugin.(*GRPCProvider).ApplyResourceChange call. The plugin logs may contain more details.
β΅
Stack trace from the terraform-provider-google-beta_v4.44.1_x5 plugin:
panic: interface conversion: interface {} is nil, not map[string]interface {}
goroutine 62 [running]:
github.com/hashicorp/terraform-provider-google-beta/google-beta.expandLinuxNodeConfig(...)
github.com/hashicorp/terraform-provider-google-beta/google-beta/node_config.go:749
github.com/hashicorp/terraform-provider-google-beta/google-beta.nodePoolUpdate(0xc000e8c680, {0x2f77760?, 0xc000e80900}, 0xc0010745d0, {0x0, 0x0}, 0x1a3185c5000)
github.com/hashicorp/terraform-provider-google-beta/google-beta/resource_container_node_pool.go:1357 +0x2c55
github.com/hashicorp/terraform-provider-google-beta/google-beta.resourceContainerNodePoolUpdate(0xc000e8c680, {0x2f77760?, 0xc000e80900?})
github.com/hashicorp/terraform-provider-google-beta/google-beta/resource_container_node_pool.go:618 +0x3b4
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).update(0x3546620?, {0x3546620?, 0xc000652780?}, 0xd?, {0x2f77760?, 0xc000e80900?})
github.com/hashicorp/terraform-plugin-sdk/v2@v2.18.0/helper/schema/resource.go:729 +0x178
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).Apply(0xc000ce7dc0, {0x3546620, 0xc000652780}, 0xc00063bd40, 0xc000e8c500, {0x2f77760, 0xc000e80900})
github.com/hashicorp/terraform-plugin-sdk/v2@v2.18.0/helper/schema/resource.go:847 +0x82c
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ApplyResourceChange(0xc00000cfc0, {0x3546620?, 0xc0006526c0?}, 0xc0006243c0)
github.com/hashicorp/terraform-plugin-sdk/v2@v2.18.0/helper/schema/grpc_provider.go:1021 +0xe3c
github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(*server).ApplyResourceChange(0xc0002ec1e0, {0x3546620?, 0xc000d2df50?}, 0xc0013c1650)
github.com/hashicorp/terraform-plugin-go@v0.10.0/tfprotov5/tf5server/server.go:813 +0x4fc
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ApplyResourceChange_Handler({0x2f05580?, 0xc0002ec1e0}, {0x3546620, 0xc000d2df50}, 0xc0013c15e0, 0x0)
github.com/hashicorp/terraform-plugin-go@v0.10.0/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:385 +0x170
google.golang.org/grpc.(*Server).processUnaryRPC(0xc0000001e0, {0x354c030, 0xc000712000}, 0xc000639b00, 0xc000d2d5c0, 0x47e0e20, 0x0)
google.golang.org/grpc@v1.50.1/server.go:1340 +0xd13
google.golang.org/grpc.(*Server).handleStream(0xc0000001e0, {0x354c030, 0xc000712000}, 0xc000639b00, 0x0)
google.golang.org/grpc@v1.50.1/server.go:1713 +0xa1b
google.golang.org/grpc.(*Server).serveStreams.func1.2()
google.golang.org/grpc@v1.50.1/server.go:965 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
google.golang.org/grpc@v1.50.1/server.go:963 +0x28a
Error: The terraform-provider-google-beta_v4.44.1_x5 plugin crashed!
This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.
Hi @joeheaton , thanks for providing information about the bug! Are you also using the beta-private-cluster module?
Could you please post some example Terraform configuration or other description in this issue to help me reproduce the bug you're seeing? For example, if you are using the module above what value are you passing in as the node_pools_linux_node_configs_sysctls
input?
I just faced this exactly problem when using google-beta provider. In my case, I could workaround the problem by replacing google-beta by google provider.
As of 4.66.0, it seems this problem is present in both google
and google-beta
providers.
Some notes of what I've noticed wrt this issue:
google-beta
provider. On the same cluster, v4.73.0 was working 22 days ago (July 10th, 2023), but is no longer working anymore. I tried v4.56.2 based on @tomzx 's comment about it failing to work as of v4.66.0. So there doesn't seem to be a correlation between provider version and this issue.terraform state rm
and then terraform import
the problematic node pool but we still end up with the same results. As a reminder, that's the following in the plan:
- linux_node_config {
- sysctls = {} -> null
}
and the following error when applying:
Error: googleapi: Error 400: At least one of ['node_version', 'image_type', 'updated_node_pool', 'locations', 'workload_metadata_config', 'upgrade_settings', 'kubelet_config', 'linux_node_config', 'tags', 'taints', 'labels', 'node_network_config', 'gcfs_config', 'gvnic', 'confidential_nodes', 'logging_config', 'fast_socket', 'resource_labels] must be specified., badRequest
linux_node_config {
sysctls = {}
}
to the Terraform and it makes the cluster that is currently failing happy. However, it causes the panic as mentioned in original post on other cluster which currently aren't running into the
Error: googleapi: Error 400: At least one of ['node_version', 'image_type', 'updated_node_pool', 'locations', 'workload_metadata_config', 'upgrade_settings', 'kubelet_config', 'linux_node_config', 'tags', 'taints', 'labels', 'node_network_config', 'gcfs_config', 'gvnic', 'confidential_nodes', 'logging_config', 'fast_socket', 'resource_labels] must be specified., badRequest
error when applying.
Unfortunately, I don't really have any conclusive findings to offer, but wanted to share any observations that I can to help
Figured out why sometimes we see the
- linux_node_config {
- sysctls = {} -> null
}
diff. It is based on whether the Google API returns back the block. You can verify by running gcloud container clusters describe ...
. If that returns back something for nodePools[*].config.linuxNodeconfig: {}
, then you'll have the above diff. Unfortunately, still not sure why some clusters will return back the field and some won't.
We are bypassing this issue for now by setting
lifecycle {
ignore_changes = [
node_config[0].linux_node_config
]
}
because we don't actually use that block. Not a permanent fix for sure.
I'm getting the same problem on the latest Google provider 5.39.1
but with the kubelet_config
. Here is the diff:
~ resource "google_container_node_pool" "pools" {
id = "projects/project1/locations/asia-east2/clusters/cluster-2/nodePools/default-spot"
name = "default-spot"
# (10 unchanged attributes hidden)
~ node_config {
- kubelet_config {
- cpu_cfs_quota = false -> null
- pod_pids_limit = 0 -> null
}
# (3 unchanged blocks hidden)
}
# (4 unchanged blocks hidden)
}
and when I try to apply it I get this output:
β Error: googleapi: Error 400: At least one of ['node_version', 'image_type', 'updated_node_pool', 'locations', 'workload_metadata_config', 'upgrade_settings', 'kubelet_config', 'linux_node_config', 'tags', 'taints', 'labels', 'node_network_config', 'gcfs_config', 'gvnic', 'confidential_nodes', 'logging_config', 'fast_socket', 'resource_labels', 'accelerators', 'windows_node_config', 'machine_type', 'disk_type', 'disk_size_gb', 'storage_pools', 'containerd_config', 'resource_manager_tags', 'performance_monitoring_unit', 'queued_provisioning', 'max_run_duration'] must be specified.
β Details:
β [
β {
β "@type": "type.googleapis.com/google.rpc.RequestInfo",
β "requestId": "0x69268b2f63ffd33c"
β }
β ]
β , badRequest
β
β with module.gke_cluster.google_container_node_pool.pools["default-spot"],
β on ../../../../modules/gke-cluster/node_pools.tf line 4, in resource "google_container_node_pool" "pools":
β 4: resource "google_container_node_pool" "pools" {
I know exactly what I did for the GCP API to start returning the kubelet_config
block. I got an email from GCP to disable the insecure kubelet port. Because it's still not implemented in Terraform (issue), I followed the GCP guide and after this when I ran TF it reported that it wants to remove the kubelet_config
block because I didn't set it in TF. For now I workaround it like @Guent4 suggested with ignore_changes
, but this is not the right way to go.
This issue should get a higher priority because Google's suggestion is causing this problem in the first place.
Community Note
modular-magician
user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned tohashibot
, a community member has claimed the issue already.Terraform Version
Terraform v1.2.9 on linux_amd64
Affected Resource(s)
Terraform Configuration Files
I am using the Google Kubernetes Engine CFT modules. I patched manually to workaround an issue where the linux_node_config was seeing an empty sysctl being set and Terraform could not remove it. The linux_node_config is being added to some clusters, but not all. When it is added to a cluster, it seems to be on all node pools in that cluster.
When running with default module, I see a perm-diff of
to ensure this parameter was set, I patched the CFT module as such https://github.com/billyfoss/terraform-google-kubernetes-engine/commit/baf9dc1a2bc4505324844c7ef44c26e8306b9e99
This gave a plan of
When running with that, I got the panic below.
Debug Output
Panic Output
Note: this is a redacted/limited output of the panic. If I see it again, I can parse out a smaller set of code to recreate it.
Expected Behavior
I was hoping to be able to trigger the linux_node_config parameter to become active in the cluster node pool without recreating the existing nodes.
Actual Behavior
Panic
Steps to Reproduce
terraform apply
Important Factoids
Note: I am running 4.33 because of https://github.com/hashicorp/terraform-provider-google/issues/12422
References
12422
b/302797579