Open Noel-Jones opened 1 month ago
I also experience this issue. Change that causes this behavior was added in PR #26137. Here is the line that forces recreation: https://github.com/hashicorp/terraform-provider-azurerm/pull/26137/files#diff-b7c1d78864b169130b7e4c32a1ff2667efb4a47c9713e26f1a888cb9e8e582bbR71
I think I found a workaround. Adding upgrade_settings block to node pool configuration prevents Terraform from trying to recreate whole cluster. This works for both azurerm_kubernetes_cluster
and azurerm_kubernetes_cluster_node_pool
resource "azurerm_kubernetes_cluster" "default" {
default_node_pool {
upgrade_settings {
max_surge = "10%"
drain_timeout_in_minutes = 30
}
}
}
resource "azurerm_kubernetes_cluster_node_pool" "node_pool" {
upgrade_settings {
max_surge = "10%"
drain_timeout_in_minutes = 30
}
}
We upgraded our cluster (via az CLI), and then upgraded azurerm provider from 3.56.0
to 4.0.1
. And we are stuck in such loop:
Changing drain_timeout_in_minutes
forces node pool to be recreated. But we cannot upgrade upgrade_settings
, because it requires parameter max_surge
:
And we cannot set max_surge
for spot pools, as it is not allowed:
And Terraform does not allow to ignore changes inside dynamic blocks.
In fact, even removing state and importing it from scratch causes this behavior. The azurerm
provider has some serious bugs.
Is there an existing issue for this?
Community Note
Terraform Version
1.9.0
AzureRM Provider Version
3.106.0
Affected Resource(s)/Data Source(s)
azurerm_kubernetes_cluster
Terraform Configuration Files
Debug Output/Panic Output
Expected Behaviour
There should be no changes to the existing AKS resource. The cluster should certainly not be recreated.
Actual Behaviour
The AKS resource is planned to be replaced:
Steps to Reproduce
Terraform apply to Create cluster with provider version 3.105.0
Update provider version to 3.106 (or up to 3.111 at time of writing)
(Terraform plan now will only show a change)
Upgrade the cluster kubernetes version (in my case it was created with 1.28.10 and upgraded to 1.29.5) and I upgraded the control plane then the node plane.
Terraform plan will now want to recreate the cluster.
I hesitated to raise this having scanned through the other issues raised since 3.106 but they are all closed and planning to recreate the cluster seems wrong. I'm not sure of the appropriate resolution. There is clearly an external change to the resource brought about by the cluster upgrade.
Before the cluster upgrade aks nodepool show returns:
After the cluster upgrade aks nodepool show returns:
It seems to me that the defaults for the upgrade settings should be 30 for drainTimeoutInMinutes (currently optional/null), 10% or 1 for maxSurge (you could argue 1 will have least impact on existing clusters) and null for nodeSoakDurationInMinutes (as now).
Maybe the drainTimeoutInMinutes and maxSurge should both be required but with further explaination in the documentation?
Important Factoids
No response
References
No response