AKS state drift for some optional fields in spot node pools (eviction_policy, node_taints)

[weisdd]

Is there an existing issue for this?

• [X] I have searched the existing issues

Community Note

• Please vote on this issue by adding a :thumbsup: reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

I'll open a PR myself soon.

Terraform Version

v1.2.8

AzureRM Provider Version

3.21.0

Affected Resource(s)/Data Source(s)

azurerm_kubernetes_cluster_node_pool

Terraform Configuration Files

provider "azurerm" {
  features {}
}

resource "azurerm_resource_group" "example" {
  name     = "test-k8s-resources"
  location = "West Europe"
}

resource "azurerm_kubernetes_cluster" "example" {
  name                = "test-k8s"
  location            = azurerm_resource_group.example.location
  resource_group_name = azurerm_resource_group.example.name
  dns_prefix          = "test-k8s"

  default_node_pool {
    name       = "default"
    node_count = 1
    vm_size    = "Standard_DS2_v2"
  }

  identity {
    type = "SystemAssigned"
  }
}

resource "azurerm_kubernetes_cluster_node_pool" "example" {
  name                  = "spot"
  kubernetes_cluster_id = azurerm_kubernetes_cluster.example.id
  vm_size               = "Standard_DS2_v2"
  node_count            = 1
  priority              = "Spot"
# NOTE: the fields below should all act as optional, left as an example
#   eviction_policy       = "Delete"
#   spot_max_price        = 0.5 # note: this is the "maximum" price
#   node_labels = {
#     "kubernetes.azure.com/scalesetpriority" = "spot"
#   }
#   node_taints = [
#     "kubernetes.azure.com/scalesetpriority=spot:NoSchedule"
#   ]
}

Debug Output/Panic Output

-

Expected Behaviour

Once the infrastructure is deployed, the subsequent calls to terraform apply / terraform plan should produce no diffs: No changes. Your infrastructure matches the configuration.

Actual Behaviour

terraform suggests to replace the spot node pool due to changes in eviction_policy and node_taints:

  # azurerm_kubernetes_cluster_node_pool.example must be replaced
-/+ resource "azurerm_kubernetes_cluster_node_pool" "example" {
      - enable_auto_scaling    = false -> null
      - enable_host_encryption = false -> null
      - enable_node_public_ip  = false -> null
      - eviction_policy        = "Delete" -> null # forces replacement
      - fips_enabled           = false -> null
      ~ id                     = "/subscriptions/64842ced-4781-416f-81ff-482b7f562581/resourceGroups/test-k8s-resources/providers/Microsoft.ContainerService/managedClusters/test-k8s/agentPools/spot" -> (known after apply)
      ~ kubelet_disk_type      = "OS" -> (known after apply)
      - max_count              = 0 -> null
      ~ max_pods               = 110 -> (known after apply)
      - min_count              = 0 -> null
        name                   = "spot"
      ~ node_labels            = {
          - "kubernetes.azure.com/scalesetpriority" = "spot"
        } -> (known after apply)
      - node_taints            = [
          - "kubernetes.azure.com/scalesetpriority=spot:NoSchedule",
        ] -> null # forces replacement
      + orchestrator_version   = (known after apply)
      ~ os_disk_size_gb        = 128 -> (known after apply)
      ~ os_sku                 = "Ubuntu" -> (known after apply)
      - tags                   = {} -> null
      - zones                  = [] -> null
        # (10 unchanged attributes hidden)
    }

Plan: 1 to add, 0 to change, 1 to destroy.

Steps to Reproduce

1. terraform apply
2. terraform plan

Important Factoids

No response

References

As you can see from https://docs.microsoft.com/en-gb/azure/aks/spot-node-pool, both eviction_policy and node_taints are optional and are set to Delete and kubernetes.azure.com/scalesetpriority=spot:NoSchedule respectively by Azure if not explicitly provided. Thus, I'd expect azurerm provider to treat those as defaults.

[stephybun]

Hey @weisdd,

This was previously raised over in https://github.com/hashicorp/terraform-provider-azurerm/issues/18237 and as explained in the response to your PR #18238 this is by design.

Thanks for taking the time to raise this issue but since this behaviour is by design I am going to mark this as a duplicate and close this issue.

[weisdd]

@stephybun alright, thanks for pointing that out!

[github-actions[bot]]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.