Adding network_plugin_mode = "Overlay" to azurerm_kubernetes_cluster breaks kubernetes provider

Is there an existing issue for this?

[X] I have searched the existing issues

Community Note

Please vote on this issue by adding a :thumbsup: reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment and review the contribution guide to help.

Terraform Version

1.2.1

AzureRM Provider Version

3.64.0

Affected Resource(s)/Data Source(s)

azurerm_kubernetes_cluster

Terraform Configuration Files

resource "azurerm_kubernetes_cluster" "aks" {
  network_profile {
    network_plugin      = "azure"
    network_policy      = null
    network_plugin_mode = "Overlay"
    }
  }
}

locals {
  k8s_host                   = module.aks.k8s_host
  k8s_cluster_ca_certificate = base64decode(module.aks.k8s_cluster_ca_certificate)
}

provider "azurerm" {
  subscription_id = local.terraform_ci_subscription_id
  client_id       = local.client_id
  client_secret   = var.client_secret
  tenant_id       = local.tenant_id
  environment     = "public"
  features {
    log_analytics_workspace {
      permanently_delete_on_destroy = true
    }
  }
}

provider "kubernetes" {
  host                   = local.k8s_host
  cluster_ca_certificate = local.k8s_cluster_ca_certificate

  exec {
    api_version = "client.authentication.k8s.io/v1beta1"
    command     = "kubelogin"
    args = [
      "get-token",
      "--environment",
      "AzurePublicCloud",
      "--server-id",
      ""00000000-0000-0000-0000-000000000000",
      "--client-id",
      var.k8s_admin_sp_client_id,
      "--tenant-id",
      "00000000-0000-0000-0000-000000000000",
      "--client-secret",
      var.k8s_admin_sp_client_secret,
      "--login",
      "spn",
      "|",
      "jq",
      ".status.token",
    ]
  }
}

Debug Output/Panic Output

module.aks.azurerm_monitor_diagnostic_setting.aks[0]: Refreshing state... [id=/subscriptions/11c8cdfe-6cbb-447f-912a-17524f161f7b/resourceGroups/tf-azure-aks-b4c31cdc-ci-TestNetworkPlugin/providers/Microsoft.ContainerService/managedClusters/tf-azure-aks-b4c31cdc-ci-TestNetworkPlugin-k8s|tf-azure-aks-b4c31cdc-ci-TestNetworkPlugin-diagnotic-setting]
module.aks.time_sleep.wait_for_aks_cluster_admin_tf_identity_propagation: Refreshing state... [id=2023-07-11T17:29:06Z]
module.aks.kubernetes_storage_class.azureFileStandardZRSRetain: Refreshing state... [id=azurefile-standard-zrs-retain]
│ Error: storageclasses.storage.k8s.io "azurefile-standard-zrs-retain" is forbidden: User "system:serviceaccount:xinfra-ci:default" cannot get resource "storageclasses" in API group "storage.k8s.io" at the cluster scope: Azure does not have opinion for this user.
│
│ with module.aks.kubernetes_storage_class.azureFileStandardZRSRetain,
│ on ../../infra_specs.tf line 260, in resource "kubernetes_storage_class" "azureFileStandardZRSRetain":
│ 260: resource "kubernetes_storage_class" "azureFileStandardZRSRetain" {
│
╵

Expected Behaviour

Terraform should have used the configuration provided to the kubernetes provider. The actual debug doesn't reveal where its pulling the kubeconfig from, but its the default and/or whatever kubernetes environment variables are set by our gitlab runner. The k8s provider config never gets read.

I'm aware that this appears more related to the kubernetes provider, but when I remove the network_plugin_mode related parameters for azurerm_kubernetes_cluster, everything goes back to operating correctly.

Actual Behaviour

Terraform ignores the kubernetes provider configuration and instead appears to use defaults of localhost:80.

Steps to Reproduce

Add the following to what calls the module

network_policy          = null
network_plugin_mode     = "Overlay"
network_ebpf_data_plane = "cilium"

terraform apply # This succeeds when none of the resources exist
terraform destroy or terraform apply # This fails when refreshing the state when it reaches the k8s resources
Remove the azurerm_kubernetes_cluster parameters above and everything works as expected.

Important Factoids

No response

References

No response

hashicorp / terraform-provider-azurerm