Auto-assigned AKS route table is removed again on next run

0x7f commented 5 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) Version

$ terraform -v
Terraform v0.12.3
+ provider.azurerm v1.30.1
+ provider.local v1.3.0
+ provider.null v2.1.2

Affected Resource(s)

azurerm_kubernetes_cluster
azurerm_virtual_network
azurerm_route_table

Terraform Configuration Files

provider "azurerm" {
  version         = ">= 1.30.0"
  tenant_id       = "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX"
  subscription_id = "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX"
}

resource "azurerm_resource_group" "resource_group" {
  name     = "test-rg"
  location = "West Europe"
}

resource "azurerm_virtual_network" "virtual_network" {
  name                = "test-vnet"
  location            = azurerm_resource_group.resource_group.location
  resource_group_name = azurerm_resource_group.resource_group.name
  address_space       = ["10.1.0.0/16"]
}

resource "azurerm_subnet" "subnet" {
  name                      = "test-subnet"
  resource_group_name       = azurerm_resource_group.resource_group.name
  virtual_network_name      = azurerm_virtual_network.virtual_network.name
  address_prefix            = "10.1.0.0/24"
}

resource "azurerm_kubernetes_cluster" "kubernetes_cluster" {
  resource_group_name = azurerm_resource_group.resource_group.name
  location            = azurerm_resource_group.resource_group.location
  name                = "test-cluster"
  dns_prefix          = "testcluster876"
  kubernetes_version  = "1.14.0"

  agent_pool_profile {
    name            = "default"
    count           = 2
    vm_size         = "Standard_D1_v2"
    os_type         = "Linux"
    os_disk_size_gb = 30
    vnet_subnet_id  = azurerm_subnet.subnet.id
  }

  service_principal {
    client_id     = var.kubernetes_cluster_service_principal_client_id
    client_secret = var.kubernetes_cluster_service_principal_client_secret
  }
}

Debug Output

https://gist.github.com/0x7f/2775e85bb6e07d9dece4644919d3f1e6

Expected Behavior

Since release 2019-06-18, when creating an AKS cluster and providing a custom subnet (with no route table assigned), the auto-generated route table will be assigned to the provided subnet. When running terraform apply (or plan) again, the route table is not disassociated from the subnet and the cluster's internal network keeps working.

Actual Behavior

On first run of terraform apply, the AKS cluster is created and its route table is automatically assigned to the provided subnet (which is also managed by terraform). But when running terraform apply again, the route table is planned to be removed from the subnet.

I totally understand why it is removed. There is no route table configured in the subnet. But I also can not reference AKS' route table in the subnet, because it would create a cyclic dependency (apart from the fact that the AKS resource is not exporting the route table id).

Right now there are only two options to get around this:

Let AKS create a subnet, add a provisioner "local-exec" to the azurerm_kubernetes_cluster resource which exports the subnet details via azure cli so you can use it in other places (e.g. configure PostgreSQL firewall). This has drawbacks as well and is hacky.
Assign a custom route table to the subnet, then AKS will not assign its own. But then you have to manually maintain the route table entries to have a working internal Kubernetes network.

Both options are not ideal. Right now, we are going with the first one. I'd like to have expected behavior though. Do you see a way how the subnet can keep the auto-assigned route table?

Steps to Reproduce

Use tf file posted above and then run terraform apply twice.

Important Factoids

Nothing particular.

References

Did search for existing/related issues and PRs, but did not find any.

WebSpider commented 5 years ago

There is another workaround, add this to the subnet:

lifecycle {
  ignore_changes = ["route_table_id"]
}

tombuildsstuff commented 5 years ago

👋

To give an update here, the association between a Subnet and a Route Table in the Azure Provider is moving from the azurerm_subnet resource to the azurerm_subnet_route_table_association resource - to not break compatibility both fields can currently be used for this purpose, however in the upcoming version 2.0 of the Azure Provider the route_table_id field will be dropped from the azurerm_subnet resource. As such this'll be fixed by 2.0 and I've added this to that milestone to ensure it's tracked/closed out as needed - but for the moment it should be possible to work around this using the ignore_changes suggestion mentioned by @WebSpider above.

Thanks!

ghost commented 4 years ago

This has been released in version 2.0.0 of the provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading. As an example:

provider "azurerm" {
    version = "~> 2.0.0"
}
# ... other configuration ...

ghost commented 4 years ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 hashibot-feedback@hashicorp.com. Thanks!

hashicorp / terraform-provider-azurerm