search

hashicorp / terraform-provider-azurerm

Terraform provider for Azure Resource Manager
https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs
Mozilla Public License 2.0
4.52k stars 4.6k forks source link

Importing the `azurerm_subnet_route_table_association` resource marks it for replacement because of case difference #14625

Open Igor-Golivets-cdw-ca opened 2 years ago

Igor-Golivets-cdw-ca commented 2 years ago

Hi, I'm trying to import the existing resources to the Terraform (subnets and route tables) and after successful import, I got that route_table_id is different and that's why the association between subnet and route table must be recreated. The RT IDs shown in the output of terraform plan are actually the same but with a different case. Another interesting thing is in the state file the imported route_table_id is all lower-cased rg name, RT name, when I adjust it manually to needed - this makes it even worse.

Community Note

Terraform (and AzureRM Provider) Version

Terraform v1.1.0 on windows_amd64

Affected Resource(s)

resource: azurerm_subnet_route_table_association

Terraform Configuration Files

resource "azurerm_route_table" "rt" {
  name                = var.name
  location            = var.location
  resource_group_name = var.resource_group_name
  dynamic "route" {
    for_each = var.routes
    content {
      name                   = route.value.name
      address_prefix         = route.value.address_prefix
      next_hop_type          = route.value.next_hop_type
      next_hop_in_ip_address = lookup(route.value, "next_hop_in_ip_address", null)
    }
  }
  disable_bgp_route_propagation = var.disable_bgp_route_propagation
  tags                          = var.tags
}

resource "azurerm_subnet_route_table_association" "route_association" {
  for_each       = toset(var.subnet_ids)
  subnet_id      = each.key
  route_table_id = azurerm_route_table.rt.id
  timeouts {}
}

Debug Output

Panic Output

Expected Behaviour

Terraform should no difference for resources that were just imported.

Actual Behaviour

I get that "subnet associations must be recreated due to the difference in attribute" route_table_id: ~ route_table_id = "/subscriptions/###########/resourceGroups/###########-rg/providers/Microsoft.Network/routeTables/###########-rt" -> "/subscriptions/###########/resourceGroups/###########-RG/providers/Microsoft.Network/routeTables/###########-RT" # forces replacement

Steps to Reproduce

  1. `terraform import 'module.route_table[\"obj-name\"].azurerm_subnet_route_table_association.route_association[\"/subscriptions/###########/resourceGroups/###########/providers/Microsoft.Network/virtualNetworks/###########/subnets/###########\"]' /subscriptions/###########/resourceGroups/###########/providers/Microsoft.Network/virtualNetworks/###########/subnets/###########
  2. terraform plan

Important Factoids

The only workaround I found so far is to add ignore config to my azurerm_subnet_route_table_association resource, but that cannot be used as a permanent solution:

  lifecycle {
     ignore_changes = ["route_table_id"]
  }

References

Igor-Golivets-cdw-ca commented 2 years ago

Hi @lonegunmanb here is the new issue I've created, let me know if you can help me. Thanks

lonegunmanb commented 2 years ago

Hello @Igor-Golivets-cdw-ca , thanks for your post, please correct me if I'm wrong:

  1. You've written code as you posted.
  2. You've imported azurerm_route_table and azurerm_subnet_route_table_association by terraform import you've described.
  3. After execute terraform plan, you've found that azurerm_subnet_route_table_association.route_table_id was changed from lower case route table name to upper case, and that caused a drift or even force new.

Would you please give me your var.name, the route table name in your terraform import command(last part of your ########### mask) for both route table and associaion? I need compose a minimum code that can reproduce this issue, maybe all arguments assigned by hard code would be best. Thanks!

Igor-Golivets-cdw-ca commented 2 years ago

Hi @lonegunmanb, one small correction is that when you import route table association you need to specify the subnet resource ID (but route table ID is probably taken from subnet properties, and seems it is taken in lower case to the state). My route table name is the following: CDW-PD-USNCZ-AVC_Priv-001_SNET_10.32.2.128_26-RT My subnet name is the following: CDW-PD-USNCZ-AVC_Priv-001_SNET_10.32.2.128_26 Thanks

lonegunmanb commented 2 years ago

Hi @lonegunmanb, one small correction is that when you import route table association you need to specify the subnet resource ID (but route table ID is probably taken from subnet properties, and seems it is taken in lower case to the state). My route table name is the following: CDW-PD-USNCZ-AVC_Priv-001_SNET_10.32.2.128_26-RT My subnet name is the following: CDW-PD-USNCZ-AVC_Priv-001_SNET_10.32.2.128_26 Thanks

Hi @Igor-Golivets-cdw-ca , thanks for clarification, so both table name and subnet name you've used in your import command and your terraform code are both in upper case, but somehow they've been converted into lower case in the tfstate and that caused the drift, right?

Igor-Golivets-cdw-ca commented 2 years ago

Yes, that's correct, everywhere name is on upper case, but somehow it is imported to lower case, I believe this makes the difference (I think it should be case insensitive comparison of resource ID of subnet).

lonegunmanb commented 2 years ago

Yes, that's correct, everywhere name is on upper case, but somehow it is imported to lower case, I believe this makes the difference (I think it should be case insensitive comparison of resource ID of subnet).

Great! I'll try whether I can reproduce it on my side, thanks for your confirmation.

lonegunmanb commented 2 years ago

Hi @Igor-Golivets-cdw-ca , I've tried to reproduce this issue but I failed. Here's my procedures:

  1. Write code below:
provider "azurerm" {
  features {}
}

resource "azurerm_resource_group" "rg" {
  location = "westus"
  name     = "ZJHE-VNET"
}

resource "azurerm_virtual_network" "vnet" {
  address_space       = ["10.0.0.0/16"]
  location            = "eastus"
  name                = "ZJHE-VNET"
  resource_group_name = azurerm_resource_group.rg.name
}

resource "azurerm_subnet" "private" {
  name                 = "PRIVATE"
  resource_group_name  = azurerm_resource_group.rg.name
  virtual_network_name = azurerm_virtual_network.vnet.name
  address_prefixes     = ["10.0.0.0/24"]
}

resource "azurerm_route_table" "rt" {
  location            = "eastus"
  name                = "RT"
  resource_group_name = azurerm_resource_group.rg.name
}

resource "azurerm_subnet_route_table_association" "this" {
  route_table_id = azurerm_route_table.rt.id
  subnet_id      = azurerm_subnet.private.id
}

output "rt_id" {
  value = azurerm_route_table.rt.id
}

output "subnet_rt_asso_id" {
  value = azurerm_subnet_route_table_association.this.id
}

output "resource_group_id" {
  value = azurerm_resource_group.rg.id
}

output "vnet_id" {
  value = azurerm_virtual_network.vnet.id
}

output "subnet_id" {
  value = azurerm_subnet.private.id
}

As you can see all names here are upper case.

  1. terraform apply -auto-approve

All resources' ids were printed as outputs.

  1. Copy this code into a new folder, re-run terraform init, then terraform import all resources one by one, using the ids printed by previous root module.

  2. Run terraform plan, then cli showed "No changes. Your infrastructure matches the configuration."

Would you please help me by providing a minimum sample code and procedures that can reproduce the issue you've met, or by pointing out what I've missed above? Thanks!

Igor-Golivets-cdw-ca commented 2 years ago

Hi @lonegunmanb , the problem here is that I don't know the code with which those resources were created initially, probably some names used in naming were different but resource explorer shows all of them in uppercase. But I don't know how to confirm/check that.

lonegunmanb commented 2 years ago

Ok, I will try import resource with lower case name, see if it could successfully import resource with upper case name and leave lower case name in state file, if it works, maybe it can explain what you've met.

Igor-Golivets-cdw-ca commented 2 years ago

Seems I found the cause of it, in the resource JSON of the subnet I can see route table ID with lowercased: RG and route table name. So the one who created this didn't use the correct case there, and now terraform takes these values on import and doesn't match them.

lonegunmanb commented 2 years ago

Seems I found the cause of it, in the resource JSON of the subnet I can see route table ID with lowercased: RG and route table name. So the one who created this didn't use the correct case there, and now terraform takes these values on import and doesn't match them.

So someone else created route table with lower case name in a terraform json code file, and then you import this route table into your hcl code by terraform import upper case name route table id right? Then if table name was lower case when it was created by json tf code, why you can see it in upper case?

Igor-Golivets-cdw-ca commented 2 years ago

So someone else created route table with lower case name in a terraform json code file, and then you import this route table into your hcl code by terraform import upper case name route table id right? Then if table name was lower case when it was created by json tf code, why you can see it in upper case? Hard to tell now, but maybe resource name and display name were different during the creation, Azure is not consistent in this:

image

lonegunmanb commented 2 years ago

Hi @Igor-Golivets-cdw-ca , I've reproduced this issue on my side, but I don't think my way is a plugin's bug. Here's my procedures:

  1. Apply code as below:
provider "azurerm" {
  features {}
}

resource "azurerm_resource_group" "rg" {
  location = "westus"
  name     = "ZJHE-VNET"
}

resource "azurerm_virtual_network" "vnet" {
  address_space       = ["10.0.0.0/16"]
  location            = "eastus"
  name                = "ZJHE-VNET"
  resource_group_name = azurerm_resource_group.rg.name
}

resource "azurerm_subnet" "private" {
  name                 = "PRIVATE"
  resource_group_name  = azurerm_resource_group.rg.name
  virtual_network_name = azurerm_virtual_network.vnet.name
  address_prefixes     = ["10.0.0.0/24"]
}

resource "azurerm_route_table" "rt" {
  location            = "eastus"
  name                = "RT"
  resource_group_name = azurerm_resource_group.rg.name
}

resource "azurerm_subnet_route_table_association" "this" {
  route_table_id = azurerm_route_table.rt.id
  subnet_id      = azurerm_subnet.private.id
}

output "rt_id" {
  value = azurerm_route_table.rt.id
}

output "subnet_rt_asso_id" {
  value = azurerm_subnet_route_table_association.this.id
}

output "resource_group_id" {
  value = azurerm_resource_group.rg.id
}

output "vnet_id" {
  value = azurerm_virtual_network.vnet.id
}

output "subnet_id" {
  value = azurerm_subnet.private.id
}
  1. Copy code file into a new folder.
  2. terraform import all resources one by one, but leave subnet's name / route table's name in three resources' id lowercase, for example:
> terraform import azurerm_subnet_route_table_association.this /subscriptions/###########/resourceGr
oups/ZJHE-VNET/providers/Microsoft.Network/virtualNetworks/ZJHE-VNET/subnets/private

It seems like Azure api is case insensitive, all resource id with lower case name were successfully imported.

  1. run terraform plan, you'll see drift and force replacement.

The reason of this case is due to import by incorrect resource id will success, and I think Terraform provider plugin has nothing to do about that. In my case, all I can do is remove incorrect resource state from origin state file and import it again with correct resource id. And if possible, do avoid manipulating terraform state manually, even Terraform import could do harm.

After all, this is my personal guess. I don't know if it could answer your question.

Igor-Golivets-cdw-ca commented 2 years ago

Hi @lonegunmanb , thanks for checking that, I've tried to reimport resources again with the correct case of resource ID, but then I got the other drift from the terraform, the whole resources want to be recreated, as of now the workaround I found is good for me. p.s. if terraform could not compare resource ID case sensitive that would be fine (or we could have switch to turn case sensitivity off for some resources) that would be ideal.

jaygridley commented 2 years ago

Hi guys, we are facing exactly the same issue. In the past, we have created Route Table (RT) and its Resource Group (RG)with all letters in upper-case, but now Azure API returns those in lower-case. An interesting fact is that the RT and RG were created almost 2 years ago and we started to face this issue a week ago.

I already discussed this with Microsoft Network support and the conclusion from their side is that their APIs could return some resources in a different case than it is actually stored on their backends, see https://docs.microsoft.com/en-us/azure/azure-resource-manager/management/resource-name-rules.

They are basically saying this is a "feature" on their end, not a "bug" and suggested us to solve it with Terraform community and on a provider level.

Igor-Golivets-cdw-ca commented 2 years ago

Hi guys, we are facing exactly the same issue. In the past, we have created Route Table (RT) and its Resource Group (RG)with all letters in upper-case, but now Azure API returns those in lower-case. An interesting fact is that the RT and RG were created almost 2 years ago and we started to face this issue a week ago.

I already discussed this with Microsoft Network support and the conclusion from their side is that their APIs could return some resources in a different case than it is actually stored on their backends, see https://docs.microsoft.com/en-us/azure/azure-resource-manager/management/resource-name-rules.

They are basically saying this is a "feature" on their end, not a "bug" and suggested us to solve it with Terraform community and on a provider level.

I believe Terraform should not have case sensitive comparison here anyhow, so hopefully, this can be solved.

tombuildsstuff commented 2 years ago

@jaygridley having chatted extensively with the ARM Team about this - Resource ID's in API's are supposed to be case-insensitive during requests and case-sensitive in responses - so unfortunately that appears to be a miscommunication with the Networking Team, who are returning components insensitively in the API - so ultimately this needs to be fixed upstream.

FWIW we've seen multiple examples of API's behaving incorrectly if the incorrect casing is specified for segments in the Resource ID - so unfortunately treating these case-insensitively would only further break those scenarios.

Terraform is moving away from using the Resource ID returned from Azure in favour of defining these within Terraform (as the Resource ID is predictable) - which should allow for working around this issue longer-term, however in the interim this needs to be fixed within the Networking API.

jaygridley commented 2 years ago

Unfortunately I got another push bash from Microsoft stating that even the responses are case-insensitive and they call it out in the documentation (https://docs.microsoft.com/en-us/azure/azure-resource-manager/management/resource-name-rules). They say this is how it is designed and not likely to change, but I am welcomed to raise a feature request for this.

We have workarounded this by renaming our resources in the Terraform to lower-case form which is an intermediate fix and can break at any time in the future.

Hope the future releases of the provider will bring stability to this as you mentioned.

pjolsen commented 2 years ago

Any word on which version of Terraform will start to calculate the resource ID rather than take the value from the Azure API?

rgarcia89 commented 2 years ago

I am seeing this issue too... See the difference between the id and route_table_id line. This is causing a terraform apply to always want to recreate the related resource.

Here a view from the terraform.tfstate

      "module": "module.nodepool",
      "mode": "managed",
      "type": "azurerm_subnet_route_table_association",
      "name": "aks",
      "provider": "module.nodepool.provider[\"registry.terraform.io/hashicorp/azurerm\"]",
      "instances": [
        {
          "schema_version": 0,
          "attributes": {
            "id": "/subscriptions/XXXXX/resourceGroups/AGW_LABS-RGS/providers/...",
            "route_table_id": "/subscriptions/XXXXX/resourceGroups/agw_labs-rgs/providers/...",
            "subnet_id": "/subscriptions/XXXXX/resourceGroups/AGW_LABS-RGS/providers/...",
            "timeouts": null
rpiteira commented 1 year ago

Hello, I'm running the same problem with a AKS and Route Table. Changing the state file doesn't work.

rgarcia89 commented 1 year ago

@rpiteira try to add the following to your azurerm_route_table definition. It is just a workaround, but helps for the moment.


  lifecycle { # will ignore dynamic changes made by AKS
    ignore_changes = [route]
  }
``´
rpiteira commented 1 year ago

@rpiteira try to add the following to your azurerm_route_table definition. It is just a workaround, but helps for the moment.

  lifecycle { # will ignore dynamic changes made by AKS
    ignore_changes = [route]
  }
``´

Yes it works, I confirm! But the code is not clean :) Thanks @rgarcia89