Terraform tries to re-create Bastion subnet

fnmarquez commented 2 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) Version

Terraform v1.0.3 azurerm v2.92.0

Affected Resource(s)

azurerm_virtual_network
azurerm_subnet

Terraform Configuration Files

#main.tf
provider "azurerm" {
  features {}
}

terraform {
}

resource "azurerm_network_security_group" "custom" {
  name                = "test1-nsg-custom"
  resource_group_name = var.resource_group_name
  location            = var.vnet_location
}

module "virtual_network" {
  source              = "./terraform-azurerm-virtual-network"
  resource_group_name = var.resource_group_name
  vnet_name           = "test1-vnet"
  address_space       = ["10.0.0.0/17"]
  vnet_location       = var.vnet_location
  vnet_tags = {
    environment = "dev-1"
  }
  subnets = [
    {
      name             = "test1-subnet-2"
      address_prefixes = "10.0.0.0/24"
    },
    {
      name              = "test1-subnet2sg"
      address_prefixes  = "10.0.2.0/24"
      security_group_id = azurerm_network_security_group.custom.id
    },
    {
      name             = "AzureBastionSubnet"
      address_prefixes = "10.0.1.0/24"
    },
    {
      name             = "AzureFirewallSubnet"
      address_prefixes = "10.0.3.0/26"
    }
  ]
  nsg_name = "test1-nsg"
  nsg_tags = {
    environment = "dev"
  }
}

#variables.tf
variable "resource_group_name" {
  type        = string
  description = "Name of the resource group to be imported."
  default     = "test-ussc-dev"
}

variable "network_security_group_name" {
  type        = string
  description = "Name of the network security group to be imported."
  default     = "test-nsg"
}

variable "vnet_location" {
  type        = string
  description = "Location of vNet and NSG. Note, the NSG's location will automatically be set to match the vNet's location."
  default     = "East US"
}

#module: main.tf

data "azurerm_resource_group" "rg" {
  name = var.resource_group_name
}

resource "azurerm_virtual_network" "vnet" {
  name                = var.vnet_name
  resource_group_name = data.azurerm_resource_group.rg.name
  location            = local.location
  address_space       = var.address_space
  dns_servers         = var.dns_servers
  tags                = var.vnet_tags

  dynamic "ddos_protection_plan" {
    for_each = var.ddos_id != null ? [1] : []
    content {
      id     = var.ddos_id
      enable = true
    }
  }

  dynamic "subnet" {
    #iterates through list map provided in child module / variables.tf; translates into a map where the map key is the subnet name
    for_each = { for subnet in var.subnets : subnet.name => subnet }
    content {
      name           = subnet.value.name
      address_prefix = subnet.value.address_prefixes
      security_group = lookup(subnet.value, "security_group_id", null)
    }
  }
}

resource "azurerm_network_security_group" "nsg" {
  name                = var.nsg_name
  location            = local.location
  resource_group_name = data.azurerm_resource_group.rg.name
  tags                = var.nsg_tags
}

Debug Output

Panic Output

Expected Behaviour

Terraform shouldn't try to recreate the bastion subnet when no changes are being made to that subnet.

Actual Behaviour

Terraform recreates the bastion subnet

Steps to Reproduce

Modify a the tag value of the vnet resource
terraform apply
AzureBastion subnet is recreated

Important Factoids

References

0000

neil-yechenwei commented 2 years ago

@fnmarquez , thanks for raising this issue. Here recreating the bastion subnet you mentioned means terraform tries to update the whole subnet block, right? If yes, could you share the result by running terraform plan after modified the tag value of vnet resource? Actually, after tested with below tf config that is similar with yours on my local, seems I cannot repro the issue you mentioned. If it's possible, could you also try below tf config with latest azurerm provider to see if the issue still exists?

tf config:

provider "azurerm" {
  features {}
}

resource "azurerm_resource_group" "test" {
  name     = "acctestRG-vnetsubnet-test01"
  location = "eastus"
}

resource "azurerm_network_security_group" "test" {
  name                = "acctest-nsg-test01"
  resource_group_name = azurerm_resource_group.test.name
  location            = azurerm_resource_group.test.location

  tags = {
      env = "nsgtest"
  }
}

resource "azurerm_network_ddos_protection_plan" "test" {
  name                = "acctestddospplan-test01"
  location            = azurerm_resource_group.test.location
  resource_group_name = azurerm_resource_group.test.name
}

resource "azurerm_virtual_network" "test" {
  name                = "acctestvirtnettest01"
  resource_group_name = azurerm_resource_group.test.name
  location            = azurerm_resource_group.test.location
  address_space       = ["10.0.0.0/16"]
  dns_servers         = ["10.7.7.2", "10.7.7.7", "10.7.7.1"]

  ddos_protection_plan {
    id     = azurerm_network_ddos_protection_plan.test.id
    enable = true
  }

  subnet {
    name           = "subnet1"
    address_prefix = "10.0.2.0/24"
  }

  subnet {
    name           = "subnet2"
    address_prefix = "10.0.3.0/24"
    security_group = azurerm_network_security_group.test.id
  }

  subnet {
    name           = "AzureBastionSubnet"
    address_prefix = "10.0.1.0/24"
  }

  subnet {
    name           = "AzureFirewallSubnet"
    address_prefix = "10.0.4.0/26"
  }

  tags = {
    env = "vnettest2"
  }
}

fnmarquez commented 2 years ago

@neil-yechenwei

Terraform will perform the following actions:

  # azurerm_virtual_network.test will be updated in-place
  ~ resource "azurerm_virtual_network" "test" {
        id                      = "/subscriptions/xxxxxxxxx/resourceGroups/acctestRG-vnetsubnet-test01/providers/Microsoft.Network/virtualNetworks/acctestvirtnettest01"
        name                    = "acctestvirtnettest01"
      ~ tags                    = {
          ~ "env" = "vnettest2" -> "vnettest2-modified"
        }
        # (8 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

In this case it is not trying to update the whole subnet block. Do you think it is related to the dynamic block within the module?

neil-yechenwei commented 2 years ago

Sorry. Seems I still cannot repro it with module. There is no diff on tags after provisioned.

main.tf

provider "azurerm" {
  features {}
}

resource "azurerm_resource_group" "test" {
  name     = "acctestRG-vnetsubnet-test01"
  location = "eastus"
}

resource "azurerm_network_security_group" "test" {
  name                = "acctest-nsg-test01"
  resource_group_name = azurerm_resource_group.test.name
  location            = azurerm_resource_group.test.location

  tags = {
      env = "nsgtest"
  }
}

resource "azurerm_network_ddos_protection_plan" "test" {
  name                = "acctestddospplan-test01"
  location            = azurerm_resource_group.test.location
  resource_group_name = azurerm_resource_group.test.name
}

module "virtual_network" {
  source = "./terraform-azurerm-virtual-network"

  resource_group          = azurerm_resource_group.test.name
  location                = azurerm_resource_group.test.location
  ddos_protection_plan_id = azurerm_network_ddos_protection_plan.test.id

  vnet_tags = {
    environment = "dev-4"
  }

  subnets = [
    {
      name             = "test1-subnet-2"
      address_prefixes = "10.0.0.0/24"
    },
    {
      name              = "test1-subnet2sg"
      address_prefixes  = "10.0.2.0/24"
      security_group_id = azurerm_network_security_group.test.id
    },
    {
      name             = "AzureBastionSubnet"
      address_prefixes = "10.0.1.0/24"
    },
    {
      name             = "AzureFirewallSubnet"
      address_prefixes = "10.0.3.0/26"
    }
  ]
}

variables.tf in module:

variable "vnet_tags" {
  type        = map(string)
}

variable "subnets" {
  type = list(object({
    name = string
    address_prefixes = string
  }))
}

variable "resource_group" {
  type = string
}

variable "location" {
  type = string
}

variable "ddos_protection_plan_id" {
  type = string
}

main.tf in module:

resource "azurerm_virtual_network" "vnet" {
  name                = "acctesttest1-vnet"
  resource_group_name = var.resource_group
  location            = var.location
  address_space       = ["10.0.0.0/16"]
  dns_servers         = ["10.7.7.2", "10.7.7.7", "10.7.7.1"]
  tags                = var.vnet_tags

  ddos_protection_plan {
    id     = var.ddos_protection_plan_id
    enable = true
  }

  dynamic "subnet" {
    for_each = { for subnet in var.subnets : subnet.name => subnet }
    content {
      name           = subnet.value.name
      address_prefix = subnet.value.address_prefixes
      security_group = lookup(subnet.value, "security_group_id", null)
    }
  }
}

pavankumarda2136 commented 1 year ago

Hi @neil-yechenwei, I am facing the same issue when applying the terraform. I am not using any security_group on AzureBastionSubnet resource block, but when I do a terraform plan I am seeing the updates in place on this subnet with - security_group = "", But when I try to apply it, I am getting the error saying the subnet cannot be updated as its already in use. Not sure why the terraform plan is expecting the security group which is not supported argument for AzureBastionSubnet. Please help.

pavankumarda2136 commented 1 year ago

@fnmarquez did you resolved this issue. if yes, could you please let me know what you did ? Thanks in advance.

fnmarquez commented 1 year ago

@fnmarquez did you resolved this issue. if yes, could you please let me know what you did ? Thanks in advance.

@pavankumarda2136 Unfortunately I couldn't solve this problem.

pavankumarda2136 commented 1 year ago

@fnmarquez, Thank you. I have tried multiple ways but its not working. Seems like this is a bug, for the first plan and apply its working as expected but second plan and apply is failing since the azurebastionSubnet is expecting the security_group argument which is not supported for bastion subnet. @neil-yechenwei Can you please help on this ?

szasza576 commented 11 months ago

@fnmarquez I got similar error but with very standard subnets. I guess the root cause: once the subnet is created then Azure adds an empty reference to the security group which doesn't exist in the tfstate file and this confuses terraform. On every apply TF recreates the subnets due to the mismatch.

Terraform: 1.5.2 Azurerm: v3.64.0

  # azurerm_virtual_network.mainvnet will be updated in-place
  ~ resource "azurerm_virtual_network" "mainvnet" {
        id                      = "/subscriptions/xxxxxxxxxxxxxxxxxxx/resourceGroups/terratest/providers/Microsoft.Network/virtualNetworks/terravnet"
        name                    = "terravnet"
      ~ subnet                  = [
          - {
              - address_prefix = "10.0.0.0/24"
              - id             = "/subscriptions/xxxxxxxxxxxxxxxxxxx/resourceGroups/terratest/providers/Microsoft.Network/virtualNetworks/terravnet/subnets/vm-subnet"
              - name           = "vm-subnet"
              - security_group = ""
            },
          - {
              - address_prefix = "10.0.1.0/24"
              - id             = "/subscriptions/xxxxxxxxxxxxxxxxxxx/resourceGroups/terratest/providers/Microsoft.Network/virtualNetworks/terravnet/subnets/aks1-subnet"
              - name           = "aks1-subnet"
              - security_group = ""
            },
          + {
              + address_prefix = "10.0.0.0/24"
              + id             = "/subscriptions/xxxxxxxxxxxxxxxxxxx/resourceGroups/terratest/providers/Microsoft.Network/virtualNetworks/terravnet/subnets/vm-subnet"
              + name           = "vm-subnet"
            },
        ]
        tags                    = {}
        # (6 unchanged attributes hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

If I explicitly add the security_group = "" to the TF file then the issue disappears.

resource "azurerm_virtual_network" "mainvnet" {
  name                = var.vnet_name
  resource_group_name = azurerm_resource_group.rg.name
  location            = azurerm_resource_group.rg.location
  address_space       = [var.vnet_address_space]

  subnet {
    name              = var.subnet_VM_name
    address_prefix    = var.subnet_VM_size
    security_group    = ""   # Add this to solve the issue
  }
}

The problem is that this option is part of the "azurerm_virtual_network" but not as "azurerm_subnet" hence the following doesn't work:

resource "azurerm_subnet" "akssubnet" {
  name                 = var.AKS_subnet_name
  resource_group_name  = var.AKS_resource_group
  virtual_network_name = var.AKS_vnet
  address_prefixes     = [var.AKS_subnet_size]
  service_endpoints    = ["Microsoft.Storage", "Microsoft.KeyVault", "Microsoft.ContainerRegistry"]
  security_group       = ""   # This doesn't work.
}

hashicorp / terraform-provider-azurerm