network_rules delete

[gevraud]

Is there an existing issue for this?

• [X] I have searched the existing issues

Greenfield/Brownfield provisioning

brownfield

Terraform Version

1.8.2

Module Version

0.2.3

AzureRM Provider Version

3.116

Affected Resource(s)/Data Source(s)

azurerm_storage_account_network_rules

Terraform Configuration Files

module "terraform_backend" {
  source  = "Azure/avm-res-storage-storageaccount/azurerm"
  version = "0.2.3"

  enable_telemetry              = false
  account_replication_type      = "RAGRS" 
  account_tier                  = "Standard"
  account_kind                  = "StorageV2"
  access_tier                   = "Hot" 
  resource_group_name           = azurerm_resource_group.rg_sa.name
  location                      = azurerm_resource_group.rg_sa.location
  name                          = var.storage_account_name
  min_tls_version               = "TLS1_2"
  shared_access_key_enabled     = true
  public_network_access_enabled = true
  managed_identities = {
    system_assigned            = true
  }

  blob_properties = {
    versioning_enabled            = true
    change_feed_enabled           = true
    change_feed_retention_in_days = 90
    last_access_time_enabled      = true
    permanent_delete_enabled      = false

    delete_retention_policy = {
      days = 30
    }
    container_delete_retention_policy = {
      days = 30
    }
  }

  sas_policy = { 
    expiration_action = "Log"
    expiration_period = "00.02:00:00"
  }
  // NETWORK part
  network_rules = {
    bypass                     = ["Logging", "Metrics", "AzureServices"]
    default_action             = "Deny"
    virtual_network_subnet_ids = [  "/subscriptions/xxxx/resourceGroups/yyyy/providers/Microsoft.Network/virtualNetworks/xxxx/subnets/xxx",    "/subscriptions/xxxx/resourceGroups/yyyy/providers/Microsoft.Network/virtualNetworks/vnet/subnets/xxx"
    ]
  }

  containers = {
    tfstate = {
      name                  = "mytfsate"
      container_access_type = "private"
    }
  }

  tags = var.tags

  role_assignments = {
    role_assignment_1 = {
      role_definition_id_or_name       = "Owner"
      principal_id                     = data.azurerm_client_config.current.object_id
      skip_service_principal_aad_check = false
    },
    role_assignment_4 = {
      role_definition_id_or_name       = "Storage Blob Data Contributor"
      principal_id                     = data.azurerm_client_config.current.object_id
      skip_service_principal_aad_check = false
    },
  }

  private_endpoints = {
    pe1 = {
      name                            = "pe-${var.storage_account_name}-001"
      subnet_resource_id              = module.vnet.subnets[var.subnet1].id
      subresource_name                = "blob"
      private_dns_zone_resource_ids   = [azurerm_private_dns_zone.this.id]
      private_service_connection_name = "psc-${var.storage_account_name}-001"
      network_interface_name          = "nic-pe-${var.storage_account_name}-001"
      inherit_tags                    = true
      inherit_lock                    = false
      role_assignments = {
        role_assignment_1 = {
          role_definition_id_or_name = "Contributor"
          principal_id               = data.azurerm_client_config.current.object_id
        }
      }
    }
  }
}

tfvars variables values

location                 = "westeurope"

Debug Output/Panic Output

# module.terraform_backend_prod.azapi_resource.containers["tfstate"] will be updated in-place
  ~ resource "azapi_resource" "containers" {
      ~ body                      = jsonencode(
            {
              - properties = {
                  - metadata     = null
                  - publicAccess = "None"
                }
            }
        ) -> {
          + properties = {
              + immutableStorageWithVersioning = null
              + metadata                       = {}
              + publicAccess                   = "None"
            }
        }
        id                        = "/subscriptions/xxxx/resourceGroups/rg/providers/Microsoft.Storage/storageAccounts/sa/blobServices/default/containers/mytfstate"
        name                      = "tfstate"
      ~ output                    = jsonencode({}) -> (known after apply)
        tags                      = {}
      ~ type                      = "Microsoft.Storage/storageAccounts/blobServices/containers@2022-09-01" -> "Microsoft.Storage/storageAccounts/blobServices/containers@2023-01-01"
        # (6 unchanged attributes hidden)
    }

  # module.terraform_backend_prod.azurerm_storage_account_network_rules.this[0] will be destroyed
  # (because azurerm_storage_account_network_rules.this is not in configuration)
  - resource "azurerm_storage_account_network_rules" "this" {
      - bypass                     = [
          - "AzureServices",
          - "Logging",
          - "Metrics",
        ] -> null
      - default_action             = "Deny" -> null
      - id                         = "/subscriptions/xxxx/resourceGroups/rg/providers/Microsoft.Storage/storageAccounts/sa" -> null
      - ip_rules                   = [] -> null
      - storage_account_id         = "/subscriptions/xxxx/resourceGroups/rg/providers/Microsoft.Storage/storageAccounts/sa" -> null
      - virtual_network_subnet_ids = [
          - "/subscriptions/xxxx/resourceGroups/rg/providers/Microsoft.Network/virtualNetworks/xxxx/subnets/xxxx",
          - "/subscriptions/xxxx/resourceGroups/rg/providers/Microsoft.Network/virtualNetworks/xxxx/subnets/xxxx",
        ] -> null
    }

  # module.terraform_backend_prod.time_sleep.wait_for_rbac_before_container_operations[0] will be destroyed
  # (because time_sleep.wait_for_rbac_before_container_operations is not in configuration)
  - resource "time_sleep" "wait_for_rbac_before_container_operations" {
      - create_duration  = "30s" -> null
      - destroy_duration = "0s" -> null
      - id               = "2024-08-19T11:32:00Z" -> null
      - triggers         = {
          - "role_assignments" = jsonencode(
                {
                  - role_assignment_1 = {
                      - condition                              = null
                      - condition_version                      = null
                      - delegated_managed_identity_resource_id = null
                      - description                            = null
                      - principal_id                           = "aaaa"
                      - principal_type                         = null
                      - role_definition_id_or_name             = "Owner"
                      - skip_service_principal_aad_check       = false
                    }
                  - role_assignment_4 = {
                      - condition                              = null
                      - condition_version                      = null
                      - delegated_managed_identity_resource_id = null
                      - description                            = null
                      - principal_id                           = "eeeee"
                      - principal_type                         = null
                      - role_definition_id_or_name             = "Storage Blob Data Contributor"
                      - skip_service_principal_aad_check       = false
                    }
                }
            )
        } -> null
    }

Plan: 0 to add, 1 to change, 2 to destroy.

Expected Behaviour

nothing should change

Actual Behaviour

it tries to delete the network_rules

Steps to Reproduce

No response

Important Factoids

No response

References

No response

[chinthakaru]

@gevraud Thank you for reaching out. I can see the details about the azurerm_storage_account_network_rules resource along with your error details. However, we do not use the azurerm_storage_account_network_rules within this module. Could you please run terraform init -upgrade and test it again? #RR

[gevraud]

Hello,

I did the terraform init -upgrade but the behavior is the same.

The problem wasn't there in version 0.1.3, 0.2.0. I just moved from 0.1.3 to 0.2.0 without any problem.

It appears in version 0.2.1.

Regards

[chinthakaru]

@gevraud, a new version has been released with several updates. Could you please try again using version 0.2.4?

RR

[gevraud]

Same again. I rollbacked to version 0.2.0, which displays "No changes" after the plan

[chinthakaru]

@gevraud I can't reproduce this error on my tenant. Please can you share your full configuration here ?

RR

[gevraud]

Sure

data "azurerm_client_config" "current" {}
resource "azurerm_resource_group" "rg_sa" {
  name     = var.rg_sa
  location = var.location
  tags     = var.tags
}
module "terraform_backend_prod" {
  source  = "Azure/avm-res-storage-storageaccount/azurerm"
  version = "0.2.0"

  enable_telemetry              = false
  account_replication_type      = "RAGRS" 
  account_tier                  = "Standard"
  account_kind                  = "StorageV2"
  access_tier                   = "Hot" 
  resource_group_name           = azurerm_resource_group.rg_sa.name
  location                      = azurerm_resource_group.rg_sa.location
  name                          = var.storage_account_name
  min_tls_version               = "TLS1_2"
  shared_access_key_enabled     = true
  public_network_access_enabled = true
  managed_identities = {
    system_assigned            = true
  }

  blob_properties = {
    versioning_enabled            = true
    change_feed_enabled           = true
    change_feed_retention_in_days = 90
    last_access_time_enabled      = true
    permanent_delete_enabled      = false

    delete_retention_policy = {
      days = 30
    }
    container_delete_retention_policy = {
      days = 30
    }
  }

  sas_policy = { // 2h token for CICD
    expiration_action = "Log"
    expiration_period = "00.02:00:00"
  }

  network_rules = {
    bypass                     = ["Logging", "Metrics", "AzureServices"]
    default_action             = "Deny"
    virtual_network_subnet_ids = [
      sub_id1,
      sub_id2
    ]
  }

  containers = {
    tfstate = {
      name                  = "tfstate"
      container_access_type = "private"
    }
  }

  tags = var.tags

  role_assignments = {
    role_assignment_1 = {
      role_definition_id_or_name       = "Owner" 
      principal_id                     = data.azurerm_client_config.current.object_id
      skip_service_principal_aad_check = false
    },
    role_assignment_4 = {
      role_definition_id_or_name       = "Storage Blob Data Contributor"
      principal_id                     = data.azurerm_client_config.current.object_id
      skip_service_principal_aad_check = false
    },
  }

  private_endpoints = {
    pe1 = {
      name                            = "pe-${var.storage_account_name}-001"
      subnet_resource_id              = module.vnet_prod.subnets[var.subnet1].id
      subresource_name                = "blob"
      private_dns_zone_resource_ids   = [azurerm_private_dns_zone.this.id]
      private_service_connection_name = "psc-${var.storage_account_name}-001"
      network_interface_name          = "nic-pe-${var.storage_account_name}-001"
      inherit_tags                    = true
      inherit_lock                    = false
      role_assignments = {
        role_assignment_1 = {
          role_definition_id_or_name = "Contributor"
          principal_id               = data.azurerm_client_config.current.object_id
        }
      }
    }
  }

  depends_on = [azurerm_resource_group.rg_sa, module.vnet_prod]
}
module "vnet_prod" {
  source  = "Azure/avm-res-network-virtualnetwork/azurerm"
  version = "0.1.4"

  enable_telemetry              = false
  name                          = var.vnet_prod_name
  location                      = azurerm_resource_group.rg_sa.location
  resource_group_name           = azurerm_resource_group.rg_sa.name
  virtual_network_address_space = var.vnet_prod_range
  virtual_network_dns_servers = {
    dns_servers = var.dns_servers
  }
  subnets = {
    "${var.subnet1}" = {
      address_prefixes  = var.subnet1_address_prefixes
      service_endpoints = ["Microsoft.Storage","Microsoft.KeyVault"]
    }
  }
  tags       = var.tags
  depends_on = [azurerm_resource_group.rg_sa]
}

module "nsg" {
  source              = "Azure/avm-res-network-networksecuritygroup/azurerm"
  version             = "0.2.0"

  enable_telemetry    = false
  resource_group_name = azurerm_resource_group.rg_sa.name
  name                = module.naming.network_security_group.name_unique
  location            = var.location
  security_rules      = var.rules
}

resource "azurerm_subnet_network_security_group_association" "private" {
  network_security_group_id = module.nsg.resource.id
  subnet_id                 = module.vnet_prod.subnets[var.subnet1].id
}

resource "azurerm_private_dns_zone" "this" {
  name                = "privatelink.blob.core.windows.net"
  resource_group_name = azurerm_resource_group.rg_sa.name
  tags                = var.tags
}

resource "azurerm_private_dns_zone" "vault" {
  name                = "privatelink.vaultcore.azure.net"
  resource_group_name = azurerm_resource_group.rg_sa.name
  tags                = var.tags
}

resource "azurerm_private_dns_zone_virtual_network_link" "private_links" {
  name                  = "${module.vnet_prod.vnet_resource.name}-link"
  private_dns_zone_name = azurerm_private_dns_zone.this.name
  resource_group_name   = azurerm_resource_group.rg_sa.name
  virtual_network_id    = module.vnet_prod.virtual_network_id
}

// forced to create this to resolve the private IP
resource "azurerm_private_dns_zone_virtual_network_link" "private_links_hub" {
  name                  = "${module.vnet_prod.vnet_resource.name}-hub-link"
  private_dns_zone_name = azurerm_private_dns_zone.this.name
  resource_group_name   = azurerm_resource_group.rg_sa.name
  virtual_network_id    = "vnet_id"
}

resource "azurerm_private_dns_zone_virtual_network_link" "private_links-vault" {
  name                  = "${module.vnet_prod.vnet_resource.name}-vault-link"
  private_dns_zone_name = azurerm_private_dns_zone.vault.name
  resource_group_name   = azurerm_resource_group.rg_sa.name
  virtual_network_id    = module.vnet_prod.virtual_network_id
}

resource "azurerm_private_dns_zone_virtual_network_link" "private_links_vault-hub" {
  name                  = "${module.vnet_prod.vnet_resource.name}-vault-hub-link"
  private_dns_zone_name = azurerm_private_dns_zone.vault.name
  resource_group_name   = azurerm_resource_group.rg_sa.name
  virtual_network_id    = "vnet_id"
}

resource "azurerm_route_table" "this" {
  resource_group_name = azurerm_resource_group.rg_sa.name
  location            = azurerm_resource_group.rg_sa.location
  name                = "rt_tfbackend_to_fw"
  disable_bgp_route_propagation = true
}
resource "azurerm_route" "all" {
  name                   = "all_to_fw"
  next_hop_type          = "VirtualAppliance"
  next_hop_in_ip_address = "172.16.10.240"
  address_prefix         = "0.0.0.0/0"
  resource_group_name    = azurerm_resource_group.rg_sa.name
  route_table_name       = azurerm_route_table.this.name
}

resource "azurerm_subnet_route_table_association" "this" {
  subnet_id      = module.vnet_prod.subnets[var.subnet1].id
  route_table_id = azurerm_route_table.this.id
}

[gevraud]

I guess it comes from this https://github.com/Azure/terraform-azurerm-avm-res-storage-storageaccount/blob/v0.2.0/main.tf#L325, which not used anymore in the newest versions

[chinthakaru]

@gevraud I attempted to execute your example in my environment, but I did not observe any configuration drift. Do you experience the same behavior even after deleting the .terraform directory?

[gevraud]

I have already tested that and it didn't work.

Did you test network_rules with version 0.2.0 and then upgrade to 0.2.4 ?

As network_rules is still in the module, I expected it to be deleted and recreated . But it's just deleting

[gevraud]

To make another test.

I removed virtual_network_subnet_ids and make the plan again

  # module.terraform_backend_prod.azurerm_storage_account.this will be updated in-place
  ~ resource "azurerm_storage_account" "this" {
        id                                 = "/subscriptions/xxxx/resourceGroups/rg/providers/Microsoft.Storage/storageAccounts/sa"
        name                               = "sa"
        tags                               = {
            "deployed_by" = "Terraform"
        }
        # (97 unchanged attributes hidden)

      ~ network_rules {
          ~ virtual_network_subnet_ids = [
              - "/subscriptions/xxxx/resourceGroups/rg/providers/Microsoft.Network/virtualNetworks/vnet/subnets/sub1",
              - "/subscriptions/xxxx/resourceGroups/rg/providers/Microsoft.Network/virtualNetworks/vnet/subnets/sub2",
            ]
            # (3 unchanged attributes hidden)
        }

        # (5 unchanged blocks hidden)
    }

  # module.terraform_backend_prod.azurerm_storage_account_network_rules.this[0] will be destroyed
  # (because azurerm_storage_account_network_rules.this is not in configuration)
  - resource "azurerm_storage_account_network_rules" "this" {
      - bypass                     = [
          - "AzureServices",
          - "Logging",
          - "Metrics",
        ] -> null
      - default_action             = "Deny" -> null
      - id                         = "/subscriptions/xxxx/resourceGroups/rg-/providers/Microsoft.Storage/storageAccounts/sa" -> null
      - ip_rules                   = [] -> null
      - storage_account_id         = "/subscriptions/xxx/resourceGroups/rg/providers/Microsoft.Storage/storageAccounts/sa" -> null
      - virtual_network_subnet_ids = [
              - "/subscriptions/xxxx/resourceGroups/rg/providers/Microsoft.Network/virtualNetworks/vnet/subnets/sub1",
              - "/subscriptions/xxxx/resourceGroups/rg/providers/Microsoft.Network/virtualNetworks/vnet/subnets/sub2",
        ] -> null
    }

Plan: 0 to add, 1 to change, 1 to destroy.

It seems the network_rules are still there, even if it's deleting the old azurerm_storage_account_network_rules

[gevraud]

yes the terraform state list command shows both resources

module.terraform_backend_prod.azurerm_storage_account.this module.terraform_backend_prod.azurerm_storage_account_network_rules.this[0]

terraform state show of module.terraform_backend_prod.azurerm_storage_account.this displays the network_rules

network_rules {
    bypass                     = [
        "AzureServices",
        "Logging",
        "Metrics",
    ]
    default_action             = "Deny"
    ip_rules                   = []
    virtual_network_subnet_ids = [
        "sub1",
        "sub2",
    ]
}

[chinthakaru]

@gevraud Thanks for updating the details. Please can you add following section to config and see if the error goes away ?

removed { from = module.terraform_backend_prod.azurerm_storage_account.this.azurerm_storage_account_network_rules.this[0] lifecycle { destroy = false }}

[gevraud]

it doesn't work

│ Error: Resource instance keys not allowed
│
│   on avm_storage_account.tf line 97, in removed:
│   97:   from = module.terraform_backend_prod.azurerm_storage_account.this.azurerm_storage_account_network_rules.this[0]
│
│ Resource address must be a resource (e.g. "test_instance.foo"), not a resource instance (e.g. "test_instance.foo[1]").

you maybe meant

removed {  
  from = module.terraform_backend_prod.azurerm_storage_account_network_rules.this  # from terraform state list
  lifecycle { 
    destroy = false  
    }
}

which gives :

Terraform will perform the following actions:

 # module.terraform_backend_prod.azurerm_storage_account_network_rules.this[0] will no longer be managed by Terraform, but will not be destroyed
 # (destroy = false is set in the configuration)
 . resource "azurerm_storage_account_network_rules" "this" {
        id                         = "/subscriptions/xxx/resourceGroups/rg/providers/Microsoft.Storage/storageAccounts/sa"
        # (5 unchanged attributes hidden)
    }

Plan: 0 to add, 0 to change, 0 to destroy.

[chinthakaru]

@gevraud Thanks for the update . This works. I'll add this block to module and it will avoid the drift in future.

[gevraud]

worked.

Thx for help