search

PaloAltoNetworks / terraform-azurerm-vmseries-modules

Terraform Reusable Modules for VM-Series on Azure
https://registry.terraform.io/modules/PaloAltoNetworks/vmseries-modules/azurerm/latest
MIT License
49 stars 58 forks source link

Terraform Destroy errors #199

Closed kthix closed 1 year ago

kthix commented 1 year ago

Describe the bug

Using example: transit_vnet_dedicated Deployment was success. But when Terraform destroy is tested I do get errors.

Expected behavior

It should remove all deployed resources with the destroy command

Current behavior

I do think dependency issues are in play.

Possible solution

Steps to reproduce

terraform apply terraform destroy second time terraform destory => other errors

Screenshots

Context

Your Environment

Running it in Europ-West region.

On first run of terraform destroy:

module.inbound_vmseries["inboundfw00"].azurerm_virtual_machine.this: Still destroying... [id=/subscriptions/821d377f-76b2-4014-8c46-...mpute/virtualMachines/pantfinboundfw00, 2m40s elapsed]
module.outbound_vmseries["outboundfw01"].azurerm_virtual_machine.this: Still destroying... [id=/subscriptions/821d377f-76b2-4014-8c46-...pute/virtualMachines/pantfoutboundfw01, 2m40s elapsed]
module.inbound_vmseries["inboundfw01"].azurerm_public_ip.this["0"]: Still destroying... [id=/subscriptions/821d377f-76b2-4014-8c46-...ork/publicIPAddresses/inboundfw01-mgmt, 10s elapsed]
module.inbound_vmseries["inboundfw01"].azurerm_public_ip.this["0"]: Destruction complete after 11s
module.outbound_vmseries["outboundfw01"].azurerm_virtual_machine.this: Destruction complete after 2m42s
module.inbound_vmseries["inboundfw00"].azurerm_virtual_machine.this: Destruction complete after 2m43s
random_password.this: Destroying... [id=none]
random_password.this: Destruction complete after 0s
╷
│ Error: removing Network Security Group Association from Subnet: (Name "subnet-private" / Virtual Network Name "vnet-vmseries" / Resource Group "panw-rg"): network.SubnetsClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="ReferencedResourceNotProvisioned" Message="Cannot proceed with operation because resource /subscriptions/821d377f-76b2-4014-8c46-fd4468cfd8f0/resourceGroups/panw-rg/providers/Microsoft.Network/networkInterfaces/outboundfw00-private/ipConfigurations/primary used by resource /subscriptions/821d377f-76b2-4014-8c46-fd4468cfd8f0/resourceGroups/panw-rg/providers/Microsoft.Network/virtualNetworks/vnet-vmseries/subnets/subnet-private is not in Succeeded state. Resource is in Failed state and the last operation that updated/is updating the resource is PutNicOperation." Details=[]
│
│
╵
╷
│ Error: removing Network Security Group Association from Subnet: (Name "subnet-public" / Virtual Network Name "vnet-vmseries" / Resource Group "panw-rg"): network.SubnetsClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="ReferencedResourceNotProvisioned" Message="Cannot proceed with operation because resource /subscriptions/821d377f-76b2-4014-8c46-fd4468cfd8f0/resourceGroups/panw-rg/providers/Microsoft.Network/networkInterfaces/inboundfw00-public/ipConfigurations/primary used by resource /subscriptions/821d377f-76b2-4014-8c46-fd4468cfd8f0/resourceGroups/panw-rg/providers/Microsoft.Network/virtualNetworks/vnet-vmseries/subnets/subnet-public is not in Succeeded state. Resource is in Failed state and the last operation that updated/is updating the resource is PutNicOperation." Details=[]
│
│
╵
╷
│ Error: waiting for removal of Backend Address Pool Association for NIC "outboundfw00-private" (Resource Group "panw-rg"): Code="OperationNotAllowed" Message="Operation 'startTenantUpdate' is not allowed on VM 'pantfoutboundfw00' since the VM is marked for deletion. You can only retry the Delete operation (or wait for an ongoing one to complete)." Details=[]
│
│
╵
╷
│ Error: waiting for removal of Backend Address Pool Association for NIC "inboundfw00-public" (Resource Group "panw-rg"): Code="OperationNotAllowed" Message="Operation 'startTenantUpdate' is not allowed on VM 'pantfinboundfw00' since the VM is marked for deletion. You can only retry the Delete operation (or wait for an ongoing one to complete)." Details=[]
│
│
╵
╷
│ Error: waiting for removal of Backend Address Pool Association for NIC "outboundfw01-private" (Resource Group "panw-rg"): Code="OperationNotAllowed" Message="Operation 'startTenantUpdate' is not allowed on VM 'pantfoutboundfw01' since the VM is marked for deletion. You can only retry the Delete operation (or wait for an ongoing one to complete)." Details=[]
│
│
╵
╷
│ Error: removing Route Table Association from Subnet "subnet-private" (Virtual Network "vnet-vmseries" / Resource Group "panw-rg"): network.SubnetsClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="ReferencedResourceNotProvisioned" Message="Cannot proceed with operation because resource /subscriptions/821d377f-76b2-4014-8c46-fd4468cfd8f0/resourceGroups/panw-rg/providers/Microsoft.Network/networkInterfaces/outboundfw00-private/ipConfigurations/primary used by resource /subscriptions/821d377f-76b2-4014-8c46-fd4468cfd8f0/resourceGroups/panw-rg/providers/Microsoft.Network/virtualNetworks/vnet-vmseries/subnets/subnet-private is not in Succeeded state. Resource is in Failed state and the last operation that updated/is updating the resource is PutNicOperation." Details=[]
│
│
╵

second try: terraform destroy:
module.inbound_lb.azurerm_public_ip.this["frontend01"]: Destroying... [id=/subscriptions/821d377f-76b2-4014-8c46-fd4468cfd8f0/resourceGroups/panw-rg/providers/Microsoft.Network/publicIPAddresses/frontend01]
module.inbound_lb.azurerm_public_ip.this["frontend01"]: Still destroying... [id=/subscriptions/821d377f-76b2-4014-8c46-...t.Network/publicIPAddresses/frontend01, 10s elapsed]
module.inbound_lb.azurerm_public_ip.this["frontend01"]: Destruction complete after 10s
╷
│ Error: removing Network Security Group Association from Subnet: (Name "subnet-public" / Virtual Network Name "vnet-vmseries" / Resource Group "panw-rg"): network.SubnetsClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="ReferencedResourceNotProvisioned" Message="Cannot proceed with operation because resource /subscriptions/821d377f-76b2-4014-8c46-fd4468cfd8f0/resourceGroups/panw-rg/providers/Microsoft.Network/networkInterfaces/inboundfw00-public/ipConfigurations/primary used by resource /subscriptions/821d377f-76b2-4014-8c46-fd4468cfd8f0/resourceGroups/panw-rg/providers/Microsoft.Network/virtualNetworks/vnet-vmseries/subnets/subnet-public is not in Succeeded state. Resource is in Failed state and the last operation that updated/is updating the resource is PutNicOperation." Details=[]
│
│
╵
╷
│ Error: Invalid index
│
│   on ../../modules/vmseries/outputs.tf line 3, in output "mgmt_ip_address":
│    3:   value       = try(var.interfaces[0].create_public_ip, false) ? azurerm_public_ip.this[0].ip_address : azurerm_network_interface.this[0].ip_configuration[0].private_ip_address
│     ├────────────────
│     │ azurerm_public_ip.this is object with no attributes
│
│ The given key does not identify an element in this collection value. An object only supports looking up attributes by name, not by numeric index.
welcome-to-palo-alto-networks[bot] commented 1 year ago

:tada: Thanks for opening your first issue here! Welcome to the community!

pimielowski commented 1 year ago

@kthix Thanks for reporting a bug, could you please format a bit the output from terraform? It will help to read it :) image

kthix commented 1 year ago

Hi, hope it helps. I don't have the possibility to re-run it so hope the add code format helped on the existing post.

FoSix commented 1 year ago

@kthix the 1st time you run tf destroy you get an error message related to AzureRM and the fact that Azure reports an action back to terraform as finished successfully but the state of the resource on Azure side is not updated yet. There is no good fix for that.

The 2nd time you run destroy - I'm guessing that this happens when you use Terraform 1.3.x branch. We've seen that issue and it looks like it's related to Terraform directly although we were not able to pin point what is exactly causing this problem. For now, as a workaround, try to use 1.2 branch.

FoSix commented 1 year ago

tested with version 1.3.6, this looks like a problem with building dependencies. It happens not only for outputs, 1.3.6 produced this error while testing:

╷
│ Error: Unsupported attribute
│
│   on main.tf line 218, in module "outbound_vmseries":
│  218:       lb_backend_pool_id  = module.outbound_lb.backend_pool_id
│     ├────────────────
│     │ module.outbound_lb is object with 2 attributes
│
│ This object does not have an attribute named "backend_pool_id".

module.outbound_lb was destroyed in previous tf destroy run (which errored out due to problems with subnet status returned by AzureRM):

module.outbound_lb.azurerm_lb.lb: Destruction complete after 21s

To workaround that issue this line in vmseries module invocation:

      lb_backend_pool_id  = module.outbound_lb.backend_pool_id

was replaced with this one:

      lb_backend_pool_id  = try(module.outbound_lb.backend_pool_id, null)