Cosmos Db Account and Attached Private Endpoint Cannot be Destroyed

jenny-curry commented 3 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) Version

Terraform v0.14.6 AzureRM v2.55.0

Affected Resource(s)

azurerm_cosmosdb_account

Terraform Configuration Files

locals { ips = join(",", module.ips.egress_ips[*].cidr) }

resource "azurerm_cosmosdb_account" "cosmosdb" { name = var.cosmos_db_account_name location = var.location resource_group_name = var.resource_group_name offer_type = var.offer_type kind = var.kind enable_automatic_failover = true ip_range_filter = local.ips tags = merge(var.tags, {Automation="Terraform"})

consistency_policy { consistency_level = var.consistency_level max_interval_in_seconds = var.max_interval_in_seconds max_staleness_prefix = var.max_staleness_prefix }

geo_location { location = var.failover_location failover_priority = var.failover_priority }

geo_location { prefix = var.cosmos_db_account_name location = var.location failover_priority = var.geo_failover_priority } }

resource "azurerm_private_endpoint" "endpoint" { depends_on = [azurerm_cosmosdb_account.cosmosdb] name = azurerm_cosmosdb_account.cosmosdb.name location = var.location resource_group_name = var.resource_group_name subnet_id = data.azurerm_subnet.subnet[0].id

private_service_connection { name = "privateserviceconnection" private_connection_resource_id = azurerm_cosmosdb_account.cosmosdb.id is_manual_connection = false subresource_names = ["Sql"] } private_dns_zone_group { name = "dns-zone-group" private_dns_zone_ids = [data.azurerm_private_dns_zone.zone.id] } }

Debug Output

https://gist.github.com/jmeadowcroft/497227c207c821355f656ebea4fe49f2

Expected Behaviour

Terraform destroy should have completed

Actual Behaviour

Error on destroy of private endpoint attached to Cosmos Db Account. This is executed immediately after the creation of the private endpoint and Cosmos Db Account through Terraform apply. See below the process of the destroy:

module.cosmosdbaccount.module.ips.null_resource.updateips: Destroying... [id=7032999081199589177] module.cosmosdbaccount.module.ips.null_resource.updateips: Destruction complete after 0s module.cosmosdbaccount.azurerm_private_endpoint.endpoint[0]: Destroying... [id=/subscriptions/e52a03e7-da01-4e65-923e-ac0dcedcc4ac/resourceGroups/RG-DevTest-AKS/providers/Microsoft.Network/privateEndpoints/31580138] module.cosmosdbaccount.azurerm_private_endpoint.endpoint[0]: Still destroying... [id=/subscriptions/e52a03e7-da01-4e65-923e-...soft.Network/privateEndpoints/31580138, 10s elapsed] module.cosmosdbaccount.azurerm_private_endpoint.endpoint[0]: Still destroying... [id=/subscriptions/e52a03e7-da01-4e65-923e-...soft.Network/privateEndpoints/31580138, 20s elapsed]

Error: waiting for deletion of Private Endpoint "31580138" (Resource Group "RG-DevTest-AKS"): Code="PreconditionFailed" Message="Call to Microsoft.DocumentDB/databaseAccounts failed." Details=[] InnerError={"error":"Source: Nrp.Frontend.ClientCommon. Microsoft.WindowsAzure.Networking.Nrp.Frontend.Common.OperationException: There is already an operation in progress which requires exclusive lock on this service 31580138. Please retry the operation after sometime.\r\nActivityId: 9b30811e-8ab4-4941-b1a1-36da86b9e326, Microsoft.Azure.Documents.Common/2.11.0\r\nCode: PreconditionFailed\r\nInnerError: \r\n at Microsoft.WindowsAzure.Networking.Nrp.Frontend.Client.Common.OperationChannel.\u003cReadResult\u003ed34`2.MoveNext() in X:\\bt\\1079313\\repo\\src\\sources\\Frontend\\FrontendClientCommon\\OperationChannel.cs:line 413\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at Microsoft.WindowsAzure.Networking.Nrp.Frontend.Client.Common.OperationChannel.\u003cExecuteOperation\u003ed302.MoveNext() in X:\\\\bt\\\\1079313\\\\repo\\\\src\\\\sources\\\\Frontend\\\\FrontendClientCommon\\\\OperationChannel.cs:line 0\\r\\n--- End of stack trace from previous location where exception was thrown ---\\r\\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\\r\\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\\r\\n at Microsoft.WindowsAzure.Networking.Nrp.Common.ArmAuthProviderFacade.PrivateLinkServiceRpFacade.\u003cDeletePrivateEndpointConnectionProxy\u003ed__15.MoveNext() in X:\\\\bt\\\\1079313\\\\repo\\\\src\\\\sources\\\\Common\\\\ArmAuthProviderFacade\\\\PrivateLinkServiceRpFacade.cs:line 124\\r\\n--- End of stack trace from previous location where exception was thrown ---\\r\\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\\r\\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\\r\\n at Microsoft.WindowsAzure.Networking.Nrp.Frontend.Operations.Csm.DeletePrivateEndpointOperation.\u003cCallDeleteOnFirstPartyAsync\u003ed__20.MoveNext() in X:\\\\bt\\\\1079313\\\\repo\\\\src\\\\sources\\\\Frontend\\\\FrontEndOperations\\\\Csm\\\\DeletePrivateEndpointOperation.cs:line 531\\r\\n--- End of stack trace from previous location where exception was thrown ---\\r\\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\\r\\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\\r\\n at Microsoft.WindowsAzure.Networking.Nrp.Frontend.Operations.Csm.DeletePrivateEndpointOperation.\u003cRunBackgroundTaskAsync\u003ed__15.MoveNext() in X:\\\\bt\\\\1079313\\\\repo\\\\src\\\\sources\\\\Frontend\\\\FrontEndOperations\\\\Csm\\\\DeletePrivateEndpointOperation.cs:line 110\\r\\n--- End of stack trace from previous location where exception was thrown ---\\r\\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\\r\\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\\r\\n at Microsoft.WindowsAzure.Networking.Nrp.Frontend.Operations.Operation1.\u003cStartRunBackgroundTaskAsync\u003ed__139.MoveNext() in X:\\bt\\1079313\\repo\\src\\sources\\Frontend\\FrontEndOperations\\Operation.cs:line 419"}

Steps to Reproduce

Terraform Apply
Terraform Destroy

jenny-curry commented 3 years ago

@favoretti, not sure if anyone has looked at this issue, but is currently causing a lot of problems in a pipeline we have. When can we have someone take a look at this?

favoretti commented 3 years ago

I can try and see if I can reproduce it some time next week, but not earlier than that unfortunately.

rcskosir commented 1 year ago

Thanks for opening this issue. This was a problem in the 2.x version of the provider which is no longer actively maintained. If this is still an issue with the 3.x version of the provider please do let us know by opening a new issue, thanks!

github-actions[bot] commented 6 months ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

hashicorp / terraform-provider-azurerm