Open Pinaki-1 opened 2 years ago
@pinaki-accenture thanks for opening this issue here. Unfortunately, I could not reproduce this issue with the following tf configuration and steps. Could you reproduce with it? Could you provide the full tf configuration and reproduce steps to help reproduce and troubleshoot?
Terraform and AzureRM Provider Version:
Step1: Create resource azurerm_backup_container_storage_account with the following tf configuration.
terraform {
required_providers {
azurerm = {
source = "hashicorp/azurerm"
version = "3.0.1"
}
}
}
provider "azurerm" {
features {
}
}
resource "azurerm_resource_group" "test" {
name = "RG-example"
location = "eastus"
}
resource "azurerm_recovery_services_vault" "testvlt" {
name = "vault-example"
location = azurerm_resource_group.test.location
resource_group_name = azurerm_resource_group.test.name
sku = "Standard"
soft_delete_enabled = true
}
resource "azurerm_storage_account" "test" {
name = "unlikely23exst"
resource_group_name = azurerm_resource_group.test.name
location = azurerm_resource_group.test.location
account_tier = "Standard"
account_replication_type = "LRS"
}
resource "azurerm_storage_share" "test" {
name = "ss-example"
storage_account_name = "${azurerm_storage_account.test.name}"
quota = 1
metadata = {}
lifecycle {
ignore_changes = [metadata] // Ignore changes Azure Backup makes to the metadata
}
}
resource "azurerm_backup_container_storage_account" "test" {
resource_group_name = azurerm_resource_group.test.name
recovery_vault_name = azurerm_recovery_services_vault.testvlt.name
storage_account_id = azurerm_storage_account.test.id
depends_on = [
azurerm_storage_share.test,
]
}
Step2: Run Terraform Destroy
=> Everything works smoothly.
Actual: All resources could be destroyed .
Hello @sinbai, I am facing a problem if I associate any diagnostic settings, private endpoint, or RBAC. I just updated your example with RBAC and the diagnostic settings: main.tf.zip
Step 2 : init > apply > destroy
Actual:
azurerm_recovery_services_vault.testvlt: Destroying... [id=/subscriptions/d4c5fd09-c075-47f9-87f6-027c6d3b7fe5/resourceGroups/RG-example/providers/Microsoft.RecoveryServices/vaults/vault-example]
azurerm_storage_share.test: Destruction complete after 4s
azurerm_recovery_services_vault.testvlt: Destruction complete after 10s
╷
│ Error: deleting Monitor Diagnostics Setting "example" for Resource "subscriptions/d4c5fd09-c075-47f9-87f6-027c6d3b7fe5/resourceGroups/RG-example/providers/Microsoft.Storage/storageAccounts/unlikely23exst/fileServices/default": insigh
ts.DiagnosticSettingsClient#Delete: Failure responding to request: StatusCode=409 -- Original Error: autorest/azure: Service returned an error. Status=409 Code="ScopeLocked" Message="The scope '/subscriptions/d4c5fd09-c075-47f9-87f6-02
7c6d3b7fe5/resourceGroups/RG-example/providers/Microsoft.Storage/storageAccounts/unlikely23exst/fileServices/default/providers/Microsoft.Insights/diagnosticSettings/example' cannot perform delete operation because following scope(s) are locked: '/subscriptions/d4c5fd09-c075-47f9-87f6-027c6d3b7fe5/resourcegroups/rg-example/providers/microsoft.storage/storageAccounts/unlikely23exst'. Please remove the lock and try again."
│
│
╵
╷
│ Error: authorization.RoleAssignmentsClient#Delete: Failure responding to request: StatusCode=409 -- Original Error: autorest/azure: Service returned an error. Status=409 Code="ScopeLocked" Message="The scope '/subscriptions/d4c5fd09-
c075-47f9-87f6-027c6d3b7fe5/resourceGroups/RG-example/providers/Microsoft.Storage/storageAccounts/unlikely23exst/providers/Microsoft.Authorization/roleAssignments/d6ab9f32-d2bd-ac5a-9341-8676549c100b' cannot perform delete operation because following scope(s) are locked: '/subscriptions/d4c5fd09-c075-47f9-87f6-027c6d3b7fe5/resourcegroups/rg-example/providers/microsoft.storage/storageAccounts/unlikely23exst'. Please remove the lock and try again."
│
│
╵
PS: Lock was already removed but still terraform is throwing an error. And most importantly, if I run the destroy again it succeeds. But I am running some Terratests and all those fails if destruction does not happen in a single run. :(
Hello Team, Kindly let me know if the updated example file giving you the same problem. Thanksm
Tried with the latest Azurerm version 3.50.0 but still having the same issue.
Error: waiting for deletion of Private Endpoint "pep-snt001-sta001-soiaqwenj-eus1-d" (Resource Group "rsg-soiaqwenj-eus1-d-021"): Code="ScopeLocked" Message="Call to Microsoft.Storage/storageAccounts failed. Error message: The scope '/subscriptions/------------------/resourceGroups/rsg-soiaqwenj-eus1-d-021/providers/Microsoft.Storage/storageAccounts/stasoiaqwenjeus1d001/privateEndpointConnectionProxies/pep-snt001-sta001-soiaqwenj-eus1-d.1fb46a6b-62e2-4825-a67f-8b50b30fa8e6' cannot perform delete operation because following scope(s) are locked: '/subscriptions/------------------/resourcegroups/rsg-soiaqwenj-eus1-d-021/providers/microsoft.storage/storageAccounts/stasoiaqwenjeus1d001'. Please remove the lock and try again." Details=[] InnerError={"error":"Source: Nrp.Frontend.ClientCommon. Microsoft.WindowsAzure.Networking.Nrp.Frontend.Common.OperationException: The scope '/subscriptions/------------------/resourceGroups/rsg-soiaqwenj-eus1-d-021/providers/Microsoft.Storage/storageAccounts/stasoiaqwenjeus1d001/privateEndpointConnectionProxies/pep-snt001-sta001-soiaqwenj-eus1-d.1fb46a6b-62e2-4825-a67f-8b50b30fa8e6' cannot perform delete operation because following scope(s) are locked: '/subscriptions/------------------/resourcegroups/rsg-soiaqwenj-eus1-d-021/providers/microsoft.storage/storageAccounts/stasoiaqwenjeus1d001'. Please remove the lock and try again.\\r\\nCode: ScopeLocked\\r\\nInnerError: \\r\\n at Microsoft.WindowsAzure.Networking.Nrp.Frontend.Client.Common.OperationChannel.\u003cReadResultAsync\u003ed__35`2.MoveNext() in X:\\\\bt\\\\1017528\\\\repo\\\\src\\\\sources\\\\Frontend\\\\FrontendClientCommon\\\\OperationChannel.cs:line 448\\r\\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\\r\\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\\r\\n at Microsoft.WindowsAzure.Networking.Nrp.Frontend.Client.Common.OperationChannel.\u003cExecuteOperation\u003ed__31`2.MoveNext() in X:\\\\bt\\\\1017528\\\\repo\\\\src\\\\sources\\\\Frontend\\\\FrontendClientCommon\\\\OperationChannel.cs:line 0\\r\\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\\r\\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\\r\\n at Microsoft.WindowsAzure.Networking.Nrp.Common.ArmAuthProviderFacade.PrivateLinkServiceRpFacade.\u003cDeletePrivateEndpointConnectionProxy\u003ed__14.MoveNext() in X:\\\\bt\\\\1017528\\\\repo\\\\src\\\\sources\\\\Common\\\\ArmAuthProviderFacade\\\\PrivateLinkServiceRpFacade.cs:line 122\\r\\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\\r\\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\\r\\n at Microsoft.WindowsAzure.Networking.Nrp.Frontend.Operations.Csm.DeletePrivateEndpointOperation.\u003cCallDeleteOnFirstPartyAsync\u003ed__29.MoveNext() in X:\\\\bt\\\\1017528\\\\repo\\\\src\\\\sources\\\\Frontend\\\\FrontEndOperations\\\\Csm\\\\DeletePrivateEndpointOperation.cs:line 661\\r\\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\\r\\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\\r\\n at Microsoft.WindowsAzure.Networking.Nrp.Frontend.Operations.Csm.DeletePrivateEndpointOperation.\u003cRunBackgroundTaskAsync\u003ed__24.MoveNext() in X:\\\\bt\\\\1017528\\\\repo\\\\src\\\\sources\\\\Frontend\\\\FrontEndOperations\\\\Csm\\\\DeletePrivateEndpointOperation.cs:line 118\\r\\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\\r\\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\\r\\n at Microsoft.WindowsAzure.Networking.Nrp.Frontend.Operations.Operation`1.\u003cStartRunBackgroundTaskAsync\u003ed__174.MoveNext() in X:\\\\bt\\\\1017528\\\\repo\\\\src\\\\sources\\\\Frontend\\\\FrontEndOperations\\\\Operation.cs:line 569"}
Is there any update on this issue?
Hello Team, same happens if I use azurerm_data_protection_backup_instance_blob_storage
for blob backup. Looks like there is some delay happening in releasing the resource lock and while destroying terraform is not waiting for that time. can you please check?
@Pinaki-1 How did you get past this ? As I have the same issue ... and I see you open this over a year ago...
I couldn't able to find any workaround. Hence I skipped this option from my terratest testcases.
Issue is still here with azurerm 3.100.0 ;(
So, I played around a bit with Terraform's apply -targets
and Go. Came up with this. Hopefully, this helps someone else :)
terraformOptionsPreTest
builds the storage account and the infrastructure needed to enable backup (Backup vault, etc.)
terraformOptions
links the storage account to the backup vault and starts the backup with azurerm_backup_container_storage_account
.
func TestWithSpecialDelay(t testing.T) { t.Log("Starting segmented Module test with special delay") terraformOptionsPreTest := terraform.WithDefaultRetryableErrors(t, &terraform.Options{ TerraformDir: ".", Targets: []string{ "random_string.testsuffix", "data.azurerm_subscription.current", "module.key_vault", "module.ba_vault", "module.storage_standard", }, }) terraformOptions := terraform.WithDefaultRetryableErrors(t, &terraform.Options{ TerraformDir: ".", Targets: []string{ "module.blob_backup", }, }) // Create storage account, backup vault and the link of the storage account to backup vault separately from each other. terraform.InitAndApply(t, terraformOptionsPreTest) // Idempotent fails. terraform.InitAndApply(t, terraformOptions) // Idempotent fails. // https://github.com/hashicorp/terraform-provider-azurerm/issues/17478 // Destroy the storage account account, backup vault and the link of the storage account to backup vault separately from each other to avoid the bug described in GitHub above. terraform.Destroy(t, terraformOptions) // // To be able to destroy in waves. time.Sleep(30 time.Second) terraform.Destroy(t, terraformOptionsPreTest) }
Is there an existing issue for this?
Community Note
Terraform Version
1.2.3
AzureRM Provider Version
3.0.1
Affected Resource(s)/Data Source(s)
azurerm_backup_container_storage_account
Terraform Configuration Files
Debug Output/Panic Output
Expected Behaviour
Terraform should be able to destroy all dependent components(like RBAC, and private endpoints) and finally be able to delete the storage account. Because post destroying "azurerm_backup_container_storage_account", resource lock was also got deleted.
Actual Behaviour
Post destruction of "azurerm_backup_container_storage_account" and file shares terraform unable to perform :
reason - Service returned an error. Status=409 Code="ScopeLocked"
Steps to Reproduce
No response
Important Factoids
No response
References
https://github.com/hashicorp/terraform-provider-azurerm/issues/5025