hashicorp / terraform-provider-azurerm

Terraform provider for Azure Resource Manager
https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs
Mozilla Public License 2.0
4.6k stars 4.65k forks source link

Terraform destroy error - "Service returned an error. Status=409 Code="ScopeLocked"" #17478

Open Pinaki-1 opened 2 years ago

Pinaki-1 commented 2 years ago

Is there an existing issue for this?

Community Note

Terraform Version

1.2.3

AzureRM Provider Version

3.0.1

Affected Resource(s)/Data Source(s)

azurerm_backup_container_storage_account

Terraform Configuration Files

resource "azurerm_backup_container_storage_account" "this" {
  count = local.is_backup_enabled ? 1 : 0

  resource_group_name = var.recovery_service_vault.resource_group_name
  recovery_vault_name = var.recovery_service_vault.name
  storage_account_id  = azurerm_storage_account.resource.id

  depends_on = [
    azurerm_storage_account.resource,
    azurerm_storage_share.this,
  ]
}

Debug Output/Panic Output

module.filestorage.azurerm_backup_container_storage_account.this[0]: Destruction complete after 1m14s
[..truncated..]
module.filestorage.azurerm_storage_share.this["abc"]: Destruction complete after 0s
module.filestorage.azurerm_storage_share.this["def"]: Destruction complete after 1s
module.recovery_service_vault.azurerm_recovery_services_vault.this: Destruction complete after 2s
╷
│ Error: authorization.RoleAssignmentsClient#Delete: Failure responding to request: StatusCode=409 -- Original Error: autorest/azure: Service returned an error. Status=409 Code="ScopeLocked" Message="The scope '/subscriptions/***/resourceGroups/***/providers/Microsoft.Storage/storageAccounts/***/providers/Microsoft.Authorization/roleAssignments/***' cannot perform delete operation because following scope(s) are locked: '/subscriptions/***/resourcegroups***/providers/microsoft.storage/storageAccounts/***'. Please remove the lock and try again."
│ 
│ 
╵
╷
│ Error: authorization.RoleAssignmentsClient#Delete: Failure responding to request: StatusCode=409 -- Original Error: autorest/azure: Service returned an error. Status=409 Code="ScopeLocked" Message="The scope '/subscriptions/***/resourceGroups/***/providers/Microsoft.Storage/storageAccounts/***/providers/Microsoft.Authorization/roleAssignments/***' cannot perform delete operation because following scope(s) are locked: '/subscriptions/***/resourcegroups/***/providers/microsoft.storage/storageAccounts/***'. Please remove the lock and try again."
│

Expected Behaviour

Terraform should be able to destroy all dependent components(like RBAC, and private endpoints) and finally be able to delete the storage account. Because post destroying "azurerm_backup_container_storage_account", resource lock was also got deleted.

Actual Behaviour

Post destruction of "azurerm_backup_container_storage_account" and file shares terraform unable to perform :

  1. authorization.RoleAssignmentsClient#Delete
  2. deleting Monitor Diagnostics Setting
  3. deletion of Private Endpoint

reason - Service returned an error. Status=409 Code="ScopeLocked"

Steps to Reproduce

No response

Important Factoids

No response

References

https://github.com/hashicorp/terraform-provider-azurerm/issues/5025

sinbai commented 2 years ago

@pinaki-accenture thanks for opening this issue here. Unfortunately, I could not reproduce this issue with the following tf configuration and steps. Could you reproduce with it? Could you provide the full tf configuration and reproduce steps to help reproduce and troubleshoot?

Terraform and AzureRM Provider Version: image

Step1: Create resource azurerm_backup_container_storage_account with the following tf configuration.

terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "3.0.1"
    }
  }
}

provider "azurerm" {
  features {
  }
}

resource "azurerm_resource_group" "test" {
  name     = "RG-example"
  location = "eastus"
}

resource "azurerm_recovery_services_vault" "testvlt" {
  name                = "vault-example"
  location            = azurerm_resource_group.test.location
  resource_group_name = azurerm_resource_group.test.name
  sku                 = "Standard"

  soft_delete_enabled = true
}

resource "azurerm_storage_account" "test" {
  name                = "unlikely23exst"
  resource_group_name = azurerm_resource_group.test.name

  location                 = azurerm_resource_group.test.location
  account_tier             = "Standard"
  account_replication_type = "LRS"
}

resource "azurerm_storage_share" "test" {
  name                 = "ss-example"
  storage_account_name = "${azurerm_storage_account.test.name}"
  quota                = 1
  metadata             = {}

  lifecycle {
    ignore_changes = [metadata] // Ignore changes Azure Backup makes to the metadata
  }
}

resource "azurerm_backup_container_storage_account" "test" {
  resource_group_name = azurerm_resource_group.test.name
  recovery_vault_name = azurerm_recovery_services_vault.testvlt.name
  storage_account_id  = azurerm_storage_account.test.id
   depends_on = [
    azurerm_storage_share.test,
  ]
}

Step2: Run Terraform Destroy => Everything works smoothly.

Actual: All resources could be destroyed .

Pinaki-1 commented 2 years ago

Hello @sinbai, I am facing a problem if I associate any diagnostic settings, private endpoint, or RBAC. I just updated your example with RBAC and the diagnostic settings: main.tf.zip

Step 2 : init > apply > destroy

Actual:

azurerm_recovery_services_vault.testvlt: Destroying... [id=/subscriptions/d4c5fd09-c075-47f9-87f6-027c6d3b7fe5/resourceGroups/RG-example/providers/Microsoft.RecoveryServices/vaults/vault-example]
azurerm_storage_share.test: Destruction complete after 4s
azurerm_recovery_services_vault.testvlt: Destruction complete after 10s
╷
│ Error: deleting Monitor Diagnostics Setting "example" for Resource "subscriptions/d4c5fd09-c075-47f9-87f6-027c6d3b7fe5/resourceGroups/RG-example/providers/Microsoft.Storage/storageAccounts/unlikely23exst/fileServices/default": insigh
ts.DiagnosticSettingsClient#Delete: Failure responding to request: StatusCode=409 -- Original Error: autorest/azure: Service returned an error. Status=409 Code="ScopeLocked" Message="The scope '/subscriptions/d4c5fd09-c075-47f9-87f6-02
7c6d3b7fe5/resourceGroups/RG-example/providers/Microsoft.Storage/storageAccounts/unlikely23exst/fileServices/default/providers/Microsoft.Insights/diagnosticSettings/example' cannot perform delete operation because following scope(s) are locked: '/subscriptions/d4c5fd09-c075-47f9-87f6-027c6d3b7fe5/resourcegroups/rg-example/providers/microsoft.storage/storageAccounts/unlikely23exst'. Please remove the lock and try again."
│
│
╵
╷
│ Error: authorization.RoleAssignmentsClient#Delete: Failure responding to request: StatusCode=409 -- Original Error: autorest/azure: Service returned an error. Status=409 Code="ScopeLocked" Message="The scope '/subscriptions/d4c5fd09-
c075-47f9-87f6-027c6d3b7fe5/resourceGroups/RG-example/providers/Microsoft.Storage/storageAccounts/unlikely23exst/providers/Microsoft.Authorization/roleAssignments/d6ab9f32-d2bd-ac5a-9341-8676549c100b' cannot perform delete operation because following scope(s) are locked: '/subscriptions/d4c5fd09-c075-47f9-87f6-027c6d3b7fe5/resourcegroups/rg-example/providers/microsoft.storage/storageAccounts/unlikely23exst'. Please remove the lock and try again."
│
│
╵

PS: Lock was already removed but still terraform is throwing an error. And most importantly, if I run the destroy again it succeeds. But I am running some Terratests and all those fails if destruction does not happen in a single run. :(

Pinaki-1 commented 2 years ago

Hello Team, Kindly let me know if the updated example file giving you the same problem. Thanksm

Pinaki-1 commented 1 year ago

Tried with the latest Azurerm version 3.50.0 but still having the same issue.

Error: waiting for deletion of Private Endpoint "pep-snt001-sta001-soiaqwenj-eus1-d" (Resource Group "rsg-soiaqwenj-eus1-d-021"): Code="ScopeLocked" Message="Call to Microsoft.Storage/storageAccounts failed. Error message: The scope '/subscriptions/------------------/resourceGroups/rsg-soiaqwenj-eus1-d-021/providers/Microsoft.Storage/storageAccounts/stasoiaqwenjeus1d001/privateEndpointConnectionProxies/pep-snt001-sta001-soiaqwenj-eus1-d.1fb46a6b-62e2-4825-a67f-8b50b30fa8e6' cannot perform delete operation because following scope(s) are locked: '/subscriptions/------------------/resourcegroups/rsg-soiaqwenj-eus1-d-021/providers/microsoft.storage/storageAccounts/stasoiaqwenjeus1d001'. Please remove the lock and try again." Details=[] InnerError={"error":"Source: Nrp.Frontend.ClientCommon. Microsoft.WindowsAzure.Networking.Nrp.Frontend.Common.OperationException: The scope '/subscriptions/------------------/resourceGroups/rsg-soiaqwenj-eus1-d-021/providers/Microsoft.Storage/storageAccounts/stasoiaqwenjeus1d001/privateEndpointConnectionProxies/pep-snt001-sta001-soiaqwenj-eus1-d.1fb46a6b-62e2-4825-a67f-8b50b30fa8e6' cannot perform delete operation because following scope(s) are locked: '/subscriptions/------------------/resourcegroups/rsg-soiaqwenj-eus1-d-021/providers/microsoft.storage/storageAccounts/stasoiaqwenjeus1d001'. Please remove the lock and try again.\\r\\nCode: ScopeLocked\\r\\nInnerError: \\r\\n   at Microsoft.WindowsAzure.Networking.Nrp.Frontend.Client.Common.OperationChannel.\u003cReadResultAsync\u003ed__35`2.MoveNext() in X:\\\\bt\\\\1017528\\\\repo\\\\src\\\\sources\\\\Frontend\\\\FrontendClientCommon\\\\OperationChannel.cs:line 448\\r\\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\\r\\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\\r\\n   at Microsoft.WindowsAzure.Networking.Nrp.Frontend.Client.Common.OperationChannel.\u003cExecuteOperation\u003ed__31`2.MoveNext() in X:\\\\bt\\\\1017528\\\\repo\\\\src\\\\sources\\\\Frontend\\\\FrontendClientCommon\\\\OperationChannel.cs:line 0\\r\\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\\r\\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\\r\\n   at Microsoft.WindowsAzure.Networking.Nrp.Common.ArmAuthProviderFacade.PrivateLinkServiceRpFacade.\u003cDeletePrivateEndpointConnectionProxy\u003ed__14.MoveNext() in X:\\\\bt\\\\1017528\\\\repo\\\\src\\\\sources\\\\Common\\\\ArmAuthProviderFacade\\\\PrivateLinkServiceRpFacade.cs:line 122\\r\\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\\r\\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\\r\\n   at Microsoft.WindowsAzure.Networking.Nrp.Frontend.Operations.Csm.DeletePrivateEndpointOperation.\u003cCallDeleteOnFirstPartyAsync\u003ed__29.MoveNext() in X:\\\\bt\\\\1017528\\\\repo\\\\src\\\\sources\\\\Frontend\\\\FrontEndOperations\\\\Csm\\\\DeletePrivateEndpointOperation.cs:line 661\\r\\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\\r\\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\\r\\n   at Microsoft.WindowsAzure.Networking.Nrp.Frontend.Operations.Csm.DeletePrivateEndpointOperation.\u003cRunBackgroundTaskAsync\u003ed__24.MoveNext() in X:\\\\bt\\\\1017528\\\\repo\\\\src\\\\sources\\\\Frontend\\\\FrontEndOperations\\\\Csm\\\\DeletePrivateEndpointOperation.cs:line 118\\r\\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\\r\\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\\r\\n   at Microsoft.WindowsAzure.Networking.Nrp.Frontend.Operations.Operation`1.\u003cStartRunBackgroundTaskAsync\u003ed__174.MoveNext() in X:\\\\bt\\\\1017528\\\\repo\\\\src\\\\sources\\\\Frontend\\\\FrontEndOperations\\\\Operation.cs:line 569"}
gyuvaraj10 commented 1 year ago

Is there any update on this issue?

Pinaki-1 commented 1 year ago

Hello Team, same happens if I use azurerm_data_protection_backup_instance_blob_storage for blob backup. Looks like there is some delay happening in releasing the resource lock and while destroying terraform is not waiting for that time. can you please check?

madheaduk commented 1 year ago

@Pinaki-1 How did you get past this ? As I have the same issue ... and I see you open this over a year ago...

Pinaki-1 commented 1 year ago

I couldn't able to find any workaround. Hence I skipped this option from my terratest testcases.

Netkracker commented 6 months ago

Issue is still here with azurerm 3.100.0 ;(

Netkracker commented 6 months ago

So, I played around a bit with Terraform's apply -targets and Go. Came up with this. Hopefully, this helps someone else :)

terraformOptionsPreTest builds the storage account and the infrastructure needed to enable backup (Backup vault, etc.) terraformOptions links the storage account to the backup vault and starts the backup with azurerm_backup_container_storage_account.

func TestWithSpecialDelay(t testing.T) { t.Log("Starting segmented Module test with special delay") terraformOptionsPreTest := terraform.WithDefaultRetryableErrors(t, &terraform.Options{ TerraformDir: ".", Targets: []string{ "random_string.testsuffix", "data.azurerm_subscription.current", "module.key_vault", "module.ba_vault", "module.storage_standard", }, }) terraformOptions := terraform.WithDefaultRetryableErrors(t, &terraform.Options{ TerraformDir: ".", Targets: []string{ "module.blob_backup", }, }) // Create storage account, backup vault and the link of the storage account to backup vault separately from each other. terraform.InitAndApply(t, terraformOptionsPreTest) // Idempotent fails. terraform.InitAndApply(t, terraformOptions) // Idempotent fails. // https://github.com/hashicorp/terraform-provider-azurerm/issues/17478 // Destroy the storage account account, backup vault and the link of the storage account to backup vault separately from each other to avoid the bug described in GitHub above. terraform.Destroy(t, terraformOptions) // // To be able to destroy in waves. time.Sleep(30 time.Second) terraform.Destroy(t, terraformOptionsPreTest) }