Storage Account Backup Instance create fails when waiting for creation

ksommoxy commented 2 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) Version

Terraform v1.1.7

provider registry.terraform.io/hashicorp/azurerm v2.99.0

Affected Resource(s)

azurerm_data_protection_backup_instance_blob_storage

Terraform Configuration Files

terraform {
  required_providers {
    azurerm = {
      source                = "hashicorp/azurerm"
      version               = ">= 2.99"
      configuration_aliases = [azurerm, azurerm.subscription1, azurerm.subscription2]
    }
  }
}

# Created in subscription 1
resource "azurerm_data_protection_backup_vault" "vault" {
  provider            = azurerm.subscription1
  name                = "backup-vault-name"
  resource_group_name = "backup-vault-resource-group"
  ...
}

# Created in subscription 1
resource "azurerm_data_protection_backup_policy_blob_storage" "policy" {
  provider           = azurerm.subscription1    
  name               = "backup-vault-resource-group"
  vault_id           = "backup-vault-name"
  ...
}

# Created in subscription 2
resource "azurerm_storage_account" "storage_account" {
  provider                  = azurerm.subscription2
  name                      = "storageaccountname"
  resource_group_name       = "storage-resource-group"
  ...
}

# Created in subscription 1

resource "azurerm_data_protection_backup_instance_blob_storage" "backup_instance" {
  provider           = azurerm.subscription1
  name               = "bup-storageaccountname"
  vault_id           = azurerm_data_protection_backup_vault.vault.id
  storage_account_id = azurerm_storage_account.storage_account.id
  backup_policy_id   = azurerm_data_protection_backup_policy_blob_storage.policy.id
}

Debug Output

azurerm_data_protection_backup_instance_blob_storage.backup_instance: Still creating... [3m40s elapsed]
Error: waiting for creation/update of the DataProtection BackupInstance ("Backup Instance: (Name \"bup-storageaccountname\" / Backup Vault Name \"backup-vault-name\" / Resource Group \"backup-vault-resource-group\")"): Future#WaitForCompletion: the number of retries has been exceeded: StatusCode=403 -- Original Error: Code="AuthorizationFailed" Message="The client '<SP>' with object id '<SP>' does not have authorization to perform action 'Microsoft.DataProtection/locations/operationStatus/read' over scope '/subscriptions/<subscription1>' or the scope is invalid. If access was recently granted, please refresh your credentials."

Panic Output

N/A

Expected Behaviour

Apply returns successfully without error.

Actual Behaviour

The backup instance is created within the Vault, but an error is thrown when waiting/querying to verify the creation of the backup instance.

Steps to Reproduce

terraform apply

Important Factoids

Authenticating as a Service Principal (SP)
SP is assigned the Backup Contributor role on the Backup Vault's resource group
Backup Contributor contains the Microsoft.DataProtection/locations/operationStatus/read permission
Assigning the SP the role Backup Reader (which contains the operation status read permission) at the subscription level/scope allows the configuration to be applied successfully.
It appears when the provider does the check post-creation it is using the "wrong" scope (at the subscription level) instead of a more specific check at the Vault or resource group level. The error only specifies a subscription scope instead of the full scope to the backup vault.

References

aasier commented 2 years ago

Hello there!

I´m having a similar issue with azurerm_data_protection_backup_instance_blob_storage

azurerm_data_protection_backup_instance_blob_storage.sfo_backup_policy: Still creating... [2m20s elapsed]
╷
│ Error: waiting for BackupInstance("Backup Instance: (Name \"xxxx-xxxx-storage\" / Backup Vault Name \"xxxx-rvt-weu1-dev-000\" / Resource Group \"xxxx-xxxx-rsg-weu1-dev\")") policy protection to be completed: unexpected state 'ProtectionError', wanted target 'ProtectionConfigured'. last error: %!s(<nil>)
│ 
│   with azurerm_data_protection_backup_instance_blob_storage.sfo_backup_policy,
│   on backup.tf line 37, in resource "azurerm_data_protection_backup_instance_blob_storage" "sfo_backup_policy":
│   37: resource "azurerm_data_protection_backup_instance_blob_storage" "sfo_backup_policy" {

Terraform v1.1.9
on darwin_amd64
+ **provider registry.terraform.io/hashicorp/azurerm v3.16.0**
+ provider registry.terraform.io/hashicorp/external v2.1.0
+ provider registry.terraform.io/hashicorp/local v2.1.0
+ provider registry.terraform.io/hashicorp/null v3.1.0
+ provider registry.terraform.io/hashicorp/random v3.1.0

Your version of Terraform is out of date! The latest version
is 1.2.6. You can update by downloading from https://www.terraform.io/downloads.html

If I run again, terraform said that the resource needs to be imported into the state.

│ Error: A resource with the ID "/subscriptions/xxxxxx-e132-4dcc-9044-62535fcc159f/resourceGroups/xxx-xxxx-rsg-weu1-dev/providers/Microsoft.DataProtection/backupVaults/xxxx-rvt-weu1-dev-000/backupInstances/xxxx-xxxx-storage" already exists - to be managed via Terraform this resource needs to be imported into the State. Please see the resource documentation for "azurerm_data_protection_backup_instance_blob_storage" for more information.
│ 
│   with azurerm_data_protection_backup_instance_blob_storage.sfo_backup_policy,
│   on backup.tf line 37, in resource "azurerm_data_protection_backup_instance_blob_storage" "sfo_backup_policy":
│   37: resource "azurerm_data_protection_backup_instance_blob_storage" "sfo_backup_policy" {
│

And finally, if I re-run terraform it finishes OK.

aaroncommify commented 2 years ago

@aasier : I suspect you have a different issue to ksommoxy, I recently ran into the same as yourself when my backup vault didn't have the Storage Account Backup Contributor role assigned to it's identity, this prevents the backup instance from being created successfully, may not be your exact issue but you're certainly getting different to the original post.

@ksommoxy : You don't list the role assignment for the backup vault in your post so might be something you're missing although I'm not sure that it will fix your issue.

Here is an example for applying the role assignment:


resource "azurerm_role_assignment" "example" {
  scope                = azurerm_storage_account.example.id
  role_definition_name = "Storage Account Backup Contributor"
  principal_id         = azurerm_data_protection_backup_vault.example.identity[0].principal_id
}

a-nldisr commented 1 year ago

Ran into this issue while applying an azurerm_data_protection_backup_instance_disk.

Terraform version 1.3.6 Azurerm Provider version 3.26.0

After running again the object seems to be in azure but no in the state, exact same error.

-edit- We traced it back to the system managed identity

slzmruepp commented 1 year ago

I have the identical issue with provider version version = ">= 3.45.0" The role assigment worked. I can also not delete the backup vault from the gui, it errors with: Cannot delete the vault as there are existing instances or policies. They show up only after really a long time. Then I could delete them from the gui and rerun the failed terraform pipeline job. Did the second run, also fail.

This is the error: Error: waiting for BackupInstance("Backup Instance (Subscription: \"XXX\"\nResource Group Name: \"XXX\"\nBackup Vault Name: \"XXX\"\nBackup Instance Name: \"XXX\")") policy protection to be completed: unexpected state 'ProtectionError', wanted target 'ProtectionConfigured'. last error: %!s(<nil>)

atombravo commented 1 year ago

I'm hitting this error as well, on 3.46.0, when trying to enable backup on a storage account. From the GUI the "fix protection error" doesn't work, though attempting to delete the instance works the second time in the GUI.

Fwiw, I do have HNS enabled on the storage account.

vetlekise commented 9 months ago

@atombravo Did you figure this one out? I am struggling with the exact same issue. I am currently on v3.84 and the issue is still present. Going to try updating to v3.89 to see if it is still present..

Edit 1: Updating to v.3.89 did not fix it Edit 2: If anyone else are having issues, here's the fix: As shown in the documentation for azurerm here, you'll need to add a role assignment for the system managed identity on the backup vault and give it the role Storage Account Backup Contributor on the storage account.

resource "azurerm_role_assignment" "example" {
  scope                = azurerm_storage_account.example.id
  role_definition_name = "Storage Account Backup Contributor"
  principal_id         = azurerm_data_protection_backup_vault.example.identity[0].principal_id
}

humpalu commented 3 months ago

@atombravo I have the same issue but it seems like the HNS enabled storage accounts are not supported. https://learn.microsoft.com/en-us/azure/backup/blob-backup-support-matrix?tabs=vaulted-backup#limitations

hashicorp / terraform-provider-azurerm