Error 406 when attempting to create multiple kubernetes backup instances

Rick-healy commented 1 month ago

Is there an existing issue for this?

[X] I have searched the existing issues

Community Note

Please vote on this issue by adding a :thumbsup: reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave comments along the lines of "+1", "me too" or "any updates", they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment and review the contribution guide to help.

Terraform Version

4.241.26

AzureRM Provider Version

3.35.0

Affected Resource(s)/Data Source(s)

azurerm_data_protection_backup_instance_kubernetes_cluster

Terraform Configuration Files

Based on original content https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/data_protection_backup_instance_kubernetes_cluster ;

data "azurerm_client_config" "current" {}

resource "azurerm_resource_group" "example" {
  name     = "example"
  location = "West Europe"
}

resource "azurerm_resource_group" "snap" {
  name     = "example-snap"
  location = "West Europe"
}

resource "azurerm_data_protection_backup_vault" "example" {
  name                = "example"
  resource_group_name = azurerm_resource_group.example.name
  location            = azurerm_resource_group.example.location
  datastore_type      = "VaultStore"
  redundancy          = "LocallyRedundant"

  identity {
    type = "SystemAssigned"
  }
}

resource "azurerm_kubernetes_cluster" "example" {
  name                = "example"
  location            = azurerm_resource_group.example.location
  resource_group_name = azurerm_resource_group.example.name
  dns_prefix          = "dns"

  default_node_pool {
    name                   = "default"
    node_count             = 1
    vm_size                = "Standard_DS2_v2"
    enable_host_encryption = true
  }

  identity {
    type = "SystemAssigned"
  }
}

resource "azurerm_kubernetes_cluster_trusted_access_role_binding" "aks_cluster_trusted_access" {
  kubernetes_cluster_id = azurerm_kubernetes_cluster.example.id
  name                  = "example"
  roles                 = ["Microsoft.DataProtection/backupVaults/backup-operator"]
  source_resource_id    = azurerm_data_protection_backup_vault.example.id
}

resource "azurerm_storage_account" "example" {
  name                     = "example"
  resource_group_name      = azurerm_resource_group.example.name
  location                 = azurerm_resource_group.example.location
  account_tier             = "Standard"
  account_replication_type = "LRS"
}

resource "azurerm_storage_container" "example" {
  name                  = "example"
  storage_account_name  = azurerm_storage_account.example.name
  container_access_type = "private"
}

resource "azurerm_kubernetes_cluster_extension" "example" {
  name              = "example"
  cluster_id        = azurerm_kubernetes_cluster.example.id
  extension_type    = "Microsoft.DataProtection.Kubernetes"
  release_train     = "stable"
  release_namespace = "dataprotection-microsoft"
  configuration_settings = {
    "configuration.backupStorageLocation.bucket"                = azurerm_storage_container.example.name
    "configuration.backupStorageLocation.config.resourceGroup"  = azurerm_resource_group.example.name
    "configuration.backupStorageLocation.config.storageAccount" = azurerm_storage_account.example.name
    "configuration.backupStorageLocation.config.subscriptionId" = data.azurerm_client_config.current.subscription_id
    "credentials.tenantId"                                      = data.azurerm_client_config.current.tenant_id
  }
}

resource "azurerm_role_assignment" "test_extension_and_storage_account_permission" {
  scope                = azurerm_storage_account.example.id
  role_definition_name = "Storage Account Contributor"
  principal_id         = azurerm_kubernetes_cluster_extension.example.aks_assigned_identity[0].principal_id
}

resource "azurerm_role_assignment" "test_vault_msi_read_on_cluster" {
  scope                = azurerm_kubernetes_cluster.example.id
  role_definition_name = "Reader"
  principal_id         = azurerm_data_protection_backup_vault.example.identity[0].principal_id
}

resource "azurerm_role_assignment" "test_vault_msi_read_on_snap_rg" {
  scope                = azurerm_resource_group.snap.id
  role_definition_name = "Reader"
  principal_id         = azurerm_data_protection_backup_vault.example.identity[0].principal_id
}

resource "azurerm_role_assignment" "test_vault_msi_snapshot_contributor_on_snap_rg" {
  scope                = azurerm_resource_group.snap.id
  role_definition_name = "Disk Snapshot Contributor"
  principal_id         = azurerm_data_protection_backup_vault.example.identity[0].principal_id
}

resource "azurerm_role_assignment" "test_vault_data_operator_on_snap_rg" {
  scope                = azurerm_resource_group.snap.id
  role_definition_name = "Data Operator for Managed Disks"
  principal_id         = azurerm_data_protection_backup_vault.example.identity[0].principal_id
}

resource "azurerm_role_assignment" "test_vault_data_contributor_on_storage" {
  scope                = azurerm_storage_account.example.id
  role_definition_name = "Storage Blob Data Contributor"
  principal_id         = azurerm_data_protection_backup_vault.example.identity[0].principal_id
}

resource "azurerm_role_assignment" "test_cluster_msi_contributor_on_snap_rg" {
  scope                = azurerm_resource_group.snap.id
  role_definition_name = "Contributor"
  principal_id         = azurerm_kubernetes_cluster.example.identity[0].principal_id
}

resource "azurerm_data_protection_backup_policy_kubernetes_cluster" "example1" {
  name                = "example"
  resource_group_name = azurerm_resource_group.example.name
  vault_name          = azurerm_data_protection_backup_vault.example.name

  backup_repeating_time_intervals = ["R/2023-05-23T02:30:00+00:00/P1W"]

  retention_rule {
    name     = "Daily"
    priority = 25

    life_cycle {
      duration        = "P84D"
      data_store_type = "OperationalStore"
    }

    criteria {
      days_of_week           = ["Thursday"]
      months_of_year         = ["November"]
      weeks_of_month         = ["First"]
      scheduled_backup_times = ["2023-05-23T02:30:00Z"]
    }
  }

  default_retention_rule {
    life_cycle {
      duration        = "P14D"
      data_store_type = "OperationalStore"
    }
  }
}

resource "azurerm_data_protection_backup_policy_kubernetes_cluster" "example2" {
  name                = "example2"
  resource_group_name = azurerm_resource_group.example.name
  vault_name          = azurerm_data_protection_backup_vault.example.name

  backup_repeating_time_intervals = ["R/2023-05-23T02:30:00+00:00/P1W"]

  retention_rule {
    name     = "Weekly"
    priority = 30

    life_cycle {
      duration        = "P84D"
      data_store_type = "OperationalStore"
    }

    criteria {
      days_of_week           = ["Thursday"]
      months_of_year         = ["November"]
      weeks_of_month         = ["First"]
      scheduled_backup_times = ["2023-05-23T02:30:00Z"]
    }
  }

  default_retention_rule {
    life_cycle {
      duration        = "P14D"
      data_store_type = "OperationalStore"
    }
  }
}

resource "azurerm_data_protection_backup_instance_kubernetes_cluster" "example1" {
  name                         = "example1"
  location                     = azurerm_resource_group.example.location
  vault_id                     = azurerm_data_protection_backup_vault.example.id
  kubernetes_cluster_id        = azurerm_kubernetes_cluster.example.id
  snapshot_resource_group_name = azurerm_resource_group.snap.name
  backup_policy_id             = azurerm_data_protection_backup_policy_kubernetes_cluster.example1.id

  backup_datasource_parameters {
    excluded_namespaces              = []
    excluded_resource_types          = []
    cluster_scoped_resources_enabled = true
    included_namespaces              = []
    included_resource_types          = []
    label_selectors                  = []
    volume_snapshot_enabled          = false
  }

  depends_on = [
    azurerm_role_assignment.test_extension_and_storage_account_permission,
    azurerm_role_assignment.test_vault_msi_read_on_cluster,
    azurerm_role_assignment.test_vault_msi_read_on_snap_rg,
    azurerm_role_assignment.test_cluster_msi_contributor_on_snap_rg,
    azurerm_role_assignment.test_vault_msi_snapshot_contributor_on_snap_rg,
    azurerm_role_assignment.test_vault_data_operator_on_snap_rg,
    azurerm_role_assignment.test_vault_data_contributor_on_storage,
  ]
}
resource "azurerm_data_protection_backup_instance_kubernetes_cluster" "example2" {
  name                         = "example2"
  location                     = azurerm_resource_group.example.location
  vault_id                     = azurerm_data_protection_backup_vault.example.id
  kubernetes_cluster_id        = azurerm_kubernetes_cluster.example.id
  snapshot_resource_group_name = azurerm_resource_group.snap.name
  backup_policy_id             = azurerm_data_protection_backup_policy_kubernetes_cluster.example2.id

  backup_datasource_parameters {
    excluded_namespaces              = []
    excluded_resource_types          = []
    cluster_scoped_resources_enabled = true
    included_namespaces              = []
    included_resource_types          = []
    label_selectors                  = []
    volume_snapshot_enabled          = false
  }

  depends_on = [
    azurerm_role_assignment.test_extension_and_storage_account_permission,
    azurerm_role_assignment.test_vault_msi_read_on_cluster,
    azurerm_role_assignment.test_vault_msi_read_on_snap_rg,
    azurerm_role_assignment.test_cluster_msi_contributor_on_snap_rg,
    azurerm_role_assignment.test_vault_msi_snapshot_contributor_on_snap_rg,
    azurerm_role_assignment.test_vault_data_operator_on_snap_rg,
    azurerm_role_assignment.test_vault_data_contributor_on_storage,
  ]
}

Debug Output/Panic Output

This is my actual output from my terraform when attempting to create 2 or more instances, logs are taken from after the installation of the backup cluster extension;

2024-07-29T14:08:11.2393535Z [0m[1mmodule.akscluster.module.aks_backup_enable[0].azurerm_kubernetes_cluster_extension.main: Still creating... [3m20s elapsed][0m[0m
2024-07-29T14:08:21.2404176Z [0m[1mmodule.akscluster.module.aks_backup_enable[0].azurerm_kubernetes_cluster_extension.main: Still creating... [3m30s elapsed][0m[0m
2024-07-29T14:08:31.2406792Z [0m[1mmodule.akscluster.module.aks_backup_enable[0].azurerm_kubernetes_cluster_extension.main: Still creating... [3m40s elapsed][0m[0m
2024-07-29T14:08:41.2409286Z [0m[1mmodule.akscluster.module.aks_backup_enable[0].azurerm_kubernetes_cluster_extension.main: Still creating... [3m50s elapsed][0m[0m
2024-07-29T14:08:49.4779540Z [0m[1mmodule.akscluster.module.aks_backup_enable[0].azurerm_kubernetes_cluster_extension.main: Creation complete after 3m58s [id=/subscriptions/xxxxxx-xxxxx-xxxx-xxxxxx/resourceGroups/env1-rg-aks-dev-uks-599/providers/Microsoft.ContainerService/managedClusters/env1-akscluster-nonprd-001/providers/Microsoft.KubernetesConfiguration/extensions/aks-backup-extension][0m
2024-07-29T14:08:50.6493202Z [0m[1mmodule.akscluster.module.aks_backup_enable[0].azurerm_role_assignment.extension_id_blob_contributor_on_sa: Creating...[0m[0m
2024-07-29T14:08:50.6502365Z [0m[1mmodule.akscluster.module.aks_backup_enable[0].azurerm_role_assignment.cluster_msi_contributor_on_snap_rg: Creating...[0m[0m
2024-07-29T14:08:50.6511279Z [0m[1mmodule.akscluster.module.aks_backup_enable[0].azurerm_role_assignment.extension_id_contributor_on_sa: Creating...[0m[0m
2024-07-29T14:08:50.6537279Z [0m[1mmodule.akscluster.module.aks_backup_enable[0].azurerm_role_assignment.vault_msi_read_on_cluster: Creating...[0m[0m
2024-07-29T14:09:00.6512444Z [0m[1mmodule.akscluster.module.aks_backup_enable[0].azurerm_role_assignment.extension_id_blob_contributor_on_sa: Still creating... [10s elapsed][0m[0m
2024-07-29T14:09:00.6521248Z [0m[1mmodule.akscluster.module.aks_backup_enable[0].azurerm_role_assignment.cluster_msi_contributor_on_snap_rg: Still creating... [10s elapsed][0m[0m
2024-07-29T14:09:00.6532705Z [0m[1mmodule.akscluster.module.aks_backup_enable[0].azurerm_role_assignment.extension_id_contributor_on_sa: Still creating... [10s elapsed][0m[0m
2024-07-29T14:09:00.6555532Z [0m[1mmodule.akscluster.module.aks_backup_enable[0].azurerm_role_assignment.vault_msi_read_on_cluster: Still creating... [10s elapsed][0m[0m
2024-07-29T14:09:10.6513164Z [0m[1mmodule.akscluster.module.aks_backup_enable[0].azurerm_role_assignment.extension_id_blob_contributor_on_sa: Still creating... [20s elapsed][0m[0m
2024-07-29T14:09:10.6524669Z [0m[1mmodule.akscluster.module.aks_backup_enable[0].azurerm_role_assignment.cluster_msi_contributor_on_snap_rg: Still creating... [20s elapsed][0m[0m
2024-07-29T14:09:10.6536728Z [0m[1mmodule.akscluster.module.aks_backup_enable[0].azurerm_role_assignment.extension_id_contributor_on_sa: Still creating... [20s elapsed][0m[0m
2024-07-29T14:09:10.6558739Z [0m[1mmodule.akscluster.module.aks_backup_enable[0].azurerm_role_assignment.vault_msi_read_on_cluster: Still creating... [20s elapsed][0m[0m
2024-07-29T14:09:15.3446119Z [0m[1mmodule.akscluster.module.aks_backup_enable[0].azurerm_role_assignment.cluster_msi_contributor_on_snap_rg: Creation complete after 24s [id=/subscriptions/xxxxxx-xxxxx-xxxx-xxxxxx/resourceGroups/env1-rg-dpbv-nonprd-uk-001/providers/Microsoft.Authorization/roleAssignments/7e059240-0f3a-6488-7df4-fd460861de27][0m
2024-07-29T14:09:16.1435914Z [0m[1mmodule.akscluster.module.aks_backup_enable[0].azurerm_role_assignment.vault_msi_read_on_cluster: Creation complete after 25s [id=/subscriptions/xxxxxx-xxxxx-xxxx-xxxxxx/resourceGroups/env1-rg-aks-dev-uks-599/providers/Microsoft.ContainerService/managedClusters/env1-akscluster-nonprd-001/providers/Microsoft.Authorization/roleAssignments/c15db10c-9cfd-cebf-6be2-d7a318559f74][0m
2024-07-29T14:09:17.0524868Z [0m[1mmodule.akscluster.module.aks_backup_enable[0].azurerm_role_assignment.extension_id_contributor_on_sa: Creation complete after 26s [id=/subscriptions/xxxxxx-xxxxx-xxxx-xxxxxx/resourceGroups/env1-rg-dpbv-nonprd-uk-001/providers/Microsoft.Storage/storageAccounts/env1stdpbvnonprduks009/providers/Microsoft.Authorization/roleAssignments/71a5f7b7-6585-a4b1-2f6b-4a6a29f8be12][0m
2024-07-29T14:09:18.2590113Z [0m[1mmodule.akscluster.module.aks_backup_enable[0].azurerm_role_assignment.extension_id_blob_contributor_on_sa: Creation complete after 27s [id=/subscriptions/xxxxxx-xxxxx-xxxx-xxxxxx/resourceGroups/env1-rg-dpbv-nonprd-uk-001/providers/Microsoft.Storage/storageAccounts/env1stdpbvnonprduks009/providers/Microsoft.Authorization/roleAssignments/894524af-593f-173d-d5a7-25de72166c06][0m
2024-07-29T14:09:48.9933053Z [0m[1mmodule.akscluster.module.aks_backup_enable[0].azurerm_data_protection_backup_instance_kubernetes_cluster.instance2: Creating...[0m[0m
2024-07-29T14:09:48.9933703Z [0m[1mmodule.akscluster.module.aks_backup_enable[0].azurerm_data_protection_backup_instance_kubernetes_cluster.instance1: Creating...[0m[0m
2024-07-29T14:09:48.9951135Z [0m[1mmodule.akscluster.module.aks_backup_enable[0].azurerm_data_protection_backup_instance_kubernetes_cluster.instance3: Creating...[0m[0m
2024-07-29T14:09:58.9951637Z [0m[1mmodule.akscluster.module.aks_backup_enable[0].azurerm_data_protection_backup_instance_kubernetes_cluster.instance3: Still creating... [10s elapsed][0m[0m
2024-07-29T14:10:07.6660391Z [0m[1mmodule.akscluster.module.aks_backup_enable[0].azurerm_data_protection_backup_instance_kubernetes_cluster.instance3: Creation complete after 19s [id=/subscriptions/xxxxxx-xxxxx-xxxx-xxxxxx/resourceGroups/env1-rg-dpbv-nonprd-uk-001/providers/Microsoft.DataProtection/backupVaults/env1-dpbv-aks-nonprd-001/backupInstances/bs9-1W-AuditBackup][0m
2024-07-29T14:10:08.8004886Z [33m╵[0m[0m
2024-07-29T14:10:08.8022847Z [31m╷[0m[0m
2024-07-29T14:10:08.8023286Z [31m│[0m [0m[1m[31mError: [0m[0m[1mcreating Backup Instance (Subscription: "xxxxxx-xxxxx-xxxx-xxxxxx"
2024-07-29T14:10:08.8025903Z [31m│[0m [0mResource Group Name: "env1-rg-dpbv-nonprd-uk-001"
2024-07-29T14:10:08.8026241Z [31m│[0m [0mBackup Vault Name: "env1-dpbv-aks-nonprd-001"
2024-07-29T14:10:08.8026711Z [31m│[0m [0mBackup Instance Name: "bs9-Daily-Workloads"): performing CreateOrUpdate: unexpected status 406 (406 Not Acceptable) received with no body[0m
2024-07-29T14:10:08.8027018Z [31m│[0m [0m
2024-07-29T14:10:08.8027421Z [31m│[0m [0m[0m  with module.akscluster.module.aks_backup_enable[0].azurerm_data_protection_backup_instance_kubernetes_cluster.instance1,
2024-07-29T14:10:08.8027939Z [31m│[0m [0m  on ../modules/aks-backup/main.tf line 71, in resource "azurerm_data_protection_backup_instance_kubernetes_cluster" "instance1":
2024-07-29T14:10:08.8028867Z [31m│[0m [0m  71: resource "azurerm_data_protection_backup_instance_kubernetes_cluster" "instance1" [4m{[0m[0m
2024-07-29T14:10:08.8029140Z [31m│[0m [0m
2024-07-29T14:10:08.8029398Z [31m│[0m [0mcreating Backup Instance (Subscription:
2024-07-29T14:10:08.8029685Z [31m│[0m [0m"xxxxxx-xxxxx-xxxx-xxxxxx"
2024-07-29T14:10:08.8029992Z [31m│[0m [0mResource Group Name: "env1-rg-dpbv-nonprd-uk-001"
2024-07-29T14:10:08.8030278Z [31m│[0m [0mBackup Vault Name: "env1-dpbv-aks-nonprd-001"
2024-07-29T14:10:08.8030626Z [31m│[0m [0mBackup Instance Name: "bs9-Daily-Workloads"): performing CreateOrUpdate:
2024-07-29T14:10:08.8030995Z [31m│[0m [0munexpected status 406 (406 Not Acceptable) received with no body
2024-07-29T14:10:08.8031212Z [31m╵[0m[0m
2024-07-29T14:10:08.8042509Z [31m╷[0m[0m
2024-07-29T14:10:08.8043154Z [31m│[0m [0m[1m[31mError: [0m[0m[1mcreating Backup Instance (Subscription: "xxxxxx-xxxxx-xxxx-xxxxxx"
2024-07-29T14:10:08.8043655Z [31m│[0m [0mResource Group Name: "env1-rg-dpbv-nonprd-uk-001"
2024-07-29T14:10:08.8044084Z [31m│[0m [0mBackup Vault Name: "env1-dpbv-aks-nonprd-001"
2024-07-29T14:10:08.8044731Z [31m│[0m [0mBackup Instance Name: "bs9-4H-SecretsConfig"): performing CreateOrUpdate: unexpected status 406 (406 Not Acceptable) received with no body[0m
2024-07-29T14:10:08.8045161Z [31m│[0m [0m
2024-07-29T14:10:08.8045884Z [31m│[0m [0m[0m  with module.akscluster.module.aks_backup_enable[0].azurerm_data_protection_backup_instance_kubernetes_cluster.instance2,
2024-07-29T14:10:08.8046868Z [31m│[0m [0m  on ../modules/aks-backup/main.tf line 91, in resource "azurerm_data_protection_backup_instance_kubernetes_cluster" "instance2":
2024-07-29T14:10:08.8047451Z [31m│[0m [0m  91: resource "azurerm_data_protection_backup_instance_kubernetes_cluster" "instance2" [4m{[0m[0m
2024-07-29T14:10:08.8047826Z [31m│[0m [0m
2024-07-29T14:10:08.8048189Z [31m│[0m [0mcreating Backup Instance (Subscription:
2024-07-29T14:10:08.8048591Z [31m│[0m [0m"xxxxxx-xxxxx-xxxx-xxxxxx"
2024-07-29T14:10:08.8048988Z [31m│[0m [0mResource Group Name: "env1-rg-dpbv-nonprd-uk-001"
2024-07-29T14:10:08.8049398Z [31m│[0m [0mBackup Vault Name: "env1-dpbv-aks-nonprd-001"
2024-07-29T14:10:08.8049854Z [31m│[0m [0mBackup Instance Name: "bs9-4H-SecretsConfig"): performing CreateOrUpdate:
2024-07-29T14:10:08.8050301Z [31m│[0m [0munexpected status 406 (406 Not Acceptable) received with no body
2024-07-29T14:10:08.8050642Z [31m╵[0m[0m
2024-07-29T14:10:09.2055737Z Releasing state lock. This may take a few moments...
2024-07-29T14:10:09.3839280Z ##[error]Error: The process '/usr/local/bin/terraform' failed with exit code 1
2024-07-29T14:10:09.3858891Z ##[section]Finishing: Terraform Apply

Expected Behaviour

I'd expect 2 each of the backup instances to be created.

Actual Behaviour

When attempting to create more than one Kubernetes backup instance in Terraform (whether they are separate resource blocks or a for-each loop, the 'creating' process initiates, however as soon as the first instance reports completed, the process stops with http error code 406, and any remaining instances still being created fail.

If you re-run the terraform unchanged, then again, one instance will be created, but if there is more than one instance still to be created then the same error occurs.

You can repeat this until all are created.

The same error occurs in the destroy process.

Steps to Reproduce

Run the TF with the multiple backup instances specified. It will start to create all the instances, but as soon as the first one reports 'complete' the others will all fail with 406 error.

Important Factoids

No response

References

No response

hqhqhqhqhqhqhqhqhqhqhq commented 1 month ago

@Rick-healy Hello, thanks for raising this. I was able to recreate the problem some of the times. I believe this may be an issue with service processing requests for 2 backup resources in parallel. I will confirm with the service team on this, in the mean time, I believe adding the first back up resource as a dependency for the second back up resource in the depends_on property should solve the problem.

Let me know any concerns, further problems.

Rick-healy commented 1 month ago

Thank you @hqhqhqhqhqhqhqhqhqhqhq for responding. Yes it does sound like some parallel issue. My actual code has the instances being created in a for-each loop from an object map, so I don't think I can add dependencies in the loop for each iteration, but I can investigate other non-parallel approaches for this. Or add the instances one at a time in the TF and re-run, which isn't a massive overhead, as they aren't going to be too dynamic!

amenasse commented 1 month ago

We are seeing the same error attempting to create a backup instance in a backup vault that has an existing soft deleted instance

DTurin-IH commented 1 week ago

We are seeing the same error attempting to create a backup instance in a backup vault that has an existing soft deleted instance

-Ran into the same issue when redeploying a backup vault. Were you able to remove the soft deleted instance or find a workaround?

roquesao commented 1 week ago

Same issue when creating a backup vault from scratch

Error: creating/updating DataProtection BackupInstance ("Backup Instance (Subscription: \"xxxxxxxxxxxxxx\"\nResource Group Name: \"xxxxxx\"\nBackup Vault Name: \"xxxxxx\"\nBackup Instance Name: \"xxxx\")"): performing CreateOrUpdate: unexpected status 406 (406 Not Acceptable) received with no body

Dawiducik commented 17 hours ago

Same issue here

hashicorp / terraform-provider-azurerm