KeyVault creation and destruction timing errors

jeffwmiles commented 2 years ago

Is there an existing issue for this?

[X] I have searched the existing issues

Community Note

Please vote on this issue by adding a :thumbsup: reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

1.2.7

AzureRM Provider Version

3.19.1

Affected Resource(s)/Data Source(s)

azurerm_key_vault

Terraform Configuration Files

resource "azurerm_virtual_network" "aksmgmt" {
  name                = "aksmgmt-${var.countryCode}"
  resource_group_name = azurerm_resource_group.sharedservices.name
  location            = azurerm_resource_group.sharedservices.location
  address_space       = [var.addressSpaces["aksmgmt"]]
}

resource "azurerm_subnet" "aksmgmt_subnets" {
  for_each             = module.subnets_aksmgmt.network_cidr_blocks
  name                 = each.key
  resource_group_name  = azurerm_resource_group.sharedservices.name
  virtual_network_name = azurerm_virtual_network.aksmgmt.name
  address_prefixes     = [each.value]
  service_endpoints    = ["Microsoft.KeyVault"]
}
resource "azurerm_key_vault" "default" {
  name                            = "sharedservices-${var.countryCode}"
  resource_group_name             = azurerm_resource_group.sharedservices.name
  location                        = azurerm_resource_group.sharedservices.location
  tenant_id                       = data.azurerm_client_config.current.tenant_id
  soft_delete_retention_days      = 7
  sku_name                        = "standard"

  network_acls {
    bypass         = "AzureServices"
    default_action = "Deny"
    virtual_network_subnet_ids = [
      resource.azurerm_subnet.sharedmgmt_subnets["pipelinerunners"].id,
      resource.azurerm_subnet.datamgmt_subnets["pipelinerunners"].id,
      resource.azurerm_subnet.aksmgmt_subnets["pipelinerunners"].id,
    ]
  }
  depends_on = [
    resource.azurerm_subnet.sharedmgmt_subnets,
    resource.azurerm_subnet.datamgmt_subnets,
    resource.azurerm_subnet.aksmgmt_subnets
  ]
}
resource "azurerm_key_vault_access_policy" "pipeline_spn" {
  key_vault_id = azurerm_key_vault.default.id
  tenant_id    = data.azurerm_client_config.current.tenant_id
  object_id    = var.PipelineSpnId

  key_permissions = [
    "Get", "List", "Create", "Update", "Verify", "Delete", "Purge"
  ]
  secret_permissions = [
    "Get", "List", "Set", "Delete", "Purge"
  ]
  certificate_permissions = [
    "Get", "List", "ListIssuers", "Create", "SetIssuers", "Update", "Delete", "Purge"
  ]
}

Debug Output/Panic Output

During Apply:

[module.inceptionservices.azurerm_subnet.sharedmgmt_subnets["pipelinerunners"]: Creation complete after 5s [id=/subscriptions/redacted/resourceGroups/sharedservices-us/providers/Microsoft.Network/virtualNetworks/sharedmgmt-us/subnets/pipelinerunners]
[module.inceptionservices.azurerm_subnet.aksmgmt_subnets["pipelinerunners"]: Creation complete after 5s [id=/subscriptions/redacted/resourceGroups/sharedservices-us/providers/Microsoft.Network/virtualNetworks/aksmgmt-us/subnets/pipelinerunners]
[module.inceptionservices.azurerm_key_vault.default: Creating...

[Error: [creating Vault: (Name "sharedservices-us" / Resource Group "sharedservices-us"): keyvault.VaultsClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="VirtualNetworkNotValid" Message="Operation on Virtual Network could not be performed. StatusCode: 404 (NotFound). Error Code: ParentResourceNotFound. Error Message: Can not perform requested operation on nested resource. Parent resource 'aksmgmt-us' not found.."

During Destroy:

[module.inceptionservices.azurerm_key_vault_access_policy.pipeline_spn: Destroying... [id=/subscriptions/redacted/resourceGroups/sharedservices-us/providers/Microsoft.KeyVault/vaults/sharedservices-us/objectId/c90f4efb]
[module.inceptionservices.azurerm_key_vault_access_policy.pipeline_spn: Destruction complete after 7s
[module.inceptionservices.azurerm_key_vault.default: Destroying... [id=/subscriptions/redacted/resourceGroups/sharedservices-us/providers/Microsoft.KeyVault/vaults/sharedservices-us]

[Error: [keyvault.VaultsClient#PurgeDeleted: Failure sending request: StatusCode=404 -- Original Error: Code="DeletedVaultNotFound" Message="The specified deleted vault 'sharedservices-us' does not exist. Ensure that the vault was indeed deleted and that it is in recoverable state. If soft delete was not enabled then the vault is permanently deleted. Follow this link for more information: https://go.microsoft.com/fwlink/?linkid=2149745"

Expected Behaviour

During Apply: Since Terraform received "Creation complete" from subnet resource, the KeyVault depending on ServiceEndpoint of the subnet for Network ACLs should succeed.

During Destroy: Timing of destroy of Vault should be handled by provider, to avoid error "DeletedVaultNotFound"

Actual Behaviour

When run interactively, I am unable to reliably reproduce the issue. However, when this terraform configuration runs within a Bitbucket Pipeline, I am consistently seeing the error messages.

I will attempt to elevate logging output to Debug and add additional details.

Steps to Reproduce

No response

Important Factoids

No response

References

No response

magodo commented 2 years ago

@jeffwmiles Thank you for submitting this! I've tried with your config above and things work for me. As said, would you please provide more details like debug log so that we can further take a look into.

jeffwmiles commented 2 years ago

Here is a gist of the destroy operation at TF_LOG level DEBUG:

https://gist.github.com/jeffwmiles/a08c400d554d49455c190f24bd6ad71a

Notes to call out:

Line 486 - destroying access policy starts
Line 646 - destruction complete - access policy
Line 647 - destroying KeyVault starts
Line 699 - keyvault marked for purge, proceeding with purge
Line 709 - 404 received on POST request to execute purge of KeyVault
Line 727 - Azure error surfaced of "DeletedVaultNotFound"

I'm still working on reproducing the error seen during apply.

hashicorp / terraform-provider-azurerm