search

hashicorp / terraform-provider-azurerm

Terraform provider for Azure Resource Manager
https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs
Mozilla Public License 2.0
4.59k stars 4.63k forks source link

Unable to determine the Resource ID for the Key Vault at URL #19322

Closed tschechniker closed 11 months ago

tschechniker commented 1 year ago

Is there an existing issue for this?

Community Note

Terraform Version

1.2.9

AzureRM Provider Version

3.30.0

Affected Resource(s)/Data Source(s)

Unable to determine the Resource ID for the Key Vault at URL

Terraform Configuration Files

data "azurerm_client_config" "current" {}

resource "azurerm_resource_group" "example" {
  name     = "example-resources"
  location = "West Europe"
}

resource "azurerm_key_vault" "example" {
  name                        = "examplekeyvault"
  location                    = azurerm_resource_group.example.location
  resource_group_name         = azurerm_resource_group.example.name
  enabled_for_disk_encryption = true
  tenant_id                   = data.azurerm_client_config.current.tenant_id
  soft_delete_retention_days  = 7
  purge_protection_enabled    = false

  sku_name = "standard"

  access_policy {
    tenant_id = data.azurerm_client_config.current.tenant_id
    object_id = data.azurerm_client_config.current.object_id

    key_permissions = [
      "Get",
    ]

    secret_permissions = [
      "Get",
    ]

    storage_permissions = [
      "Get",
    ]
  }
}

resource "azurerm_key_vault_key" "example" {
  name         = "example-key"
  key_vault_id = azurerm_key_vault.example.id
  key_type     = "RSA-HSM"
  key_size     = 4096

  expiration_date = timeadd(formatdate("YYYY-MM-DD'T'12:00:00Z", timestamp()), "26297h")

  key_opts = [
    "decrypt",
    "encrypt",
    "sign",
    "unwrapKey",
    "verify",
    "wrapKey",
  ]
}

Debug Output/Panic Output

Error: Unable to determine the Resource ID for the Key Vault at URL "https://XXXX.vault.azure.net/"

Expected Behaviour

We are using Terratest to setup and destroy azure resources in a pipeline. The creation and the deletion works fine.

Actual Behaviour

The creation works fine but as soon as we run terraform destroy he cant find the resource id

Steps to Reproduce

terraform apply terraform destroy

Important Factoids

Azure Westeurope

References

No response

ThumbGen commented 1 year ago

Any plans to fix this? We can't destroy any cluster since we started to use Azure Key Vault :(

Umair841 commented 1 year ago

I am unable to delete the key with terraform destroy. I thing this is the same issue.

any fix for this? Screenshot from 2023-02-22 14-18-12

anshu14 commented 1 year ago

I am also facing same issue with azurerm 3.35 only while deleting. Any workaround or planned fix for this. I think when it tries to delete the azurerm_key_vault_key resource, it has Vault URL. So, it looks that it cannot get the Resource ID from that vault URL.

mdizcalvinoaz400-01 commented 1 year ago

I am gretting same error 3.42 image

askingcat commented 1 year ago

We are facing the same problem for our tests with azurerm 3.46.

In our case we see a flaky behavior. On a rerun the elements of the vault can often be deleted successfully.

shubham2110 commented 1 year ago

facing same problem while deleting on azure 3.47

jeffwmiles commented 1 year ago

I'm seeing this issue as well - I'm wondering how many other reporters of this issue are using Private Endpoints in addition to public access for their KeyVault (as I am)?

I have enabled debug logging in my pipeline run for Terraform, and I can tell these errors are surfaced by the azurerm_key_vault_secret in my case. 

2023-03-21T22:44:32.361Z [ERROR] provider.terraform-provider-azurerm_v3.48.0_x5: Response contains error diagnostic: diagnostic_severity=ERROR diagnostic_summary="Unable to determine the Resource ID for the Key Vault at URL "https://kv-redacted-jutp.vault.azure.net/"" tf_proto_version=5.3 tf_resource_type=azurerm_key_vault_secret tf_rpc=ApplyResourceChange @caller=github.com/hashicorp/terraform-plugin-go@v0.14.3/tfprotov5/internal/diag/diagnostics.go:55 @module=sdk.proto diagnostic_detail= tf_provider_addr=provider tf_req_id=ab559199-904c-528e timestamp=2023-03-21T22:44:32.361Z
2023-03-21T22:44:32.363Z [ERROR] vertex "module.client-basesetup.azurerm_key_vault_secret.connection-string-secret (destroy)" error: Unable to determine the Resource ID for the Key Vault at URL "https://kv-redacted-jutp.vault.azure.net/"

I have a few suspicions at this point that I'm trying to narrow down, but the debug logs aren't really giving an indication

I do know that when I re-run my destroy, AFTER my azurerm_private_endpoint resources have been removed, the remaining destroy steps succeed.

jeffwmiles commented 1 year ago

I've traced this particular issue a little further in my environment, and I think it is due to some unexpected behaviour with the Azure REST API.

When I closely inspect the debug logs of a destroy operation, I noticed two things:

  1. During the refreshing state operation (of my KeyVault, and an access policy) the provider uses the Resource List By Resource Group api to return the KeyVault object: 
    
module.default.azurerm_key_vault_access_policy.pipeline_spn: Refreshing state... [id=/subscriptions/subscriptionIDredacted/resourceGroups/rg-default-vnhs/providers/Microsoft.KeyVault/vaults/kv-default-vnhs/objectId/c90f4efb]
2023-03-23T19:20:38.690Z [DEBUG] provider.terraform-provider-azurerm_v3.48.0_x5: AzureRM Request:
GET /subscriptions/subscriptionIDredacted/resourceGroups/rg-default-vnhs/providers/Microsoft.KeyVault/vaults/kv-default-vnhs?api-version=2021-10-01 HTTP/1.1
Host: management.azure.com
User-Agent: Go/go1.19.3 (amd64-linux) go-autorest/v14.2.1 Azure-SDK-For-Go/v66.0.0 keyvault/2021-10-01 HashiCorp Terraform/1.3.7 (+https://www.terraform.io) Terraform Plugin SDK/2.10.1 terraform-provider-azurerm/dev pid-222c6c49-1b0a-5959-a213-6608f9eb8820
X-Ms-Correlation-Request-Id: 72bb7e97
Accept-Encoding: gzip: timestamp=2023-03-23T19:20:38.690Z
2023-03-23T19:20:38.781Z [DEBUG] provider.terraform-provider-azurerm_v3.48.0_x5: AzureRM Response for https://management.azure.com/subscriptions/subscriptionIDredacted/resourceGroups/rg-default-vnhs/providers/Microsoft.KeyVault/vaults/kv-default-vnhs?api-version=2021-10-01:
HTTP/2.0 200 OK
Cache-Control: no-cache
Content-Type: application/json; charset=utf-8
Date: Thu, 23 Mar 2023 19:20:37 GMT
Expires: -1
Pragma: no-cache
Server: Microsoft-IIS/10.0
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Accept-Encoding
X-Aspnet-Version: 4.0.30319
X-Content-Type-Options: nosniff
X-Ms-Client-Request-Id: 74104b8a
X-Ms-Correlation-Request-Id: 72bb7e97
X-Ms-Keyvault-Service-Version: 1.5.692.0
X-Ms-Ratelimit-Remaining-Subscription-Reads: 11978
X-Ms-Request-Id: e25c0369
X-Ms-Routing-Request-Id: EASTUS:20230323T192038Z:670e3ba7

{"id":"/subscriptions/subscriptionIDredacted/resourceGroups/rg-default-vnhs/providers/Microsoft.KeyVault/vaults/kv-default-vnhs", - SNIPPED }


2. When the destroy action takes place for an `azurerm_key_vault_secret` resource, the provider uses the [Resource List api](https://learn.microsoft.com/en-us/rest/api/resources/resources/list) to return the KeyVault object, with a filter on its resourceType and Name:

2023-03-23T19:20:58.144Z [DEBUG] module.aks-setup.azurerm_key_vault_secret.cluster_admin_passwd: applying the planned Delete change 2023-03-23T19:20:58.144Z [DEBUG] provider.terraform-provider-azurerm_v3.48.0_x5: AzureRM Request: GET /subscriptions/subscriptionIDredacted/resources?%24filter=resourceType+eq+%27Microsoft.KeyVault%2Fvaults%27+and+name+eq+%27kv-default-vnhs%27&%24top=5&api-version=2020-06-01 HTTP/1.1 Host: management.azure.com User-Agent: Go/go1.19.3 (amd64-linux) go-autorest/v14.2.1 Azure-SDK-For-Go/v66.0.0 resources/2020-06-01 HashiCorp Terraform/1.3.7 (+https://www.terraform.io) Terraform Plugin SDK/2.10.1 terraform-provider-azurerm/dev pid-222c6c49-1b0a-5959-a213-6608f9eb8820 X-Ms-Correlation-Request-Id: 444a209e Accept-Encoding: gzip: timestamp=2023-03-23T19:20:58.144Z 2023-03-23T19:20:59.072Z [DEBUG] provider.terraform-provider-azurerm_v3.48.0_x5: AzureRM Response for https://management.azure.com/subscriptions/subscriptionIDredacted/resources?%24filter=resourceType+eq+%27Microsoft.KeyVault%2Fvaults%27+and+name+eq+%27kv-default-vnhs%27&%24top=5&api-version=2020-06-01: HTTP/2.0 200 OK Cache-Control: no-cache Content-Type: application/json; charset=utf-8 Date: Thu, 23 Mar 2023 19:20:58 GMT Expires: -1 Pragma: no-cache Strict-Transport-Security: max-age=31536000; includeSubDomains Vary: Accept-Encoding X-Content-Type-Options: nosniff X-Ms-Correlation-Request-Id: 444a209e X-Ms-Ratelimit-Remaining-Subscription-Reads: 11999 X-Ms-Request-Id: 472b5153 X-Ms-Routing-Request-Id: SOUTHCENTRALUS:20230323T192059Z:472b5153

{"value":[]}: timestamp=2023-03-23T19:20:59.072Z



Surprisingly, # 1 returns the object properties of the KeyVault, and # 2 returns a blank value, even though it is the same KeyVault! This is where the original error is coming from; the Secret resource has the ID reference, but during the destroy operation the lookup for the KeyVault doesn't return any values. 

I have confirmed this both in Postman, and in the Resource Explorer. The latter for example, when I navigate through `Subscriptions -> Subid -> Providers -> Microsoft.KeyVault` I do not see my "kv-default-vnhs" at all, even though I can browse and manage it in the Azure Portal.
![image](https://user-images.githubusercontent.com/40004712/227408929-53c4a963-dc16-404c-9097-67f9f6fd8359.png)

But if I navigate through the resource group, I do see it!
![image](https://user-images.githubusercontent.com/40004712/227409046-ad5ba6b8-ca86-4ac0-82fd-53fbb7de19e6.png)

Even more strange, within the Resource List api (not scoped through a resource group) it returns KeyVaults that no longer exist (such as the "kv-*-vlnb" in the first image; not even in a Soft-Delete stage. 

I cannot understand how this can be - a resource can exist and function, but only partially discover-able by the API, hidden if you are looking at the subscription scope instead of resource group? I will likely open a support ticket with Azure about this, but if there are any internal Hashicorp or Microsoft resources who see this, help would certainly be appreciated. 

Edit: 20 minutes after typing this, while trying to gather screenshots for an Azure support ticket, my Postman queries magically showed the KeyVault and had removed the actual non-existent ones. My hunch is some kind of delay in replication of objects across regions, which is terrible for a CI/CD pipeline of a Terraform module that is doing an apply and destroy 😟

Edit 2: it appears this terraform provider issue is directly related to similar behaviour found in Azure CLI: https://github.com/Azure/azure-cli/issues/23428
immae1 commented 1 year ago

Same on my side; on a terraform destroy I get Error: Unable to determine the resource ID for the key vault at URL if the key vault was previously created in a terraform apply run. If I run terraform destroy again, the key vault is removed and destroy is successful. Right now this is a blocking issue for me as I rely on terraform apply plus destroy in my pipelines so that I can be sure that my development is always able to build and destroy the whole stack. FYI, I have already bumped to the versions azurerm: 3.48.0 and terraform: 1.4.1.

@jeffwmiles are you still facing this issue? Greetz

immae1 commented 1 year ago

@tschechniker anything new about this topic? Do you have a workaround?

adeturner commented 1 year ago

A workaround would be useful here too

danieltroyer commented 1 year ago

I've got this issue blocking me today. Azure Key Vaults, created on previous days so less likely to be propagation lag, refresh during terraform plan properly, but they fail with terraform destroy throwing error Error: Unable to determine the Resource ID for the Key Vault at URL ..... Even the plan stage of destroy shows the correct resource IDs for the secrets and vaults, but the apply stage fails with the error.

Adding to the fun, the state file shows the full and correct resource IDs but won't use them. I workaround would be an override command to allow the use of IDs from state rather than the API values that the user above reports are missing.

w4rgrum commented 1 year ago

I have the same issue with azurerm v3.78.0. Manually testing with curl showed propagation lag on the resources List api on my end (~15-20 mins), which causes the failure when the KV is new and a destroy is called (typical use case seems to be TF module tests). Which is basically what was described in this previous comment: https://github.com/hashicorp/terraform-provider-azurerm/issues/19322#issuecomment-1482167240 The destroy probably need to use a different API to get this fixed.

phoehnel commented 11 months ago

It is interesting that the behavior seems to be random. We have pipelines with Terratest that regularly build and destroy KeyVaults and some destroys work fine, some exit with "unable to determine the Resource ID for the Key Vault at URL".

Our pipeline logs unfortunately only go back 6 months, but in this period I could not find any pipeline with this error until yesterday. Since yesterday every pipeline failed with it, even using the same provider version and tf-files that once worked.

quanthereal commented 9 months ago

I'm running Terraform v1.6.3 and azurerm v3.89.0 with similar setup.

Unfortunately I still run into this issue in my tests module trying to destroy a key vault and a key vault key even though it seems to be fixed with azurerm v3.83.

dkirrane commented 7 months ago

I'm hitting this issue with Terraform 1.6.5. It does seem to happen randomly.

    azurerm = {
      source  = "hashicorp/azurerm"
      version = "3.96.0"
    }
Uqqasha commented 6 months ago

Experiencing this same issue with Terraform 1.7.5 and azurerm 3.91.0, although it happens randomly and not on every terraform destroy run.

github-actions[bot] commented 5 months ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.