Key vault is not accessible immediately after creation

serbezki commented 3 weeks ago

Is there an existing issue for this?

[X] I have searched the existing issues

Community Note

Please vote on this issue by adding a :thumbsup: reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave comments along the lines of "+1", "me too" or "any updates", they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment and review the contribution guide to help.

Terraform Version

1.9.7

AzureRM Provider Version

4.7.0

Affected Resource(s)/Data Source(s)

azurerm_key_vault

Terraform Configuration Files

resource "azurerm_key_vault" "keyvault" {
public_network_access_enabled = false
...
}

resource "azurerm_private_endpoint" "kv_private_endpoint" {
...
}

resource "azurerm_key_vault_secret" "key_vault_secret" {
 depends_on      = [azurerm_private_endpoint.kv_private_endpoint]
...
}

Debug Output/Panic Output

2024-10-31T12:49:11.489Z [ERROR] provider.terraform-provider-azurerm_v4.7.0_x5: Response contains error diagnostic: tf_rpc=ApplyResourceChange @caller=github.com/hashicorp/terraform-plugin-go@v0.23.0/tfprotov5/internal/diag/diagnostics.go:58 @module=sdk.proto diagnostic_detail="" tf_resource_type=azurerm_key_vault_secret tf_provider_addr=registry.terraform.io/hashicorp/azurerm tf_req_id=64153fe1-4710-260d-af18-08d0f4229866 diagnostic_severity=ERROR diagnostic_summary="checking for presence of existing Secret \"secret-name\" (Key Vault \"https://vault-name.vault.azure.net/\"): keyvault.BaseClient#GetSecret: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code=\"Forbidden\" Message=\"Connection is not an approved private link and caller was ignored because bypass is not set to 'AzureServices' and PublicNetworkAccess is set to 'Disabled'. \r\nVault: vault-name;location=northeurope\" InnerError={\"code\":\"ForbiddenByConnection\"}" tf_proto_version=5.6 timestamp=2024-10-31T12:49:11.489Z
2024-10-31T12:49:11.490Z [ERROR] vertex "module.key_vault.module.kv.azurerm_key_vault_secret.key_vault_secret[\"secret-name\"]" error: checking for presence of existing Secret "secret-name" (Key Vault "https://vault-name.vault.azure.net/"): keyvault.BaseClient#GetSecret: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="Forbidden" Message="Connection is not an approved private link and caller was ignored because bypass is not set to 'AzureServices' and PublicNetworkAccess is set to 'Disabled'. \r\nVault: vault-name;location=northeurope" InnerError={"code":"ForbiddenByConnection"}

Expected Behaviour

Vault, private endpoint and then secrets all get created without any problems.

Actual Behaviour

Secret creation fails because vault is inaccessible. It works fine when you re-run.

Steps to Reproduce

Create vault with public access disabled, private endpoint and secret in the same config.

Important Factoids

No response

References

No response

serbezki commented 3 weeks ago

An extra note that can be of help - I tried adding a 1 minute delay after the endpoint creation and before the secrets creation. That didn't work, regardless of how long the delay is.

Most likely terraform keeps the connection open and doesn't realize the vault has a different IP now. I'm wondering how that connection can be re-initialized. I thought a about using a data block and referencing that for the secrets, but that messes up the plan since before any of this is created the data block points to a non-existent resource.

magodo commented 3 weeks ago

The error message indicates your machine running Terraform is not from a subnet that has the access to the private endpoint. Meanwhile, since you have no network rule to allowlist the IP of your machine, it is blocked and you got a 403. So please either update the network rule, or ensure your machine is from the vnet that peers to/is the vnet of the private endpoint.

serbezki commented 3 weeks ago

If that were the problem then why would it work fine on the next run?

The configuration is okay, it only fails to work through it in one go. That's why I believe it should be a provider fix, anything that works fine after you re-run should be handled in the provider imo.

magodo commented 2 weeks ago

Presumably you have a azurerm_key_vault_access_policy defined, and you haven't defined a dependency on it in your azurerm_key_vault_secret?

hashicorp / terraform-provider-azurerm