Failed to apply `azurerm_key_vault` due to token expiration

duzitong commented 3 weeks ago

Is there an existing issue for this?

[X] I have searched the existing issues

Community Note

Please vote on this issue by adding a :thumbsup: reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave comments along the lines of "+1", "me too" or "any updates", they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment and review the contribution guide to help.

Terraform Version

1.9.5

AzureRM Provider Version

3.116.0

Affected Resource(s)/Data Source(s)

azurerm_key_vault

Terraform Configuration Files

resource "azurerm_key_vault" "DeploymentKeyVault" {
  name                = var.keyvaultName
  location            = var.resourceGroup.location
  resource_group_name = var.resourceGroup.name
  tags                = {}

  enabled_for_deployment          = true
  enabled_for_template_deployment = true
  enabled_for_disk_encryption     = true
  tenant_id                       = data.azurerm_client_config.current.tenant_id
  soft_delete_retention_days    = 30
  enable_rbac_authorization     = true
  public_network_access_enabled = true
  sku_name                      = "standard"

  timeouts {
    create = "30m"
    read   = "30m"
    delete = "30m"
  }
}

Debug Output/Panic Output

module.base.module.hci.azurerm_key_vault.DeploymentKeyVault: Still creating... [9m31s elapsed]
module.base.module.hci.azurerm_key_vault.DeploymentKeyVault: Still creating... [9m41s elapsed]
module.base.module.hci.azurerm_key_vault.DeploymentKeyVault: Still creating... [9m51s elapsed]
module.base.module.hci.azurerm_key_vault.DeploymentKeyVault: Still creating... [10m1s elapsed]
╷
│ Error: retrieving `contact` for KeyVault: autorest/Client#Do: Preparing request failed: StatusCode=0 -- Original Error: obtaining token: clientCredentialsToken: received HTTP status 401 with response: {"error":"invalid_client","error_description":"AADSTS700024: Client assertion is not within its valid time range. Current time: 2024-08-23T05:30:34.3067712Z, assertion valid from 2024-08-23T05:04:22.0000000Z, expiry time of assertion 2024-08-23T05:14:22.0000000Z. Review the documentation at https://learn.microsoft.com/entra/identity-platform/certificate-credentials . Trace ID: bec8946e-1249-4b79-bbbb-f9ca71ac5700 Correlation ID: 1186a2ec-3bde-4835-8897-b47fb2ada3ee Timestamp: 2024-08-23 05:30:34Z","error_codes":[700024],"timestamp":"2024-08-23 05:30:34Z","trace_id":"bec8946e-1249-4b79-bbbb-f9ca71ac5700","correlation_id":"1186a2ec-3bde-4835-8897-b47fb2ada3ee","error_uri":"https://login.microsoftonline.com/error?code=700024"}
│ 
│   with module.base.module.hci.azurerm_key_vault.DeploymentKeyVault,
│   on ..\..\modules\hci\keyvault.tf line 3, in resource "azurerm_key_vault" "DeploymentKeyVault":
│    3: resource "azurerm_key_vault" "DeploymentKeyVault" {
│ 
╵

Expected Behaviour

Key vault can be created

Actual Behaviour

Creation failed after 10 minutes.

Steps to Reproduce

terraform init terraform apply

Important Factoids

No response

References

No response

wuxu92 commented 3 weeks ago

Hi @duzitong, is this an occasional issue or a persistent one? Also, what type of authentication is being used in this run?

duzitong commented 3 weeks ago

When the creation takes longer than 10 minutes, this issue always occurs. Because the first-time creation fails, the resource is tainted. A recreation often takes shorter time (about 6 minutes), the creation will succeed.

I'm using OIDC auth by setting the token to ARM_OIDC_TOKEN_FILE_PATH env. I added a job in background to refresh token in the file. This token can be used to create super long running resources.

harshavmb commented 2 weeks ago

Can you check whether time is synced on your terraform controller? If the time on controller is having a big drift, token will become invalid.

duzitong commented 2 weeks ago

Time is synced. As the error message shows, the token lifetime is 10 minutes. The creation fails after 10 minutes. I'm using Azure DevOps pipelines to run terraform apply. The service connection is created with federated identity. So, the token is short-lived.

I added a job to refresh the oidc token every 5 minutes in the ARM_OIDC_TOKEN_FILE_PATH. But azurerm provider doesn't pick the latest token from the file.

wuxu92 commented 2 weeks ago

@duzitong I suspect the issue might be with the authentication from the OIDC environment. Retrieving contact is a data-plane API and requires a callback to obtain the actual token for data-plane operations, if I remember correctly. I'll review the code later. Just out of curiosity, do you encounter this error when creating other long-running resources that take more than 10 minutes, or is it only with the key vault?

duzitong commented 2 weeks ago

I didn't create any other long running resources by AzureRM provider. But I have created one resource by azapi provider by refreshing token in ARM_OIDC_TOKEN_FILE_PATH over 2 hours. The creation succeeds but terraform crashed when persisting the state.

Posted the issue: https://github.com/hashicorp/terraform/issues/35632

harshavmb commented 1 week ago

Appears to be duplicate of #15894. In general it's for all supported auth types not just ARM_OIDC_TOKEN_FILE_PATH.

duzitong commented 1 week ago

OIDC token for GitHub Actions is actually perfectly handled. Although refreshing OIDC token from the file can solve the issue, handling token refreshing in the file is still an overhead.

It's better to implement similar mechanism for Azure DevOps.

hashicorp / terraform-provider-azurerm