search

jason-johnson / azure-pipelines-tasks-terraform

Azure Pipelines extension for Terraform
MIT License
122 stars 51 forks source link

Terraform apply fails with clientCredentialsToken already expired #426

Open robbert-nlo opened 3 months ago

robbert-nlo commented 3 months ago

Describe the bug Terraform apply sometimes fails in DevOps pipelines using OIDC authentication due to expired credentials:

Error: error loading state: autorest/Client#Do: Preparing request failed: StatusCode=0 -- Original Error: clientCredentialsToken: received HTTP status 401 with response: {"error":"invalid_client","error_description":"AADSTS700024: Client assertion is not within its valid time range. Current time: 2024-03-28T13:09:23.9070719Z, assertion valid from 2024-03-28T12:38:32.0000000Z, expiry time of assertion 2024-03-28T12:48:31.0000000Z. Review the documentation at https://docs.microsoft.com/azure/active-directory/develop/active-directory-certificate-credentials .

This issue appears when an apply is run using a plan created more than 10 minutes earlier. For example when there is a manual approval step between plan and apply which took over 10 minutes, or when re-running a failed apply job after 10 minutes of creating the plan.

The root cause seems to be that terraform is saving the temporary OIDC credentials in the plan file, which is then attempted to be reused later, when the temporary credentials have expired. Issue for root cause: https://github.com/hashicorp/terraform/issues/34322

Similar issue in the MS DevLabs extension: https://github.com/microsoft/azure-pipelines-terraform/issues/201 PR for proposed workaround in MS DevLabs extension: https://github.com/microsoft/azure-pipelines-terraform/pull/214

I have verified that the known workaround of switch the task to AzureCLI@2 (and passing env vars for authentication) fixes the issue.

Is it possible to port this workaround to this extension?

To Reproduce Steps to reproduce the behavior:

  1. Setup DevOps service connections using User Assigned Managed Identities and workload identity federation (OIDC auth)
  2. Setup pipeline with regular terraform init, plan, apply steps. Put a ManualValidation@ job in between the plan and apply.
  3. Execute pipeline, wait > 10 minutes before manually approving and continuing with the apply
  4. See error

Expected behavior I expect the apply step to run.

Screenshots If applicable, add screenshots to help explain your problem.

Pipeline Logs Include logs that help demonstrate the problem. Please make sure to redact any sensitive info such as secrets.

Agent Configuration

Additional context Add any other context about the problem here.

robbert-nlo commented 3 months ago

FYI @jaredfholgate Tagging you since I've noticed you are working on a workaround for this issue in the DevLabs extension.