ChainedTokenCredential: Filed to acquire a token

[argonorth]

Hi team,

A few months ago, I noticed AzAPI authentication became very unstable, exhibiting intermittent behavior during the authentication process. The error occurs during the Terraform Plan phase. Lately, the failure ratio during authentication has increased significantly, reaching around 95%.

I'm currently using authentication based on a Service Principal The issue occurs in any version greater than 1.11, including the new 1.13.1.

As a workaround, I downgraded the AzApi version to 1.11. It's now working flawlessly.

[ms-henglu]

Hello @argonorth ,

Thank you for taking time to report this!

Which way are you using to authenticate with Azure? If you're not using the managed identity, you could turn it off by setting "use_msi = false" in the provider block. I'll keep investigating the root cause for this issue.

[argonorth]

Hey @ms-henglu I am currently using Service Principal and Client Secret authentication. When using version 1.13.1 and setting use-msi to false, I receive the following error.

provider "azapi" {
  use_msi = false
}

[ms-henglu]

Would you please share the debug logs?

Here's how:

1. Add the following two environment variables: TF_LOG=DEBUG TF_LOG_PATH=./terraform.log
2. Run terraform plan command, the logs will be save to the terraform.log file.

[argonorth]

Hey @ms-henglu,

I can't share the entire terraform.log file. The issue is similar to https://github.com/Azure/terraform-provider-azapi/issues/475. However, upgrading to versions 1.13.1 and 1.14 did not resolve the problem. It worth mentioning that the error is intermittent, and the error messages keep changing as if the application is randomly picking authentication methods.

2024-07-10T13:45:30.058Z [INFO]  CLI command args: []string{"plan", "-var-file=../projects/p4yv2/variables/sit.tfvars", "-out=tfplan", "-input=false"}
2024-07-10T13:45:30.070Z [INFO]  Testing if Service Principal / Client Certificate is applicable for Authentication..
2024-07-10T13:45:30.070Z [INFO]  Testing if Multi Tenant Service Principal / Client Secret is applicable for Authentication..
2024-07-10T13:45:30.070Z [INFO]  Testing if Service Principal / Client Secret is applicable for Authentication..
2024-07-10T13:45:30.070Z [INFO]  Using Service Principal / Client Secret for Authentication
2024-07-10T13:45:30.070Z [INFO]  Getting OAuth config for endpoint https://login.microsoftonline.com/ with  tenant  a91e3dd4-d85d-48e2-a591-3017cb57b4b8
2024-07-10T13:45:30.070Z [DEBUG] Obtaining an MSAL / Microsoft Graph token for Resource Manager..
2024-07-10T13:45:30.070Z [DEBUG] Obtaining an MSAL / Microsoft Graph token for Storage..
2024-07-10T13:45:31.665Z [DEBUG] checking for provisioner in "."
2024-07-10T13:45:31.665Z [DEBUG] checking for provisioner in "/azp/_work/_tool/terraform/1.5.7/x64"
2024-07-10T13:45:31.669Z [INFO]  backend/local: starting Plan operation
2024-07-10T13:45:31.852Z [DEBUG] Azure Backend Request: 

2024-07-10T11:51:20.642Z [DEBUG] provider.terraform-provider-azapi_v1.11.0: Jul 10 11:51:20.641972 Retry: non-retriable error *azidentity.AuthenticationFailedError: timestamp=2024-07-10T11:51:20.641Z
2024-07-10T11:51:20.643Z [ERROR] vertex "module.app_services_xxx[\"xxx\"].azapi_update_resource.azurerm_linux_web_app[\"xxx\"]" error: reading "Resource: (ResourceId \"/subscriptions/xxx/resourceGroups/xxx/providers/Microsoft.Web/sites/xxx\" / Api Version \"2022-09-01\")": ChainedTokenCredential authentication failed
GET http://169.254.169.254/metadata/identity/oauth2/token
--------------------------------------------------------------------------------
RESPONSE 400 Bad Request
--------------------------------------------------------------------------------
{
  "error": "invalid_request",
  "error_description": "Identity not found"
}
--------------------------------------------------------------------------------

[argonorth]

I thoroughly analyzed the logs and identified a specific issue related to the tenant ID. It appears that the service is consistently failing to recognize the tenant ID, which may be causing disruptions or errors in its operation.

provider.terraform-provider-azapi_v1.11.0: newDefaultAzureCredential failed to initialize environment credential:

[ms-henglu]

Hi @argonorth - would you please try the latest azapi v1.14.0? Thanks

[ms-henglu]

I'll close this issue as it's resolved, but feel free to reopen it if there's any question.

[LiamLeane]

@ms-henglu Confirming this is still an issue on 1.15, exactly the same error

2024-08-20T20:29:23.022Z [DEBUG] provider.terraform-provider-azapi_v1.15.0: [DEBUG] failed to initialize client secret credential: invalid tenantID. You can locate your tenantID by following the instructions listed here: https://learn.microsoft.com/partner-center/find-ids-and-domain-names

Its in an existing workspace with AzureRM which is working as expected with tenant/subscription/client id/client secret passed via environment variable.

[ms-henglu]

Hi @LiamLeane - Please confirm that the tenant ID is passed via environment variable: ARM_TENANT_ID.

[CorrenSoft]

Having a similar error, when creating a nodepool

│ checking for presence of existing Resource: (ResourceId
│ "/subscriptions/***/resourceGroups/***/agentPools/***"
│ / Api Version "2023-01-01"): ChainedTokenCredential authentication failed
│ GET http://169.254.169.254/metadata/identity/oauth2/token
│ --------------------------------------------------------------------------------
│ RESPONSE 400 Bad Request
│ --------------------------------------------------------------------------------
│ {
│   "error": "invalid_request",
│   "error_description": "Identity not found"
│ }
│ --------------------------------------------------------------------------------

azAPI v 1.15.0 Using a Managed Identity with federated credentials, and deploying from Github workflows.