AADSTS90061 error attempting to acquire token

[markphillips100]

Getting a 400 error with an undocumented AADSTS90061 error code when attempting to acquire a token from AAD:

Azure.Identity.AuthenticationFailedException: ClientAssertionCredential authentication failed: AADSTS90061: Request to External OIDC endpoint failed.
Trace ID: 86c81de3-efbd-4c0f-854d-cb8f329e5b00
Correlation ID: a92568a5-cc37-4136-ad4d-1ba82988abc2
Timestamp: 2022-10-21 04:36:48Z
 ---> MSAL.NetCore.4.39.0.0.MsalServiceException:
        ErrorCode: invalid_request
Microsoft.Identity.Client.MsalServiceException: AADSTS90061: Request to External OIDC endpoint failed.
Trace ID: 86c81de3-efbd-4c0f-854d-cb8f329e5b00
Correlation ID: a92568a5-cc37-4136-ad4d-1ba82988abc2
Timestamp: 2022-10-21 04:36:48Z
   at Microsoft.Identity.Client.Internal.Requests.RequestBase.HandleTokenRefreshErrorAsync(MsalServiceException e, MsalAccessTokenCacheItem cachedAccessTokenItem)
   at Microsoft.Identity.Client.Internal.Requests.ClientCredentialRequest.ExecuteAsync(CancellationToken cancellationToken)
   at Microsoft.Identity.Client.Internal.Requests.RequestBase.RunAsync(CancellationToken cancellationToken)
   at Microsoft.Identity.Client.ApiConfig.Executors.ConfidentialClientExecutor.ExecuteAsync(AcquireTokenCommonParameters commonParameters, AcquireTokenForClientParameters clientParameters, CancellationToken cancellationToken)
   at Azure.Identity.AbstractAcquireTokenParameterBuilderExtensions.ExecuteAsync[T](AbstractAcquireTokenParameterBuilder`1 builder, Boolean async, CancellationToken cancellationToken)
   at Azure.Identity.MsalConfidentialClient.AcquireTokenForClientCoreAsync(String[] scopes, String tenantId, Boolean async, CancellationToken cancellationToken)
   at Azure.Identity.MsalConfidentialClient.AcquireTokenForClientAsync(String[] scopes, String tenantId, Boolean async, CancellationToken cancellationToken)
   at Azure.Identity.ClientAssertionCredential.GetTokenAsync(TokenRequestContext requestContext, CancellationToken cancellationToken)
        StatusCode: 400
        ResponseBody: {"error":"invalid_request","error_description":"AADSTS90061: Request to External OIDC endpoint failed.\r\nTrace ID: 86c81de3-efbd-4c0f-854d-cb8f329e5b00\r\nCorrelation ID: a92568a5-cc37-4136-ad4d-1ba82988abc2\r\nTimestamp: 2022-10-21 04:36:48Z","error_codes":[90061],"timestamp":"2022-10-21 04:36:48Z","trace_id":"86c81de3-efbd-4c0f-854d-cb8f329e5b00","correlation_id":"a92568a5-cc37-4136-ad4d-1ba82988abc2"}
...

Steps To Reproduce

1. Followed steps for self-managed clusters in docs, including the KinD cluster key setup.
2. Created user managed identity, federated identity, etc.
3. Verified that deployed pod with service account has the AZURE environment values provided and azure-identity-token mount.
4. Pod attempts to use cosmos via the DefaultAzureCredential and the above exception occurs.

Expected behavior Expected an AAD token to be made available to cosmos client.

Logs No errors in webhook but then seems no issue supplying the env and mount to pod anyway.

Environment

Kubernetes version Client Version: v1.24.0 Server Version: v1.23.5

Not using proxy sidecar annotation as I assume that's only required if migrating from aad-pod-identity.

[markphillips100]

Big oversight on my part as I didn't actually persist the jwks doc, only the openid-configuration json. Having said that, anybody know what the AADSTS90061 code means? It's not documented here.

[udayxhegde]

Azure AD is not able to access the necessary details to validate the token: you need all the aspects of OIDC endpoint available (such as jwks), and the error here seems to indicate that Azure AD is not able to find the necessary information

[markphillips100]

Thank you @udayxhegde, and @aramase for adding to troubleshooting.