AADSTS700211 invalid_client

[hasland]

Describe the bug I'm getting the error AADSTS700211: WorkloadIdentityCredential: Microsoft Entra ID error '(invalid_client) AADSTS700211: No matching federated identity record found for presented assertion issuer 'https://oidc.prod-aks.azure.com/879-...-a6425/'. Please check your federated identity credential Subject, Audience and Issuer against the presented assertion. https://docs.microsoft.com/en-us/azure/active-directory/develop/workload-identity-federation Trace ID: REDACTED Correlation ID: REDACTED Timestamp: 2023-12-08 16:17:27Z'

Steps To Reproduce

Deploy Workload Identity v1.2.0 inside an AKS cluster v1.27.7

Create an AAD Service Principal and a k8s service account using the azwi cli:

azwi serviceaccount create \
  --service-account-name "saName" \
  --skip-phases role-assignment \
  --subscription-id "<SUB_ID>" \
  --aad-application-name "<AAD_APP_NAME>" \
  --service-account-issuer-url "https://oidc.prod-aks.azure.com/879-...-a6425/" \
  --service-account-namespace <K8S_NAMESPACE>

Create a pod associating the SA and with a label azure.workload.identity/use: "true"

Trying to authenticate and get a Key Vault secret using Python 3.10 with DefaultAzureCredential(), using azure-identity==1.15.0 lib

Expected behavior

Get the secret value.

Logs

WorkloadIdentityCredential: Microsoft Entra ID error '(invalid_client) AADSTS700211: No matching federated identity record found for presented assertion issuer 'https://oidc.prod-aks.azure.com/879-...-6425/'. Please check your federated identity credential Subject, Audience and Issuer against the presented assertion. https://docs.microsoft.com/en-us/azure/active-directory/develop/workload-identity-federation Trace ID: REDACTED Correlation ID: REDACTED Timestamp: 2023-12-08 16:55:37Z'

Environment

• Kubernetes version (use kubectl version): AKS 1.27.7
• Cloud provider or hardware configuration: Azure Public Cloud
• OS (e.g: cat /etc/os-release): Ubuntu 20.04.6 LTS
• Kernel (e.g. uname -a): 5.15.0-1051-azure #59-Ubuntu SMP Wed Oct 11 18:49:16 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
• Install tools:
• Network plugin and version (if this is a network-related bug):
• Others:

Additional context

AAD Service Principal is created without problems:

Fri, 08 Dec 2023 13:11:16 -03  role-assignment  workflow/runner.go:133  skipping phase
Fri, 08 Dec 2023 13:11:20 -03  aad-application  create/aadapplication.go:71  created an AAD application  {"name": "$AAD_NAME", "clientID": "d861-...-7de", "objectID": "4e11-...-8eec"}
Fri, 08 Dec 2023 13:11:20 -03  serviceaccount/create.go:175  --service-principal-name not specified, falling back to AAD application name  {"warning": true}
Fri, 08 Dec 2023 13:11:21 -03  aad-application  create/aadapplication.go:95  created service principal  {"name": "$AAD_NAME", "clientID": "d861-..-7de", "objectID": "9e0-..-9512"}
Fri, 08 Dec 2023 13:11:22 -03  service-account  create/serviceaccount.go:94  created kubernetes service account  {"namespace": "$K8S_NAMESPACE", "name": "$K8S_SA_NAME"}
Fri, 08 Dec 2023 13:11:24 -03  federated-identity  create/federatedidentitycredential.go:96  added federated credential  {"objectID": "4e11-...-8eec", "subject": "system:serviceaccount:$K8S_NAMESPACE:$K8S_SA_NAME"}

Name:                REDACTED
Namespace:           REDACTED
Labels:              <none>
Annotations:         azure.workload.identity/client-id: d861-...-7de
                     azure.workload.identity/tenant-id: 73d-...-26f
Image pull secrets:  <none>
Mountable secrets:   <none>
Tokens:              <none>
Events:              <none>

OIDC: az aks show -n $AKS_NAME -g $AKS_RG --query "oidcIssuerProfile.issuerUrl" -otsv https://oidc.prod-aks.azure.com/879-...-a6425/

On AAD App Registration I can see all the values populated correctly (issuer, namespace, service account name).

[shubham2110]

+1

[aramase]

Have you looked at the troubleshooting guide: https://azure.github.io/azure-workload-identity/docs/troubleshooting.html#aadsts70021-no-matching-federated-identity-record-found-for-presented-assertion?

Check exact match for following in the Azure AD App:

1. Issuer URL
2. Service account name and namespace; this is the subject field in FIC system:serviceaccount:<sa namespace>:<sa name>.

[hasland]

@aramase Yep, all the data in Azure AD App has the right values (issuer URL, service account, namespace). azwi populates it with the parameters that we pass to it, we do not populate it manually. We checked N times, recreated the service account and the Azure AD app, but the problem persists.

Azure AAD App:

Service account:

[aeb-dev]

Having the same issue.

I am trying the following docs:

https://learn.microsoft.com/en-us/azure/application-gateway/for-containers/quickstart-deploy-application-gateway-for-containers-alb-controller?tabs=install-helm-windows

https://learn.microsoft.com/en-us/azure/application-gateway/for-containers/quickstart-create-application-gateway-for-containers-managed-by-alb-controller?tabs=new-subnet-aks-vnet

The result is the same error. I can see the resources on azure-portal but I am not sure what the problem is.

[borqosky]

having the same issue with flux addon and workload identity on AKS

[Orbifoldt]

I ran into the same error message (without using the azwi cli) while using the Azure Secret store CSI driver, and for the life of me I couldn't figure out what was going on. Finally, I checked the secret store provider logs, and it turned out that I was deploying with the wrong keyvault configured 😮‍💨 . So if someone encounters this error you might want to double check your resource / authorization configuration

[chrismilson]

I had the same issue; restarting the keda-controller fixed the issue.

[veerun14]

same issue here. any luck? Issuer URL (with the trailing slash), subject and audience all look good. Still hitting this error

[veerun14]

adding correlation IDs if that helps for debuggin on your side

Content: {"error":"invalid_client","error_description":"AADSTS700211: No matching federated identity record found for presented assertion issuer 'https://eastus2.oic.prod-aks.azure.com/7******************/c*****/'. Please check your federated identity credential Subject, Audience and Issuer against the presented assertion. https://docs.microsoft.com/en-us/azure/active-directory/develop/workload-identity-federation Trace ID: 93eb6af6-43b5-4b16-ad6e-63cf2a050b00 Correlation ID: 8f043819-8577-46b9-b330-a72e2e114665 Timestamp: 2024-03-01 07:41:07Z","error_codes":[700211],"timestamp":"2024-03-01 07:41:07Z","trace_id":"93eb6af6-43b5-4b16-ad6e-63cf2a050b00","correlation_id":"8f043819-8577-46b9-b330-a72e2e114665"}

[veerun14]

This works if I use user-assigned managed identity only and not with an AAD app. Is this communicated anywhere?

I just created a user-assigned managed identity and add a federated credential to that MI and was able to get a credential. No more errors. In fact I was able to access the target azure resource as well (KV in my case)

[avarnon]

We are encountering the same issue as the original poster. I have validated the federated credential against the JWT token mounted to the pods and everything seems to match. Any updates?

{
  "aud": [
    "api://AzureADTokenExchange"
  ],
  "exp": 1710782177,
  "iat": 1710778577,
  "iss": "https://eastus2.oic.prod-aks.azure.com/yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyyyyyy/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/",
  "kubernetes.io": {
    "namespace": "my-namespace",
    "pod": {
      "name": "test-pod",
      "uid": "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa"
    },
    "serviceaccount": {
      "name": "my-service-account",
      "uid": "bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb"
    }
  },
  "nbf": 1710778577,
  "sub": "system:serviceaccount:my-namespace:my-service-account"
}

[matthchr]

I was also having this same problem. It took me longer than I care to admit to realize that I had a trailing \n character in the FIC issuer which the portal was not displaying. It was only visible in the raw JSON of the FederatedIdentityCredential.

Once I fixed that, everything was working well.

@aramase - wondering if these other issues might be a similar issue in either the issuer or subject identifier section... do we have a contact on the identity team who could verify if/why they even accept newlines in these fields? It seems like a bit of a foot-gun.

[aramase]

@aramase - wondering if these other issues might be a similar issue in either the issuer or subject identifier section... do we have a contact on the identity team who could verify if/why they even accept newlines in these fields? It seems like a bit of a foot-gun.

Have started a thread internally to discuss catching such errors as part of API validation. I'll update this thread once we hear back from the Entra team.

[scschneider]

This has been a complete time suck trying to figure out what the issue was here and happy to have landed on this post with the appropriate search terms. As stated above, there is a problem with line endings. In my situation, \r was added to the end.

I get my OIDC Issuer from the following command:

az aks show --name "${CLUSTER_NAME}" \
    --resource-group "${RESOURCE_GROUP}" \
    --query "oidcIssuerProfile.issuerUrl" \
    --output tsv

The above output is stored as a variable and then used in the command to create the federated credential:

az identity federated-credential create --name ${FEDERATED_IDENTITY_CREDENTIAL_NAME} \
    --identity-name "${USER_ASSIGNED_IDENTITY_NAME}" \
    --resource-group "${RESOURCE_GROUP}" \
    --issuer "${AKS_OIDC_ISSUER}" \
    --subject "system:serviceaccount:${SERVICE_ACCOUNT_NAMESPACE}:${SERVICE_ACCOUNT_NAME}" \
    --audience api://AzureADTokenExchange

The output from creating the federated credential shows a line ending:

{
  "audiences": [
    "api://AzureADTokenExchange"
  ],
  "id": "<FIC_RESOURCE_ID>",
  "issuer": "https://westus3.oic.prod-aks.azure.com/aaaaaaaa-86f1-41af-91ab-aaaaaaaaaaaa/bbbbbbbb-0443-47ec-b4c5-bbbbbbbbbbbb/\r",
  "name": "<FIC_NAME>",
  "resourceGroup": "<RESOURCE_GROUP>",
  "subject": "system:serviceaccount:default:workload-identity-sa",
  "systemData": null,
  "type": "Microsoft.ManagedIdentity/userAssignedIdentities/federatedIdentityCredentials"
}

Edit I have been able to remove any special line endings in my bash deployment scripts using //[$'\t\r\n ']

Ref: https://stackoverflow.com/questions/19345872/how-to-remove-a-newline-from-a-string-in-bash

[andrescodas]

One more victim of the trailing \r character bumping up this issue to get fixed... Thanks for reporting!