Quickstart pod is failing to fetch the secret from the keyvault configured

[deeksha345]

Describe the bug I am getting the following log from the quick start pod: PS C:\Users\deesharma> kubectl logs quick-start E0514 00:40:13.289601 1 main.go:60] "failed to get secret from keyvault" err=< FromAssertion(): http call(https://login.microsoftonline.com/72f988bf-86f1-41af-91ab-2d7cd011db47/oauth2/v2.0/token)(POST) error: reply status code was 401: {"error":"invalid_client","error_description":"AADSTS700211: No matching federated identity record found for presented assertion issuer 'https://oidcdiscovery-northamerica-endpoint-gbcge4adgqebgxev.z01.azurefd.net/8797542c-e7da-4e1d-83ba-2eba338862be'. Please check your federated identity credential Subject, Audience and Issuer against the presented assertion. https://docs.microsoft.com/en-us/azure/active-directory/develop/workload-identity-federation Trace ID: 6e11a1f9-3492-4ac1-9251-4561eb284c00 Correlation ID: f76fcf1e-c45e-4cd8-91a3-df20fd45b15c Timestamp: 2024-05-14 00:40:08Z","error_codes":[700211],"timestamp":"2024-05-14 00:40:08Z","trace_id":"6e11a1f9-3492-4ac1-9251-4561eb284c00","correlation_id":"f76fcf1e-c45e-4cd8-91a3-df20fd45b15c"}

keyvault="https://wife2e-kv.vault.azure.net/" secretName="wife2e-secret"

Discovery doc and keys for the cluster are properly stored at: https://oidcdiscovery-northamerica-endpoint-gbcge4adgqebgxev.z01.azurefd.net/8797542c-e7da-4e1d-83ba-2eba338862be/.well-known/openid-configuration and https://oidcdiscovery-northamerica-endpoint-gbcge4adgqebgxev.z01.azurefd.net/8797542c-e7da-4e1d-83ba-2eba338862be/openid/v1/jwks

The open id docs that AAD requires for authn are properly stored at the issuer url. Where is the token that gets mounted to the quick start pod stored? If I can check the token issuer to ensure it matches the cluster's issuer url then that will help me troubleshoot the issue further.

My FIC: { "audiences": [ "api://AzureADTokenExchange" ], "id": "/subscriptions/069b2617-846e-4b82-ad41-5d097465478b/resourcegroups/wirepro/providers/Microsoft.ManagedIdentity/userAssignedIdentities/wife2e-uami/federatedIdentityCredentials/wife2e-fic-testrun2", "issuer": "https://oidcdiscovery-northamerica-endpoint-gbcge4adgqebgxev.z01.azurefd.net/8797542c-e7da-4e1d-83ba-2eba338862be/", "name": "wife2e-fic-testrun2", "resourceGroup": "wirepro", "subject": "system:serviceaccount:default:wife2e-sa", "systemData": null, "type": "Microsoft.ManagedIdentity/userAssignedIdentities/federatedIdentityCredentials" }

Steps To Reproduce I am testing this scenario on an Arc Connected Cluster setting up the workload identity feature and webhook using the following steps: https://msazure.visualstudio.com/One/_wiki/wikis/One.wiki/487139/-HOW-TO-Enable-Workload-Identity-for-Arc-enabled-K8s-Clusters?anchor=new-connected-clsuter

Expected behavior The quick start pod is deployed and properly able to fetch the key vault secret.

Logs

Environment A Kind k8s cluster, connected to Azure Arc

• Kubernetes version (use kubectl version):
• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Network plugin and version (if this is a network-related bug):
• Others:

Additional context

[aramase]

The issuer you have configured in FIC has a trailing slash which seems to be incorrect. The issuer that's in your token is returned as part of the error you posted - assertion issuer 'https://oidcdiscovery-northamerica-endpoint-gbcge4adgqebgxev.z01.azurefd.net/8797542c-e7da-4e1d-83ba-2eba338862be'. and as you can see there is no trailing slash in the issuer you have configured in the cluster. The --service-account-issuer configured in the cluster will need to exactly match the value configured in FIC. This is part of troubleshooting guide: https://azure.github.io/azure-workload-identity/docs/troubleshooting.html#aadsts70021-no-matching-federated-identity-record-found-for-presented-assertion.

The open id docs that AAD requires for authn are properly stored at the issuer url. Where is the token that gets mounted to the quick start pod stored? If I can check the token issuer to ensure it matches the cluster's issuer url then that will help me troubleshoot the issue further.

The issuer in the token is returned as part of the error. The token file path is configured as an env var AZURE_FEDERATED_TOKEN_FILE (ref: https://azure.github.io/azure-workload-identity/docs/quick-start.html#7-deploy-workload).

[deeksha345]

I was able to test successfully once I added the trailing slash! Didn't mean to spend your time for a simple spelling issue!