Failure to get token (keyvault client)

[64J0]

Have you

• [x] Read Troubleshooting Guide
• [x] Read Known Issues
• [x] Searched on GitHub issues

Describe the bug

The POD that's trying to get the identity is throwing this event:

Warning  FailedMount  15s (x6 over 7m11s)   kubelet            MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod namespace/server-aaaa-bbb, err: rpc error: code = Unknown desc = failed to mount objects, error: failed to get keyvault client: failed to get authorizer for keyvault client: nmi response failed with status code: 404, response body: getting assigned identities for pod namespace/server-aaaa-bbb in CREATED state failed after 16 attempts, retry duration [5]s, error: <nil>. Check MIC pod logs for identity assignment errors

In MIC I get this kind of log:

I1201 14:23:27.132710       1 mic.go:648] uai-name identity not found when using default/azure-pod-identity-binding binding

From the NMI:

I1201 14:23:04.442952       1 server.go:239] status (404) took 80008623829 ns for req.method=GET reg.path=/host/token/ req.remote=127.0.0.1
E1201 14:24:32.490281       1 server.go:281] failed to get identities, error: getting assigned identities for pod namespace/server-aaaa-bbb in CREATED state failed after 16 attempts, retry duration [5]s, error: <nil>. Check MIC pod logs for identity assignment errors
I1201 14:24:32.490357       1 server.go:239] status (404) took 80019682776 ns for req.method=GET reg.path=/host/token/ req.remote=127.0.0.1
E1201 14:26:08.549508       1 server.go:281] failed to get identities, error: getting assigned identities for pod namespace/server-aaaa-bbb in CREATED state failed after 16 attempts, retry duration [5]s, error: <nil>. Check MIC pod logs for identity assignment errors
I1201 14:26:08.549556       1 server.go:239] status (404) took 80031191001 ns for req.method=GET reg.path=/host/token/ req.remote=127.0.0.1
I1201 14:26:21.508445       1 server.go:413] exception pod kube-system/ama-logs-sgpfz token handling
I1201 14:26:21.508477       1 server.go:356] fetching token for user assigned MSI for resource: https://monitoring.azure.com/
I1201 14:26:21.635385       1 server.go:239] status (200) took 127024916 ns for req.method=GET reg.path=/metadata/identity/oauth2/token req.remote=10.1.0.217
E1201 14:28:00.696492       1 server.go:281] failed to get identities, error: getting assigned identities for pod namespace/server-aaaa-bbb in CREATED state failed after 16 attempts, retry duration [5]s, error: <nil>. Check MIC pod logs for identity assignment errors
I1201 14:28:00.696541       1 server.go:239] status (404) took 80008526916 ns for req.method=GET reg.path=/host/token/ req.remote=127.0.0.1

Steps To Reproduce

Expected behavior

AAD Pod Identity version

pod-identity    kube-system     1               2022-12-01 11:12:55.70607755 -0300 -03  deployed        aad-pod-identity-4.1.14                 1.8.13

Kubernetes version

• v1.23.12

Additional context

[64J0]

I'm not sure why, but after installing the configuration again I started getting the same error from the MIC:

I1201 15:14:15.670757       1 mic.go:648] uai-account identity not found when using default/azure-pod-identity-binding binding

The pod-identity resource looks like:

apiVersion: aadpodidentity.k8s.io/v1
kind: AzureIdentity
metadata:
  name: "uai-aks" 
spec:
  type: 0                                 
  resourceID: "subscription-path"
  clientID: "client-id"
---
apiVersion: aadpodidentity.k8s.io/v1
kind: AzureIdentityBinding
metadata:
  name: azure-pod-identity-binding
spec:
  azureIdentity: "uai-aks-staging"
  selector: "azure-pod-identity-binding-selector"

[64J0]

Noticed that the problem is that the azureIdentity in the AzureIdentityBinding resource was wrong.

[64J0]

After fixing it, my error is now:

failed to generate identity assignment state, error: failed to get a list of user-assigned identites from node aks-generic-1 [...]

[64J0]

I'll close this issue since it's related to another problem.