ManagedIdentityCredential authentication failed

[ardixit-msft-la]

Which component are you using?:

capz-controller-manager

What version of the component are you using?:

Component version: v1.15.2

What k8s version are you using (kubectl version)?:

kubectl version Output

$ kubectl version

Client Version: v1.29.2 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.28.9

What environment is this in?:

Azure

What did you expect to happen?:

As managed user identity has contributor accesses at subscription level, the expectation was to see access to azure resources without auth error.

What happened instead?:

Hitting following error: failed to get user-assigned identity ClientID: ManagedIdentityCredential authentication failed GET http://169.254.169.254/metadata/identity/oauth2/token

    RESPONSE 400 Bad Request
    --------------------------------------------------------------------------------
    {
      "error": "invalid_request",
      "error_description": "Identity not found"
    }
    -------

How to reproduce it (as minimally and precisely as possible):

How to reproduce it (as minimally and precisely as possible):

1. Create a user assigned managed identity with contributor access to the subscription where workload cluster and management cluster will be created.
2. Create AKS managed cluster to be used as management cluster with user assigned managed identity created in step 1.
3. Configure the AKS managed cluster to be used as management cluster with authentication mechanism of user managed identity based on the steps provided https://capz.sigs.k8s.io/topics/identities and https://cluster-api.sigs.k8s.io/user/quick-start.html
4. Try creating the workload cluster with attached template yaml. cluster-template-userassigned.txt

5. Monitor the capz-controller-manager logs for following error: failed to get user-assigned identity ClientID: ManagedIdentityCredential authentication failed GET http://169.254.169.254/metadata/identity/oauth2/token

   RESPONSE 400 Bad Request
   --------------------------------------------------------------------------------
   {
     "error": "invalid_request",
     "error_description": "Identity not found"
   }
   --------------------------------------------------------------------------------
   To troubleshoot, visit https://aka.ms/azsdk/go/identity/troubleshoot#managed-id

[voelzmo]

ping @elmiko this seems related to cluster-api, could you maybe have a look? Thanks!

[elmiko]

good spot @voelzmo , this definitely seems related to cluster-api.

@jackfrancis you might have interest in this given the capz callout.

[jackfrancis]

Agree w/ @elmiko, this is either an issue in the Azure cluster-autoscaler provider, or w/ the CAPZ AKS flow

cc @nojnhuh @willie-yao

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten