ManagedIdentityCredential authentication failed

[ardixit-msft-la]

Which component are you using?:

capz-controller-manager

What version of the component are you using?:

Component version: v1.15.2

What k8s version are you using (kubectl version)?:

kubectl version Output

$ kubectl version

Client Version: v1.29.2 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.28.9

What environment is this in?:

Azure

What did you expect to happen?:

As managed user identity has contributor accesses at subscription level, the expectation was to see access to azure resources without auth error.

What happened instead?:

Hitting following error: failed to get user-assigned identity ClientID: ManagedIdentityCredential authentication failed GET http://169.254.169.254/metadata/identity/oauth2/token

    RESPONSE 400 Bad Request
    --------------------------------------------------------------------------------
    {
      "error": "invalid_request",
      "error_description": "Identity not found"
    }
    -------

How to reproduce it (as minimally and precisely as possible):

How to reproduce it (as minimally and precisely as possible):

1. Create a user assigned managed identity with contributor access to the subscription where workload cluster and management cluster will be created.
2. Create AKS managed cluster to be used as management cluster with user assigned managed identity created in step 1.
3. Configure the AKS managed cluster to be used as management cluster with authentication mechanism of user managed identity based on the steps provided https://capz.sigs.k8s.io/topics/identities and https://cluster-api.sigs.k8s.io/user/quick-start.html
4. Try creating the workload cluster with attached template yaml. cluster-template-userassigned.txt

5. Monitor the capz-controller-manager logs for following error: failed to get user-assigned identity ClientID: ManagedIdentityCredential authentication failed GET http://169.254.169.254/metadata/identity/oauth2/token

   RESPONSE 400 Bad Request
   --------------------------------------------------------------------------------
   {
     "error": "invalid_request",
     "error_description": "Identity not found"
   }
   --------------------------------------------------------------------------------
   To troubleshoot, visit https://aka.ms/azsdk/go/identity/troubleshoot#managed-id