Multiple user assigned identities exist, please specify the clientId / resourceId of the identity in the token request

Have you

[x] Read Troubleshooting Guide
[x] Searched on GitHub issues and Discussions

What steps did you take and what happened: We are deploying to AKS with an identity profile and added an omsagent add on profile on the aks cluster. We are also deploying a CSI Driver.

There are multiple user assigned identities on my AKS's VMSS. Despite having specified the clientId in my SecretProviderClass, I am being told to specify the clientId.

az aks show -n <name-of-cluster> -g <RG-of-cluster> --query "identity"

{
  "principalId": null,
  "tenantId": null,
  "type": "UserAssigned",
  "userAssignedIdentities": {
    "/subscriptions/REDACTED/resourceGroups/REDACTED/providers/Microsoft.ManagedIdentity/userAssignedIdentities/REDACTED": {
      "clientId": "REDACTED",
      "principalId": "REDACTED"
    }
  }
}

az vmss identity show --name <VMSSName> --resource-group <aks-ManagedRG-name>

{
  "principalId": null,
  "tenantId": null,
  "type": "UserAssigned",
  "userAssignedIdentities": {
    "/subscriptions/REDACTED/providers/Microsoft.ManagedIdentity/userAssignedIdentities/<MSI-We-assigned-in-identityProfile>": {
      "clientId": "<Value-of-clientId>",
      "principalId": "REDACTED"
    },
    "/subscriptions/REDACTED/providers/Microsoft.ManagedIdentity/userAssignedIdentities/azurepolicy-REDACTED": {
      "clientId": "REDACTED",
      "principalId": "REDACTED"
    },
    "/subscriptions/REDACTED/providers/Microsoft.ManagedIdentity/userAssignedIdentities/omsagent-REDACTED": {
      "clientId": "REDACTED",
      "principalId": "REDACTED"
    }
  }
}

SecretProviderClass.yaml

apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
  name: XYZ
spec:
  provider: azure
  parameters:
    usePodIdentity: {{ .Values.csiSecretProvider.use_pod_identity | quote }}
    useVMManagedIdentity: {{ .Values.csiSecretProvider.use_msi | quote }}
    {{- if and .Values.csiSecretProvider.use_msi .Values.cert.msi_client_id }}
    userAssignedIdentityID: {{ .Values.cert.msi_client_id | quote }}
    {{- end }}
    keyvaultName: {{ .Values.cert.keyvaultName | quote }}

    objects:  |
      array:
        - |
          objectName: {{ .Values.cert.CertName | quote }}
          objectType: secret
          objectAlias: {{ .Values.cert.pemFileName }}
    tenantId: {{ .Values.cert.tenantId | quote }}
---

The relevant values are passed via helm and ensured that the value for cert.msi_client_id is passed as the <Value-of-clientId> for the identity we assigned to the aks

helm upgrade <releaseName>
# Other values
--set cert.msi_client_id="$MsiClientId" \ # Environment Variable set by us
# other values

But the pods for our service get stuck in the ContainerCreating state and we get this error message:

Warning  FailedMount  2m8s (x56 over 108m)  kubelet  (combined from similar events): MountVolume.SetUp failed for volume "keyvault-auth-vol" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod REDACTED_POD_NAME, err: rpc error: code = Unknown desc = failed to mount objects, error: failed to get objectType:secret, objectName:REDACTED_CERT_NAME, objectVersion:: azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to https://REDACTED_KEYVAULT_NAME.vault.azure.net/secrets/REDACTED_CERT_NAME/?api-version=2016-10-01: StatusCode=400 -- Original Error: adal: Refresh request failed. Status Code = '400'. Response body: {"error":"invalid_request","error_description":"Multiple user assigned identities exist, please specify the clientId / resourceId of the identity in the token request"

What did you expect to happen: I expected the pods to mount the secret volume, and the driver to use the specified ID.

Anything else you would like to add: What's odd is that we have been using the same scripts to deploy CSI drivers, and deploy these images and those work fine without any errors. We are deploying new clusters and are facing errors with this

Which access mode did you use to access the Azure Key Vault instance: User Assigned Managed Identity

Environment:

Secrets Store CSI Driver version: (use the image tag): 0.0.19
Azure Key Vault provider version: (use the image tag): 0.0.12
Kubernetes version: (use kubectl version and kubectl get nodes -o wide): v1.24.9
Cluster type: (e.g. AKS, aks-engine, etc): AKS
Installation method: (Helm , Deployment yamls, AKS managed add-on): Helm

Azure / secrets-store-csi-driver-provider-azure

Multiple user assigned identities exist, please specify the clientId / resourceId of the identity in the token request #1133