rpc error: code = Unknown desc = ManagedIdentityCredential authentication failed. ManagedIdentityCredential authentication failed. the requested identity isn't assigned to this resource

What steps did you take and what happened:

I am trying to setup new Velero installation. I am using EntraID service principle/app registration with secret option. Deployment logs following error. I did pass clientid, secret and other necessary info. storage account is in different subscription (see my installation command below). Based on error, Velero is not picking up my clientid. But I did pass client id in credential-velero file.

time="2024-10-25T01:04:17Z" level=info msg="Validating BackupStorageLocation" backup-storage-location=velero/default controller=backup-storage- location logSource="pkg/controller/backup_storage_location_controller.go:141" time="2024-10-25T01:04:17Z" level=error msg="fail to validate backup store" backup-storage-location=velero/default controller=backup-storage-lo cation error="rpc error: code = Unknown desc = ManagedIdentityCredential authentication failed. ManagedIdentityCredential authentication failed . the requested identity isn't assigned to this resource\nGET http://169.254.169.254/metadata/identity/oauth2/token\n-------------------------- ------------------------------------------------------\nRESPONSE 400 Bad Request\n------------------------------------------------------------- -------------------\n{\n \"error\": \"invalid_request\",\n \"error_description\": \"Multiple user assigned identities exist, please specify t he clientId / resourceId of the identity in the token request\"\n}\n--------------------------------------------------------------------------- -----\nTo troubleshoot, visit https://aka.ms/azsdk/go/identity/troubleshoot#managed-id\nGET http://169.254.169.254/metadata/identity/oauth2/tok en\n--------------------------------------------------------------------------------\nRESPONSE 400 Bad Request\n------------------------------- -------------------------------------------------\n{\n \"error\": \"invalid_request\",\n \"error_description\": \"Multiple user assigned iden tities exist, please specify the clientId / resourceId of the identity in the token request\"\n}\n--------------------------------------------- -----------------------------------\nTo troubleshoot, visit https://aka.ms/azsdk/go/identity/troubleshoot#managed-id" error.file="/go/src/githu b.com/vmware-tanzu/velero/pkg/persistence/object_store.go:206" error.function="github.com/vmware-tanzu/velero/pkg/persistence.(*objectBackupSto re).IsValid" logSource="pkg/controller/backup_storage_location_controller.go:144" time="2024-10-25T01:04:17Z" level=info msg="BackupStorageLocation is invalid, marking as unavailable" backup-storage-location=velero/default co ntroller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:120" time="2024-10-25T01:04:17Z" level=error msg="Current BackupStorageLocations available/unavailable/unknown: 0/1/0, BackupStorageLocation \"defau lt\" is unavailable: rpc error: code = Unknown desc = ManagedIdentityCredential authentication failed. ManagedIdentityCredential authentication failed. the requested identity isn't assigned to this resource\nGET http://169.254.169.254/metadata/identity/oauth2/token\n------------------- -------------------------------------------------------------\nRESPONSE 400 Bad Request\n------------------------------------------------------ --------------------------\n{\n \"error\": \"invalid_request\",\n \"error_description\": \"Multiple user assigned identities exist, please sp ecify the clientId / resourceId of the identity in the token request\"\n}\n-------------------------------------------------------------------- ------------\nTo troubleshoot, visit https://aka.ms/azsdk/go/identity/troubleshoot#managed-id\nGET http://169.254.169.254/metadata/identity/oau th2/token\n--------------------------------------------------------------------------------\nRESPONSE 400 Bad Request\n------------------------ --------------------------------------------------------\n{\n \"error\": \"invalid_request\",\n \"error_description\": \"Multiple user assign ed identities exist, please specify the clientId / resourceId of the identity in the token request\"\n}\n-------------------------------------- ------------------------------------------\nTo troubleshoot, visit https://aka.ms/azsdk/go/identity/troubleshoot#managed-id)" controller=backup -storage-location logSource="pkg/controller/backup_storage_location_controller.go:178" time="2024-10-25T01:04:17Z" level=info msg="plugin process exited" backup-storage-location=velero/default cmd=/plugins/velero-plugin-for-micros oft-azure controller=backup-storage-location id=53 logSource="pkg/plugin/clientmgmt/process/logrus_adapter.go:80" plugin=/plugins/velero-plugin -for-microsoft-azure

velero install \ --provider azure \ --plugins velero/velero-plugin-for-microsoft-azure:v1.10.1 \ --bucket $BLOB_CONTAINER \ --secret-file ./credentials-velero.appreg \ --backup-location-config useAAD=true,resourceGroup=$AZURE_BACKUP_RES OURCE_GROUP,storageAccount=$AZURE_STORAGE_ACCOUNT_ID,subscriptionId=$AZURE_BACKUP_SUBSCRIPTION_ID \ --snapshot-location-config apiTimeout=$VELERO_SNAPSHOT_TIMEOUT,resourceGroup=$AZURE_BACKUP_RESOURCE_GROUP,subscriptionId=$AZURE_BACKUP_SUBSCRIPTION_ID

What did you expect to happen: backup location to be in available state.

The following information will help us better understand what's going on:

Anything else you would like to add: Instllation command:

Environment:

Velero version (use velero version): 1.14.1 (Azure Plugin 1.10.1)
Velero features (use velero client config get features):
Kubernetes version (use kubectl version):1.30.3
Kubernetes installer & version:
Cloud provider or hardware configuration:aks
OS (e.g. from /etc/os-release):

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

:+1: for "I would like to see this bug fixed as soon as possible"
:-1: for "There are more important bugs to focus on right now"

vmware-tanzu / velero

rpc error: code = Unknown desc = ManagedIdentityCredential authentication failed. ManagedIdentityCredential authentication failed. the requested identity isn't assigned to this resource #8348