Velero - Azure Managed Identity Issues

[MarshallKiruba]

We are trying to implement managed identity in our AKS clusters. We are using steps in below documentation to implement managed identity in our Velero pods.

https://github.com/vmware-tanzu/velero-plugin-for-microsoft-azure#option-2-use-aad-pod-identity

The steps they mentioned is for installation through Velero plugin but we are using helm charts to implement velero in our clusters.

So we followed below steps

1. Installed AAD pod identity.
2. Added aadpodidbinding label to our Velero deployment.
3. Removed service principal details from the velero secret and kept only AZURE_SUBSCRIPTION_ID, AZURE_RESOURCE_GROUP and AZURE_CLOUD_NAME secrets.
4. Restarted our Velero pods.

We are getting below error

{"backupstoragelocation":"default","controller":"backupstoragelocation","error.file":"/go/src/github.com/vmware-tanzu/velero-plugin-for-microsoft-azure/velero-plugin-for-microsoft-azure/object_store.go:147","error.function":"main.getStorageAccountKey","error.message":"rpc error: code = Unknown desc = unable to get all required environment variables: the following keys do not have values: AZURE_TENANT_ID, AZURE_CLIENT_ID, AZURE_CLIENT_SECRET","level":"error","logSource":"pkg/controller/backupstoragelocation_controller.go:87","msg":"Error getting a backup store","time":"2021-08-30T10:30:08Z"}

Please help us fix this issue.

[jenting]

Sorry, currently I don't have the Azure account to test it. Probably we need to ask other community users about this issue.

[MarshallKiruba]

@jenting Thanks for your response.

Do I need to post about this issue anywhere else as well?

[jenting]

Perhaps you could check the GitHub discussion on the vmware-tanzu/velero repo.

[badalk]

Did you try opaque secret for the AZURE_TENANT_ID, AZURE_CLIENT_ID, and i guess subscription id Its clearly not able to get those values from the k8s secret. Look at the velero logs or even azure pod identity pods logs (nmi and mmi pods)

Regards Badal

[tdevout]

Hi everyone,

I'm facing persistent issues when trying to connect a Managed Identity to AKS and Velero. Despite trying several solutions, I'm encountering errors like 404 and 403.

ERROR:

time="2024-09-08T22:13:46Z" level=error msg="Current BackupStorageLocations available/unavailable/unknown: 0/0/1, BackupStorageLocation \"default\" is unavailable: rpc error: code = Unknown desc = azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to https://management.azure.com/.../storageAccounts/velero/listKeys?%24expand=kerb&api-version=2019-06-01: StatusCode=404 -- Original Error: adal: Refresh request failed. Status Code = '404'. Response body: clientID in request: REDACTED, getting assigned identities for pod velero/velero-7d8f4f5996-ghsfs in CREATED state failed after 16 attempts, retry duration [5]s, error: . Check MIC pod logs for identity assignment errors\n Endpoint http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&client_id=REDACTED&resource=https%3A%2F%2Fmanagement.azure.com%2F"

NOTE: There's no error in MIC pods.

Has anyone successfully resolved this issue? Any guidance or recommended steps to troubleshoot would be greatly appreciated!

Thanks in advance!