Azure Workload Identity authentication with AD fails

[idanme-tr]

What steps did you take and what happened:

I am trying to deploy Velero Helm charts to AKS using Workload Identity. I've followed the Azure plugin guide with workload identity configurations.

For some reason, Velero cannot retrieve the storage account's properties. I've provided the managed identity with more permissions than needed to make sure I do not miss anything.

I understand that this issue might not be a bug but a misconfiguration, but I can't find what it is. When I am using Storage account key and not Workload identity it works fine.

What did you expect to happen: I expected Velero to be able to authenticate using the workload identity and to be able to backup and restore as it should.

The following information will help us better understand what's going on:

If you are using velero v1.7.0+:
Please use velero debug --backup <backupname> --restore <restorename> to generate the support bundle, and attach to this issue, more options please refer to velero debug --help

bundle-2024-10-20-11-47-04.tar.gz

If you are using earlier versions:
Please provide the output of the following commands (Pasting long output into a GitHub gist or other pastebin is fine.)

• kubectl logs deployment/velero -n velero

  time="2024-10-20T08:33:35Z" level=info msg="Validating BackupStorageLocation" backup-storage-location=velero/default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:141"
  time="2024-10-20T08:33:41Z" level=info msg="failed to retrieve the storage account properties: ManagedIdentityCredential: ManagedIdentityCredential: Get \"http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&client_id=&resource=https%3A%2F%2Fmanagement.core.windows.net%2F\": context deadline exceeded, fallback to use the default URI \"https://intaksvelerobackups.blob.core.windows.net\"" backupLocation=velero/default cmd=/plugins/velero-plugin-for-microsoft-azure controller=backup-sync logSource="/go/pkg/mod/github.com/vmware-tanzu/velero@v1.14.1/pkg/util/azure/storage.go:208" pluginName=velero-plugin-for-microsoft-azure
  time="2024-10-20T08:33:41Z" level=info msg="auth with Azure AD" backupLocation=velero/default cmd=/plugins/velero-plugin-for-microsoft-azure controller=backup-sync logSource="/go/pkg/mod/github.com/vmware-tanzu/velero@v1.14.1/pkg/util/azure/storage.go:114" pluginName=velero-plugin-for-microsoft-azure

• velero backup describe <backupname> or kubectl get backup/<backupname> -n velero -o yaml
• velero backup logs <backupname>
• velero restore describe <restorename> or kubectl get restore/<restorename> -n velero -o yaml
• velero restore logs <restorename>

Anything else you would like to add:

I am adding my Helm configurations. Lines that were commented out were different attempts but were also unsuccessful.

velero:
  configuration:
    backupStorageLocation:
      - name: default
        provider: velero.io/azure
        bucket: int-aks-we02
        config:
          storageAccount: intaksvelerobackups
          resourceGroup: int-aks-velero-backups-rg
          # subscriptionId: ***********
          # storageAccountURI: https://intaksvelerobackups.blob.core.windows.net
          # activeDirectoryAuthorityURI: https://login.microsoftonline.com/
          useAAD: "true"
    volumeSnapshotLocation:
      - name: default
        provider: velero.io/azure
        config:
          resourceGroup: MC_int-aks-we02-rg_int-aks-we02_westeurope
          # subscriptionId: ***********
          # incremental: true
          # activeDirectoryAuthorityURI: https://login.microsoftonline.com/

  credentials:
    secretContents:
      cloud: |
        AZURE_SUBSCRIPTION_ID=***********
        AZURE_RESOURCE_GROUP=MC_int-aks-we02-rg_int-aks-we02_westeurope
        AZURE_CLOUD_NAME=AzurePublicCloud

  nodeAgent:
    enabled: true

  rbac:
    create: true
    clusterAdministrator: true
    clusterAdministratorName: cluster-admin

  serviceAccount:
    server:
      create: true
      name: "int-aks-we02-velero-sa"
      annotations:
        azure.workload.identity/client-id: ***********

  initContainers:
    - name: velero-plugin-for-microsoft-azure
      image: velero/velero-plugin-for-microsoft-azure:v1.10.1
      volumeMounts:
        - mountPath: /target
          name: plugins

  podLabels:
    azure.workload.identity/use: "true"

  schedules:
    daily:
      schedule: "0 2 * * *"
      template:
        ttl: 1h0m0s
        includedNamespaces: []
        excludedNamespaces: []
        storageLocation: default

Environment:

• Velero version (use velero version):

  Client:
  Version: v1.13.2
  Git commit: -
  Server:
  Version: v1.14.1

• Velero features (use velero client config get features): features <NOT SET>

• Kubernetes version (use kubectl version):

  Client Version: v1.29.0
  Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
  Server Version: v1.30.3

• Kubernetes installer & version: AKS 1.30.3

• Cloud provider or hardware configuration: Azure

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

• :+1: for "I would like to see this bug fixed as soon as possible"
• :-1: for "There are more important bugs to focus on right now"

[anshulahuja98]

@idanme-tr, can you share the permissions you have applied? And also your BSL configuration - does it have the storageAccountUri.

[anshulahuja98]

Also I would recommend checking if you have any leftover of PodIdentity in your cluster. That often leads to issues