kube2iam AWS/S3 BackupStorageLocation is unavailable

[sb-tmd-dev]

What steps did you take and what happened: followed the steps at

• https://github.com/vmware-tanzu/velero-plugin-for-aws#option-2-set-permissions-using-kube2iam

and troubleshooting here:

• https://velero.io/docs/v1.6/debugging-install/#nocredentialproviders-no-valid-providers-in-chain:~:text=at%20/credentials-,Using%20kube2iam,-This%20means%20that
• https://velero.io/docs/v1.9/troubleshooting/#troubleshooting-backupstoragelocation-credentials

created a policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeVolumes",
                "ec2:DescribeSnapshots",
                "ec2:CreateTags",
                "ec2:CreateVolume",
                "ec2:CreateSnapshot",
                "ec2:DeleteSnapshot"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:DeleteObject",
                "s3:PutObject",
                "s3:AbortMultipartUpload",
                "s3:ListMultipartUploadParts"
            ],
            "Resource": [
                "arn:aws:s3:::velero-backups/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::velero-backups"
            ]
        }
    ]
}

created a role named velero-irsa attached above policy and setup with Trust Policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::<ACCOUNT_ID>:oidc-provider/oidc.eks.<AWS_REGION>.amazonaws.com/id/<REDACTED>"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "oidc.eks.<AWS_REGION>.amazonaws.com/id/<REDACTED>:sub": "system:serviceaccount:infra:velero-sa"
                }
            }
        }
    ]
}

installed in namespace infra

From values.yaml

serviceAccount:
  server:
    create: true
    name: "velero-sa"
credentials:
  # Whether a secret should be used. Set to false if using oidc sa to iam role
  useSecret: false

What did you expect to happen: BackupStorageLocation is in status available

Environment:

• helm version (use helm version): 3.8.1
• helm chart version and app version chart: velero-2.31.3 , app: 1.9.1
• Kubernetes version (use kubectl version): 1.21
• Kubernetes installer & version: eks
• Cloud provider or hardware configuration: aws
• OS (e.g. from /etc/os-release):

[KrisJohnstone]

You haven't added an annotation on the service account with the role you created.