velero-plugin-for-aws v1.10.x BSL unavailable while using default profile creds and bucket is in different AWS account

[mugdha-adhav]

What steps did you take and what happened: After upgrading velero-plugin-for-aws to v1.10.x from v1.9.0, BSL is unavailable with error -

message: "BackupStorageLocation \"default\" is unavailable: rpc error: code = Unknown
    desc = error fetching config from profile, default, Error using profile: \n 2,
    partial credentials found for profile default\n"
  phase: Unavailable

Note: In this case credentials.useSecret is set to false, so we expect the AWS config to be set using default profile.

Also, the bucket is in a different AWS account from the cluster (where velero is deployed).

Downgrading velero-plugin-for-aws to v1.9.0 solves the issue.

What did you expect to happen: The BSL should be available and AWS config should be set using default profile if credentials.useSecret is set to false.

The following information will help us better understand what's going on:

If you are using velero v1.7.0+:
bundle-2024-06-27-17-44-35.tar.gz

Environment:

• Velero version (use velero version): v1.14.0
• Velero features (use velero client config get features): features:
• Kubernetes version (use kubectl version): v1.28
• Kubernetes installer & version: EKS
• Cloud provider or hardware configuration: AWS
• OS (e.g. from /etc/os-release): Amazon Linux 2

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

• :+1: for "I would like to see this bug fixed as soon as possible"
• :-1: for "There are more important bugs to focus on right now"

[blackpiglet]

/area Cloud\/AWS

[reasonerjt]

This may be introduced by this change: https://github.com/vmware-tanzu/velero-plugin-for-aws/pull/191/files

@mugdha-adhav When you say "default profile" did you set the default profile via credential files? It looks like the credential file you provide is problematic. Could you clarify whether you configured IRSA in your EKS cluster?