Velero not automatically detecting aws s3 region for buckets anymore after updating to v1.14.1

[thomaspeitz]

What steps did you take and what happened: Upgraded our velero deployment via helm chart upgrade from velero-7.1.4 to velero-7.2.1 which changes only image from velero:v1.14.0 to velero:v1.14.1.

Important we did not upgrade velero/velero-plugin-for-aws:v1.10.1 - It was always this version.

What did you expect to happen: Velero keeps working like expected and backups are working. Sadly backups were no more working.

The following information will help us better understand what's going on:

time="2024-09-09T19:34:11Z" level=error msg="Failed to determine bucket's region bucket: SECRET_BUCKET_NAME, error: operation error S3: HeadBucket, https response error StatusCode: 400, RequestID: SECRETREQUESTID, HostID: VSECRETHOSTIDA0joneyei9T2wkE0JWIpKqfYI5IzfWeJGSY1uQ==, api error BadRequest: Bad Request" backupLocation=velero/default cmd=/plugins/velero-plugin-for-aws controller=backup-sync logSource="/go/src/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:159" pluginName=velero-plugin-for-aws

Fix done by us to get it working again

# Before
configuration:
  logLevel: error
  backupStorageLocation:
  - name: default
    provider: aws
    bucket: velero-secret-bucket-name
    default: true

# After
configuration:
  logLevel: error
  backupStorageLocation:
  - name: default
    provider: aws
    bucket: velero-secret-bucket-name
    default: true
    config:
      region: eu-central-1

So basically something with the auto region detection seems to be broken. We use service accounts to access s3 api.

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

• :+1: for "I would like to see this bug fixed as soon as possible"
• :-1: for "There are more important bugs to focus on right now"

[kaovilai]

Which version combo was working? What steps are needed to setup service account credentials?

[thomaspeitz]

velero:v1.14.0 (helm chart velero-7.1.4) - was working with velero/velero-plugin-for-aws:v1.10.1

We did create the service account credentials via eksctl

# eksctl cluster.yaml
  - metadata:
      name: velero
      namespace: velero
      labels: {aws-usage: "velero"}
    attachPolicy:
      Version: "2012-10-17"
      Statement:
      - Effect: Allow
        Resource: "arn:aws:s3:::velero-some-bucket/*"
        Action:
          - s3:GetObject
          - s3:DeleteObject
          - s3:PutObject
          - s3:AbortMultipartUpload
          - s3:ListMultipartUploadParts
      - Effect: Allow
        Resource: "arn:aws:s3:::velero-some-bucket"
        Action:
          # Write Operations
          - s3:ListBucket

[kaovilai]

And I assume v1.10.0 would not work either.. since I added a fix for region hinting in 1.10.1 https://github.com/vmware-tanzu/velero-plugin-for-aws/releases/tag/v1.10.1

[kaovilai]

# Before
configuration:
  logLevel: error
  backupStorageLocation:
  - name: default
    provider: aws
    bucket: velero-secret-bucket-name
    default: true

What file format is this? I'm not familiar with configuration. backupStorageLocation

You're not creating BSL directly?