Bucket creation in recently opened AWS regions fails

[anothertobi]

Description

For recently opened AWS regions, bucket creations through restic fail with a 400 Bad Request during the BucketExists check.

Additional Context

minio-go defaults to the s3.dualstack.us-east-1.amazonaws.com endpoint when an unknown region is provided. The authorization header however still contains the provided, unknown, region. This results in a 400 Bad Request when calling the S3 API.

New regional S3 endpoints were added to minio-go in https://github.com/minio/minio-go/pull/1726 and released with version v7.0.44 (makes eu-central-2 a "known" region). This version was included in restic in https://github.com/restic/restic/commit/f4d3ed77c458af6fbef2be7fe0acfb4bf4fa509a and released with version v0.15.0.

K8up currently uses github.com/restic/restic v0.14.0 and already has a PR pending to upgrade restic to v0.16.0 https://github.com/k8up-io/k8up/pull/796 (fixes the issue).

Logs

68d0d8e880ce:/$ k8up restic
2023-10-02T13:40:04Z    INFO    k8up    Starting k8up…  {"version": "2.7.1", "date": "2023-05-12T12:32:54Z", "commit": "81d9c2bd7970208bcb0f84f7141d7716177b92cb", "go_os": "linux", "go_arch": "arm64", "go_version": "go1.19.9", "uid": 65532, "gid": 0}
2023-10-02T13:40:04Z    INFO    k8up.restic initializing
2023-10-02T13:40:04Z    INFO    k8up.restic setting up a signal handler
2023-10-02T13:40:04Z    INFO    k8up.restic.restic  using the following restic options  {"options": [""]}
2023-10-02T13:40:04Z    INFO    k8up.restic.restic.RepoInit.command restic command  {"path": "/usr/local/bin/restic", "args": ["init", "--option", ""]}
2023-10-02T13:40:04Z    INFO    k8up.restic.restic.RepoInit.command Defining RESTIC_PROGRESS_FPS    {"frequency": 0.016666666666666666}
2023-10-02T13:40:06Z    INFO    k8up.restic.restic.RepoInit.restic.stderr   Fatal: create repository at s3:https://s3.eu-central-2.amazonaws.com/mybucket failed: client.BucketExists: 400 Bad Request
2023-10-02T13:40:06Z    INFO    k8up.restic.restic.RepoInit.restic.stderr   
2023/10/02 13:40:06 unable to start k8up: failed to initialise the restic repository: cmd.Wait() err: 1

Expected Behavior

The packaged restic binary in the K8up container image can create an S3 bucket in the eu-central-2 AWS region.

Steps To Reproduce

docker run -it --entrypoint /bin/bash ghcr.io/k8up-io/k8up:v2.7.1

# note: new region eu-central-2
export RESTIC_REPOSITORY=s3:https://s3.eu-central-2.amazonaws.com/mybucket
export RESTIC_PASSWORD=foobar
export AWS_ACCESS_KEY_ID=
export AWS_SECRET_ACCESS_KEY=

k8up restic

Version of K8up

v2.7.1

Version of Kubernetes

1.27

Distribution of Kubernetes

EKS

[anothertobi]

@zugao could you maybe have a look at this one (#796 is the fix) and release a new K8up version?

[zugao]

@anothertobi just released v2.7.2. Can you let me know if the problem is fixed?

[anothertobi]

With K8up v2.7.2 the S3 bucket creation and backups in the AWS region eu-central-2 work as expected. Thanks for the new release 🎉