Velero 1.14.1 & aws plugin 1.10.1: api error SignatureDoesNotMatch

[amrap030]

What steps did you take and what happened:

When using velero version 1.14.1 and the AWS plugin version 1.10.1 I always get the following error when creating a backup:

time="2024-10-04T17:37:52+02:00" level=info msg="Backed up a total of 972 items" backup=velero/velero-daily-20241004173619 logSource="pkg/backup/backup.go:499" progress=
time="2024-10-04T17:37:52+02:00" level=info msg="Setting up backup store to persist the backup" backup=velero/velero-daily-20241004173619 logSource="pkg/controller/backup_controller.go:714"
time="2024-10-04T17:37:53+02:00" level=error msg="Error uploading log file" backup=velero-daily-20241004173619 bucket=<bucketname> error="rpc error: code = Unknown desc = error putting object velero/backups/velero-daily-20241004173619/velero-daily-20241004173619-logs.gz: operation error S3: PutObject, https response error StatusCode: 403, RequestID: e72ef188-ff29-1fff-a304-00620be16750, HostID: 7aaa0df89c574f3386ff8f3db4a2a4b1, api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. For more information, see REST Authentication and SOAP Authentication for details." error.file="/go/src/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:279" error.function="main.(*ObjectStore).PutObject" logSource="pkg/persistence/object_store.go:256" prefix=velero
time="2024-10-04T17:37:53+02:00" level=info msg="Initial backup processing complete, moving to FinalizingPartiallyFailed" backup=velero/velero-daily-20241004173619 logSource="pkg/controller/backup_controller.go:728"
time="2024-10-04T17:37:53+02:00" level=info msg="plugin process exited" backup=velero/velero-daily-20241004173619 cmd=/velero id=106 logSource="pkg/plugin/clientmgmt/process/logrus_adapter.go:80" plugin=/velero
time="2024-10-04T17:37:53+02:00" level=error msg="error encountered while scanning stdout" backup=velero/velero-daily-20241004173619 cmd=/plugins/velero-plugin-for-aws error="read |0: file already closed" logSource="pkg/plugin/clientmgmt/process/logrus_adapter.go:90"
time="2024-10-04T17:37:53+02:00" level=info msg="plugin process exited" backup=velero/velero-daily-20241004173619 cmd=/plugins/velero-plugin-for-aws id=114 logSource="pkg/plugin/clientmgmt/process/logrus_adapter.go:80" plugin=/plugins/velero-plugin-for-aws
time="2024-10-04T17:37:53+02:00" level=error msg="backup failed" backuprequest=velero/velero-daily-20241004173619 controller=backup error="rpc error: code = Unknown desc = error putting object velero/backups/velero-daily-20241004173619/velero-backup.json: operation error S3: PutObject, https response error StatusCode: 403, RequestID: e72ef18a-ff29-1fff-a304-00620be16750, HostID: 7aaa0df89c574f3386ff8f3db4a2a4b1, api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. For more information, see REST Authentication and SOAP Authentication for details." logSource="pkg/controller/backup_controller.go:287"

When I look into my S3 bucket with a viewer, I can see that there is a folder /restic created with all the expected files. However, the folder /backups is missing.

What did you expect to happen:

I expect the backup to complete without failures with a folder called /backups and its content.

The following information will help us better understand what's going on:

If you are using velero v1.7.0+:
Please use velero debug --backup <backupname> --restore <restorename> to generate the support bundle, and attach to this issue, more options please refer to velero debug --help

If you are using earlier versions:
Please provide the output of the following commands (Pasting long output into a GitHub gist or other pastebin is fine.)

• kubectl logs deployment/velero -n velero
• velero backup describe <backupname> or kubectl get backup/<backupname> -n velero -o yaml
• velero backup logs <backupname>
• velero restore describe <restorename> or kubectl get restore/<restorename> -n velero -o yaml
• velero restore logs <restorename>

Anything else you would like to add:

Environment:

• Velero version (use velero version): 1.14.1
• Velero features (use velero client config get features): n/a
• Kubernetes version (use kubectl version): 1.31
• Kubernetes installer & version: Juju Charms
• Cloud provider or hardware configuration: Openstack
• OS (e.g. from /etc/os-release): n/a

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

• :+1: for "I would like to see this bug fixed as soon as possible"
• :-1: for "There are more important bugs to focus on right now"

[shubham-pampattiwar]

Are you using the aws plugin for s3 compatible object storage or AWS S3 object storage ?

[kaovilai]

If this is GCP, it is a known issue. Please help me fill this README.md https://github.com/vmware-tanzu/velero-plugin-for-aws/pull/219/files

[amrap030]

@kaovilai No I am not using GCP, I am using Openstack as my cloud provider.

@shubham-pampattiwar it is S3 compatible storage. I am currently using velero v1.14.1 and velero/velero-plugin-for-aws:v1.9.0 and with this combination it is working. However, I would like to update to the newest version but as described, this breaks it.

[amrap030]

@kaovilai @shubham-pampattiwar FYI, I am using Openstack as my cloud provider for the Cluster, but the S3 compatible storage is based on Cloudian.