Error uploading log file SignatureDoesNotMatch

[dploeger]

What steps did you take and what happened: Create backups from schedule. The backup is marked as "failed". Using velero backup logs on the failed backup yields in "File not found" - apparently, Velero started the backup and Kopia actually produced snapshots and stored them on S3, but Velero wasn't able to store its backup and the logs to s3.

What did you expect to happen: The backups run successfully

The following information will help us better understand what's going on: bundle-2024-04-16-11-44-25.tar.gz

Anything else you would like to add: We've deployed Velero on our local Tanzu cluster with the AWS plugin to store the backup on our local Cloudian HyperStore. Until friday everything went fine.

A problem I can think of is, that we used the :latest tag with our aws-plugin init container and on Friday the most current image version was pulled and deployed.

However, we now fixed the version to 1.9.1 (which is the recommended version for 1.13.1) but the problem still persists.

Environment:

• Velero version (use velero version): Client: Version: v1.13.1 Git commit: ea5a89f83b89b2cb7a27f54148683c1ee8d57a37 Server: Version: v1.13.1
• Velero features (use velero client config get features):
• Kubernetes version (use kubectl version): Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.5+vmware.wcp.1", GitCommit:"c088f942056efb7cd7b4c8709078812b6343d5fa", GitTreeState:"clean", BuildDate:"2023-09-18T04:30:05Z", GoVersion:"go1.20.7 X:boringcrypto", Compiler:"gc", Platform:"linux/amd64"} Kustomize Version: v5.0.1 Server Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.5+vmware.2-fips.1", GitCommit:"23a5ab39c3188caaf651128b0dfed523eecd8023", GitTreeState:"clean", BuildDate:"2023-07-03T15:25:46Z", GoVersion:"go1.19.9 X:boringcrypto", Compiler:"gc", Platform:"linux/amd64"}
• Kubernetes installer & version: n/a
• Cloud provider or hardware configuration: Tanzu
• OS (e.g. from /etc/os-release): n/a

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

• :+1: for "I would like to see this bug fixed as soon as possible"
• :-1: for "There are more important bugs to focus on right now"

[reasonerjt]

@dploeger Have you checked with older version of aws-plugins like v1.8.1. We have seen issue on S3-compatible storage after adding checksum to the request. Similar to https://github.com/vmware-tanzu/velero/issues/7534 Some vendor reports 403 but it's misleading.

Could you please also check the workaround provided in aws-plugin v1.9.2 where you can skip adding the checksum to request header?

[dploeger]

@reasonerjt Thank you! Yes, both work: Downgrading to 1.8.1 and disabling the checksum in 1.9.2.

If this is the way to go, disabling the checksum should be documented somewhere.