search

vmware-tanzu / velero

Backup and migrate Kubernetes applications and their persistent volumes
https://velero.io
Apache License 2.0
8.57k stars 1.39k forks source link

[velero-plugin-for-aws:v1.9.1] Brakes S3 compatible external storage location - NetApp ONTAP 9 Release 9.13.1P2 #8152

Open leftyb opened 2 weeks ago

leftyb commented 2 weeks ago

Steps followed: Upgrade to : velero-plugin-for-aws:v1.9.1 Velero chart version : version: "6.7.0" Velero version: appVersion: 1.13.2

Previous versions: velero-plugin-for-aws:v1.8.2 Velero chart version : version: "5.2.2" Velero version: appVersion: 1.12.3

What did you expect to happen: Velero Backups properly stored to external S3 compatible storage location. Before upgrade was working properly.

From velero logs: When using checksumAlgorithm: "" Following error logs: operation error S3: PutObject, https response error StatusCode: 501, RequestID: , HostID: , api error NotImplemented: The s3 command you requested is not implemented."

When NOT setting checksumAlgorithm so by default Velero BackupStorageLocation setting to CRC32 Getting following errors from velero logs (tested all available algorithms and got same error) : operation error S3: PutObject, https response error StatusCode: 400, RequestID: , HostID: , api error InvalidArgument: x-amz-content-sha256 must be UNSIGNED-PAYLOAD, STREAMING-AWS4-HMAC-SHA256-PAYLOAD or a valid sha256 value."

S3 Provider: ONTAP 9 According to the ONTAP engineers AWS SDK V2 is supported

https://docs.netapp.com/us-en/ontap/s3-config/ontap-s3-supported-actions-reference.html#bucket-operations

similar to https://github.com/vmware-tanzu/velero/issues/7543 https://github.com/vmware-tanzu/velero/issues/7828

kaovilai commented 2 weeks ago

So we need a way to set the signer to be unsigned.. and other values from https://github.com/aws/aws-sdk-go-v2/blob/84ca95e16adf482b8a80069d5ffa85814c7f61a2/aws/signer/internal/v4/const.go#L12

kaovilai commented 2 weeks ago

Can assign me and I'll double check where if any it can be set and publish some docs.

kaovilai commented 2 weeks ago

AWS SDK V2 is supported

If it is completely supported, all algorithms will be supported, including x-amz-content-sha256 which error mentions isn't supported here by ONTAP.

We will make best effort to allow configuration that will be compatible with others however.

leftyb commented 6 days ago

Hi,

Is there any update on the issue facing with velero-plugin-for-aws > v1.9.1. On pushing backups at ONTAP 9, which is S3 compatible?

Thank you.

kaovilai commented 6 days ago

This isn't currently on my short list of priorities so if you need something in less than 3 months I'd suggest trying prior to v1.9.1 AWS plugin