search

vmware-tanzu / velero

Backup and migrate Kubernetes applications and their persistent volumes
https://velero.io
Apache License 2.0
8.66k stars 1.4k forks source link

Velero AWS plugin: S3 compatibility broken with Qumulo in recent versions #8312

Open Mise opened 3 days ago

Mise commented 3 days ago

Hello,

We are attempting to integrate Velero and Velero plugin for AWS with Qumulo storage, which the vendor claims is fully compatible with AWS S3.

However, it seems that the requests made by the Velero AWS plugin to our storage include parameters that it does not recognize. For example, some requests fail due to the inclusion of the x-id parameter, for example:

GET https://storage.domain.tld:443/bucketname/backups/test-backup-qumulo/test-backup-qumulo-volumeinfo.json.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=blablaaws4_request&X-Amz-Date=20241015T093024Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&x-id=GetObject&X-Amz-Signature=blabla"

Other requests, such as PutObject requests, fail similarly. The storage vendor asserts that x-id parameter is not part of the S3 specification, and I could not find any information about it in the S3 API reference.

We tried Velero versions 1.13.2 and 1.14.1 and AWS plugin versions 1.9.0 and 1.10, which exhibit the same behavior, and a combination of older Velero v1.12.4 with AWS plugin v1.8.2, which work correctly.

Could you please advise why newer Velero and plugin versions are including the x-id parameter in their S3 requests, and whether there is a way to configure Velero to omit this parameter when communicating with our S3 storage?

Thank you!

kaovilai commented 3 days ago

dupe of https://github.com/vmware-tanzu/velero/issues/8265

kaovilai commented 3 days ago

bump to aws-sdk-go-v2 forced addition of some headers and drop usage of older "legacy" (as aws determined) apis.

Everyone who claims S3 compatible are only via testing/replicating functionality, which can get out of date and eventually incompatible with what aws uses overtime.

kaovilai commented 3 days ago

"x-id" is an expected part of requests made by aws-sdk-go-v2 https://github.com/aws/aws-sdk-go-v2/blob/04e7aca073a0a7ed479aa37cad88a1cf58a979a1/service/s3/internal/customizations/presign_test.go#L35-L41

blackpiglet commented 2 days ago

/assign @kaovilai

kzakhark commented 2 days ago

"x-id" is an expected part of requests made by aws-sdk-go-v2 https://github.com/aws/aws-sdk-go-v2/blob/04e7aca073a0a7ed479aa37cad88a1cf58a979a1/service/s3/internal/customizations/presign_test.go#L35-L41

Is there any indication of why it is an expected part, as there's no mention of this parameter in the API reference?

kaovilai commented 2 days ago

Ask aws sdk go v2 team. We don't know.

kaovilai commented 2 days ago

aws-sdk-go v1 is expected to go into unsupported state in mid 2025. You should ask Qumulo if they would eventually implement support for aws-sdk-go-v2 or they have an alternative SDK they prefer us to use.

kaovilai commented 2 days ago

@blackpiglet this should be labeled Area/Storage/s3-compatible not aws :)

kaovilai commented 1 day ago

I'm assuming it's for debugging with customers where aws only have to ask for request URL, and not method.

More people understand URL than HTTP request methods (PUT/GET)