search

vmware-tanzu / velero

Backup and migrate Kubernetes applications and their persistent volumes
https://velero.io
Apache License 2.0
8.54k stars 1.38k forks source link

[velero-plugin-for-aws:v1.9.1] Brakes S3 compatible external storage location - oracle cloud issue #8013

Closed leftyb closed 1 month ago

leftyb commented 1 month ago

Steps followed: Upgrade to : velero-plugin-for-aws:v1.9.1 Velero chart version : version: "6.7.0" Velero version: appVersion: 1.13.2

Previous versions: velero-plugin-for-aws:v1.8.2 Velero chart version : version: "5.2.2" Velero version: appVersion: 1.12.3

What did you expect to happen: Velero Backups properly stored to external S3 compatible storage location. Before upgrade was working properly.

From velero logs: When using checksumAlgorithm: "" Following error logs: operation error S3: PutObject, https response error StatusCode: 501, RequestID: , HostID: , api error NotImplemented: The s3 command you requested is not implemented."

When NOT setting checksumAlgorithm so by default Velero BackupStorageLocation setting to CRC32 Getting following errors from velero logs (tested all available algorithms and got same error) : operation error S3: PutObject, https response error StatusCode: 400, RequestID: , HostID: , api error InvalidArgument: x-amz-content-sha256 must be UNSIGNED-PAYLOAD, STREAMING-AWS4-HMAC-SHA256-PAYLOAD or a valid sha256 value."

sseago commented 1 month ago

What s3 provider is this? We've seen issues with IBM Cloud which, in some configurations, won't work with the latest version, but I don't think we got the "not implemented" error message there. It could be that whatever s3 driver is being used needs to be upgraded to a newer version with aws sdk v2 support.

DiouKnox commented 1 month ago

i'm using --plugins velero/velero-plugin-for-aws:latest and having the same problem, if anyone knows how to solve it please let me know

kaovilai commented 1 month ago

Please specify your s3 compatible provider that hits this issue. It's been shown that not all s3 compatible has feature parity with AWS S3, which meant additional configuration may need to be done to disable/enable certain flags to make it work again.

DiouKnox commented 1 month ago

oh sorry, oracle cloud. the setup used to work just fine, but i'm not sure when it stopped.

kaovilai commented 1 month ago

You may want to try debug logging like here and send the logs over to the respective provider support channels for implementation.

In the meantime, you could try use velero aws plugins prior to v1.9

reasonerjt commented 1 month ago

@DiouKnox Since v1.9.0 we bumped up to aws-sdk-go-v2 following the suggestion by AWS. If you set checksumAlgorithm: "" and still see the failure, it's probably a compatibility issue on oracle cloud, please check with oracle cloud engineer.

I'll keep this one open but it will likely be closed as a "won't fix" after we hear back from you.

DiouKnox commented 1 month ago

oracle has confirmed that it is a compatibility issue, downgrading velero version has solved the issue. thankyou for the replies.

reasonerjt commented 1 month ago

Thanks, I'm resovling this one.

leftyb commented 1 week ago

Hi all, I apologise for the late response. The S3 backend used is ontap S3 . https://docs.netapp.com/us-en/ontap/s3-config/ontap-s3-supported-actions-reference.html#bucket-operations

Before the upgrade to Velero version: appVersion: 1.13.* Backend was working properly and could store backups. After the upgrade, stopped working. Please provide any info if any. Thank you.