search

vmware-tanzu / velero

Backup and migrate Kubernetes applications and their persistent volumes
https://velero.io
Apache License 2.0
8.4k stars 1.37k forks source link

Backup finalizer hangs with "gzip: invalid header" error #7839

Closed ameerdtm closed 2 weeks ago

ameerdtm commented 1 month ago

What steps did you take and what happened: Clean install of velero; wiped the s3 bucket. Run a backup, hangs in the Finalizing state and logs show "gzip: invalid header" errror. The S3 storage is Pure Flashblade and not AWS S3. This same configuration (velero and aws plug-in) is working with velero 1.11.

What did you expect to happen: The backup to succeed without error.

The following information will help us better understand what's going on: Attaching debug from backup.

Anything else you would like to add: Have tried removing, clearing S3 bucket, and running again with same error. AWS plugin is the recommended verson for v1.13.2

Environment:

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

blackpiglet commented 1 month ago

Please check whether your AWS plugin version is v1.9.2. https://github.com/vmware-tanzu/velero-plugin-for-aws/releases/tag/v1.9.2 In v1.9.2, the AWS plugin can support passing an empty string to checksumAlgorithm. In that way, the checksum is skipped for Velero's interaction with the OSS.

ameerdtm commented 1 month ago

Please check whether your AWS plugin version is v1.9.2. https://github.com/vmware-tanzu/velero-plugin-for-aws/releases/tag/v1.9.2 In v1.9.2, the AWS plugin can support passing an empty string to checksumAlgorithm. In that way, the checksum is skipped for Velero's interaction with the OSS.

Yes, using the v1.9.2 version of the plug in: velero/velero-plugin-for-aws:v1.9.2 50b9c6222000 Id: 353046a0175f74216db29be82fd1ea21da02d1b87bf4f8b1fadd32fdee30e30c Digest: sha256:69b54e9210bd8a7e58197b23099ac49cf841780666e48ce24f04e6ef27a1a4b5

blackpiglet commented 1 month ago

Could you help to collect the debug bundle by running command velero debug?

ameerdtm commented 1 month ago

Yes, I attached the debug file before, and also tried again just now. The UI says its uploaded but I can't see it, not sure if you can? Is there another way to post the file? Thank you.

blackpiglet commented 1 month ago

No, I couldn't find the bundle. Please consider using gist.

ameerdtm commented 1 month ago

Thank you for the recommandation. Please find the debug file in the gist. https://gist.github.com/ameerdtm/59d7ca23bc404eb4d587eefbb0535825

ameerdtm commented 4 weeks ago

Are you able to download the bundle?

ameerdtm commented 4 weeks ago

I am going to be OOO for the next week and a half; please leave this issue open if you need more information from me. Thank you.

blackpiglet commented 3 weeks ago

Yeah, I can download the bundle, but I didn't find more information than the "gzip: invalid header" in the bundle. I think it's still related to the AWS SDK version bump in the Velero AWS plugin.

ameerdtm commented 2 weeks ago

It appears that I was hitting this issue, with a Pure Storage S3 endpoint, and the fix is to explictly not set a header checksum. Thanks.

https://github.com/vmware-tanzu/velero/issues/7534

https://github.com/vmware-tanzu/velero-plugin-for-aws/pull/197/files#diff-9b87db42a9f0c061fbece1dcef102333e7bef47638f1bef069a3c82e204f8ee3