search

vmware-tanzu / velero

Backup and migrate Kubernetes applications and their persistent volumes
https://velero.io
Apache License 2.0
8.45k stars 1.37k forks source link

Backup failed with error XAmzContentSHA256Mismatch #7696

Closed leandreArturia closed 3 months ago

leandreArturia commented 3 months ago

What steps did you take and what happened:

Installed Velero 1.13.0 with CSI snapshot via Helm (vmware-tanzu/velero --version 6.0.0) . Here is my configuration : 

configuration:
    backupStorageLocation:
    - name: default
      provider: aws
      bucket: xxx
      caCert: xxx
      config:
        s3Url: minio_underCA
        pulicUrl: minio_underCA
        region: minio
        s3ForcePathStyle: true
    volumeSnapshotLocation:
    - name: default
      provider: aws
      config:
        region: minio
        s3ForcePathStyle: true
    features: EnableCSI
  snapshotsEnabled: true
  credentials:
    existingSecret: velero-credential
  initContainers:
  - name: velero-plugin-for-aws
    image: velero/velero-plugin-for-aws:v1.9.1
    imagePullPolicy: IfNotPresent
    volumeMounts:
      - mountPath: /target
        name: plugins
  - name: velero-plugin-for-csi
    image: velero/velero-plugin-for-csi:v0.7.0
    imagePullPolicy: IfNotPresent
    volumeMounts:
      - mountPath: /target
        name: plugins

I have a minIO deployed with self-signed certificate.

I was previously on velero 1.6 (with restic) without volumeSnapshot and the backup worked (with the same S3 credentials)

Now, when I try to do a backup, the volumeSnapshot work well but the backup fail:

time="2024-04-17T09:45:55Z" level=info msg="Summary for skipped PVs: []" backup=velero/jenkins-rd-backup-manual logSource="pkg/backup/backup.go:445"
time="2024-04-17T09:45:55Z" level=info msg="Backed up a total of 53 items" backup=velero/jenkins-rd-backup-manual logSource="pkg/backup/backup.go:449" progress=
time="2024-04-17T09:45:55Z" level=info msg="Setting up backup store to persist the backup" backup=velero/jenkins-rd-backup-manual logSource="pkg/controller/backup_controller.go:729"
time="2024-04-17T09:45:55Z" level=error msg="Error uploading log file" backup=jenkins-rd-backup-manual bucket=backup-k8s-dev error="rpc error: code = Unknown desc = error putting object backups/jenkins-rd-backup-manual/jenkins-rd-backup-manual-logs.gz: operation error S3: PutObject, https response error StatusCode: 400, RequestID: 17C708290FBD6CF3, HostID: 978e4e7e-71b7-4e48-84fa-5b1db1faf3d1, api error XAmzContentSHA256Mismatch: The provided 'x-amz-content-sha256' header does not match what was computed." error.file="/go/src/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:253" error.function="main.(*ObjectStore).PutObject" logSource="pkg/persistence/object_store.go:252" prefix=
time="2024-04-17T09:45:55Z" level=info msg="Initial backup processing complete, moving to FinalizingPartiallyFailed" backup=velero/jenkins-rd-backup-manual logSource="pkg/controller/backup_controller.go:743"
time="2024-04-17T09:45:55Z" level=error msg="backup failed" backuprequest=velero/jenkins-rd-backup-manual controller=backup error="rpc error: code = Unknown desc = error putting object backups/jenkins-rd-backup-manual/velero-backup.json: operation error S3: PutObject, https response error StatusCode: 400, RequestID: 17C708290FE6506E, HostID: 978e4e7e-71b7-4e48-84fa-5b1db1faf3d1, api error XAmzContentSHA256Mismatch: The provided 'x-amz-content-sha256' header does not match what was computed." logSource="pkg/controller/backup_controller.go:288"
time="2024-04-17T09:45:55Z" level=info msg="Updating backup's final status" backuprequest=velero/jenkins-rd-backup-manual controller=backup logSource="pkg/controller/backup_controller.go:307"

What did you expect to happen:

To get a backup.

The following information will help us better understand what's going on:

bundle-2024-04-17-12-10-16.tar.gz

Environment:

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

reasonerjt commented 3 months ago

@leandreArturia May I know which version of Minio are you using? I believe the change in aws-plugin v1.9.1 has been verified against Minio cc @danfengliu

Additionally, please check aws-plugin v1.9.2 for the workaround to skip adding checksum header, you will need to make a change to the BSL.

leandreArturia commented 3 months ago

@leandreArturia May I know which version of Minio are you using? I believe the change in aws-plugin v1.9.1 has been verified against Minio cc @danfengliu

Additionally, please check aws-plugin v1.9.2 for the workaround to skip adding checksum header, you will need to make a change to the BSL.

I have a pretty old minio : minio version RELEASE.2022-08-25T07-17-05Z I will check for the 1.9.2 aws plugin.

EDIT: It works with the aws plugin 1.9.2, thank you @reasonerjt

reasonerjt commented 3 months ago

@leandreArturia Thank you for the update, let me keep this open until we double checked in our pipeline.

danfengliu commented 3 months ago

In Velero nightly pipeline, Minio version is 2024-04-18T19-09-19Z without CA, and pipeline passed, no such error occurred.

 containerStatuses:
  - containerID: containerd://edabecc8a9bede9288bfc293af7a408e115989ed1d6437098787df4cc96565a1
    image: docker.io/minio/minio:RELEASE.2024-04-18T19-09-19Z
    imageID: docker.io/minio/minio@sha256:036a068d7d6b69400da6bc07a480bee1e241ef3c341c41d988ed11f520f85124
    lastState: {}
    name: minio
    ready: true
    restartCount: 0
    started: true
    state:
      running:
johnnyhuy commented 3 months ago

Either options work - went with 2

  1. Rolled back to 1.9.0 on the AWS plugin as mentioned the checksum verification wasn't added until the next patch
  2. Updated the Backup Storage Location with checksumAlgorithm: "" to skip checks on Minio 
    ---
apiVersion: velero.io/v1
kind: BackupStorageLocation
metadata:
  name: example
spec:
  config:
    region: minio
    s3ForcePathStyle: "true"
    s3Url: https://example.com
    # Disable checksumAlgorithm to avoid issues with Minio
    # https://github.com/vmware-tanzu/velero-plugin-for-aws/pull/197
    checksumAlgorithm: ""
  default: true
  objectStorage:
    bucket: example
  provider: aws
  credential:
    name: cloud-credentials
    key: cloud

    I'm using a TrueNAS Minio service, on Velero 1.13.2.

Thanks for the fix!

leandreArturia commented 3 months ago

I think we can close this if you don't mind. @johnnyhuy has summarized the 2 solutions.

jaredkipe commented 2 months ago

Same problem but with Linode Object storage, solution to empty the checksumAlgorithm works.

mzimry commented 1 month ago

@jaredkipe having the same issue with linode, but with aws v1.9.0, so no checksumAlgorithm field and still getting errors. can you please share your BackupStorageLocation configs?

Minipada commented 1 day ago

Also having the same issue with Linode and disabled checksumAlgorithm