After update velero-plugin-for-aws - backup status finalizing never completed

We updated: velero.image.tag: v1.13.0 -> v1.13.1 velero-plugin-for-aws: v1.9.0 -> v1.9.1 I updated helm chart: 6.3.0 -> 6.4.0

What did you expect to happen: Complete a shedule backup at night or complete a manual triggered backup, but the backups are stuck in status "finalizing"!

apiVersion: velero.io/v1 kind: Backup metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: > {"apiVersion":"velero.io/v1","kind":"Schedule","metadata":{"annotations":{},"labels":{"app.kubernetes.io/instance":"velero","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"velero","argocd.argoproj.io/instance":"velero","helm.sh/chart":"velero-6.4.0"},"name":"velero-scheduled-backup","namespace":"velero"},"spec":{"schedule":"0 1 *","template":{"includedNamespaces":["emp","fluentd","kube-prometheus-stack","loki-stack","mail","openldap-alpine","vault"],"ttl":"168h"},"useOwnerReferencesInBackup":true}} velero.io/resource-timeout: 10m0s velero.io/source-cluster-k8s-gitversion: v1.25.3-r0-25.2.19 velero.io/source-cluster-k8s-major-version: '1' velero.io/source-cluster-k8s-minor-version: 25+ creationTimestamp: '2024-05-21T01:00:50Z' generation: 24 labels: app.kubernetes.io/instance: velero app.kubernetes.io/managed-by: Helm app.kubernetes.io/name: velero argocd.argoproj.io/instance: velero helm.sh/chart: velero-6.4.0 velero.io/schedule-name: velero-scheduled-backup velero.io/storage-location: default managedFields:

apiVersion: velero.io/v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:annotations: .: {} f:kubectl.kubernetes.io/last-applied-configuration: {} f:velero.io/resource-timeout: {} f:velero.io/source-cluster-k8s-gitversion: {} f:velero.io/source-cluster-k8s-major-version: {} f:velero.io/source-cluster-k8s-minor-version: {} f:labels: .: {} f:app.kubernetes.io/instance: {} f:app.kubernetes.io/managed-by: {} f:app.kubernetes.io/name: {} f:argocd.argoproj.io/instance: {} f:helm.sh/chart: {} f:velero.io/schedule-name: {} f:velero.io/storage-location: {} f:ownerReferences: .: {} k:{"uid":"d55a28b9-da0d-4b0c-b83e-4d73425a923a"}: {} f:spec: .: {} f:csiSnapshotTimeout: {} f:defaultVolumesToFsBackup: {} f:hooks: {} f:includedNamespaces: {} f:itemOperationTimeout: {} f:metadata: {} f:snapshotMoveData: {} f:storageLocation: {} f:ttl: {} f:status: .: {} f:expiration: {} f:formatVersion: {} f:hookStatus: {} f:phase: {} f:progress: .: {} f:itemsBackedUp: {} f:totalItems: {} f:startTimestamp: {} f:version: {} manager: velero-server operation: Update time: '2024-05-21T01:04:27Z' name: velero-scheduled-backup-20240521010050 namespace: velero ownerReferences:

apiVersion: velero.io/v1 controller: true kind: Schedule name: velero-scheduled-backup uid: d55a28b9-da0d-4b0c-b83e-4d73425a923a resourceVersion: '508126532' uid: 7e099c1d-bff4-4828-af2c-e250a404ac70 selfLink: >- /apis/velero.io/v1/namespaces/velero/backups/velero-scheduled-backup-20240521010050 status: expiration: '2024-05-28T01:00:50Z' formatVersion: 1.1.0 hookStatus: {} phase: Finalizing progress: itemsBackedUp: 758 totalItems: 758 startTimestamp: '2024-05-21T01:00:50Z' version: 1 spec: csiSnapshotTimeout: 10m0s defaultVolumesToFsBackup: true hooks: {} includedNamespaces:

emp

fluentd

kube-prometheus-stack

loki-stack

mail

openldap-alpine

vault itemOperationTimeout: 4h0m0s metadata: {} snapshotMoveData: false storageLocation: default ttl: 168h0m0s

log of the pod:

time="2024-05-21T09:54:51Z" level=error msg="Error getting backup metadata from backup store" backup=velero-scheduled-backup-20240316010018 backupLocation=velero/default controller=backup-sync error="yaml: line 3: mapping values are not allowed in this context" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/persistence/object_store.go:315" error.function="github.com/vmware-tanzu/velero/pkg/persistence.(objectBackupStore).GetBackupMetadata" logSource="pkg/controller/backup_sync_controller.go:158" time="2024-05-21T09:54:51Z" level=info msg="Attempting to sync backup into cluster" backup=velero-scheduled-backup-20240322010024 backupLocation=velero/default controller=backup-sync logSource="pkg/controller/backup_sync_controller.go:144" time="2024-05-21T09:54:51Z" level=error msg="Error getting backup metadata from backup store" backup=velero-scheduled-backup-20240322010024 backupLocation=velero/default controller=backup-sync error="yaml: line 3: mapping values are not allowed in this context" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/persistence/object_store.go:315" error.function="github.com/vmware-tanzu/velero/pkg/persistence.(objectBackupStore).GetBackupMetadata" logSource="pkg/controller/backup_sync_controller.go:158" time="2024-05-21T09:54:51Z" level=info msg="Attempting to sync backup into cluster" backup=velero-scheduled-backup-20240319010021 backupLocation=velero/default controller=backup-sync logSource="pkg/controller/backup_sync_controller.go:144" time="2024-05-21T09:54:51Z" level=error msg="Error getting backup metadata from backup store" backup=velero-scheduled-backup-20240319010021 backupLocation=velero/default controller=backup-sync error="yaml: line 3: mapping values are not allowed in this context" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/persistence/object_store.go:315" error.function="github.com/vmware-tanzu/velero/pkg/persistence.(objectBackupStore).GetBackupMetadata" logSource="pkg/controller/backup_sync_controller.go:158" time="2024-05-21T09:55:09Z" level=info msg="Validating BackupStorageLocation" backup-storage-location=velero/default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:141" time="2024-05-21T09:55:09Z" level=info msg="BackupStorageLocations is valid, marking as available" backup-storage-location=velero/default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:126" time="2024-05-21T09:55:35Z" level=error msg="Error getting backup item operations" backup=velero/velero-scheduled-backup-20240519010048 controller=backup-finalizer error="gzip: invalid header" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/persistence/object_store.go:412" error.function=github.com/vmware-tanzu/velero/pkg/persistence.decode logSource="pkg/controller/backup_finalizer_controller.go:143" time="2024-05-21T09:55:35Z" level=error msg="Reconciler error" Backup="{\"name\":\"velero-scheduled-backup-20240519010048\",\"namespace\":\"velero\"}" controller=backup controllerGroup=velero.io controllerKind=Backup error="gzip: invalid header" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/persistence/object_store.go:412" error.function=github.com/vmware-tanzu/velero/pkg/persistence.decode logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:123" name=velero-scheduled-backup-20240519010048 namespace=velero reconcileID="\"341dc73f-337f-45a9-bed9-7965dd1b1ead\"" time="2024-05-21T09:55:51Z" level=info msg="Found 7 backups in the backup location that do not exist in the cluster and need to be synced" backupLocation=velero/default controller=backup-sync logSource="pkg/controller/backup_sync_controller.go:136" time="2024-05-21T09:55:51Z" level=info msg="Attempting to sync backup into cluster" backup=velero-scheduled-backup-20240319010021 backupLocation=velero/default controller=backup-sync logSource="pkg/controller/backup_sync_controller.go:144" time="2024-05-21T09:55:51Z" level=error msg="Error getting backup metadata from backup store" backup=velero-scheduled-backup-20240319010021 backupLocation=velero/default controller=backup-sync error="yaml: line 3: mapping values are not allowed in this context" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/persistence/object_store.go:315" error.function="github.com/vmware-tanzu/velero/pkg/persistence.(objectBackupStore).GetBackupMetadata" logSource="pkg/controller/backup_sync_controller.go:158" time="2024-05-21T09:55:51Z" level=info msg="Attempting to sync backup into cluster" backup=velero-scheduled-backup-20240320010022 backupLocation=velero/default controller=backup-sync logSource="pkg/controller/backup_sync_controller.go:144" time="2024-05-21T09:55:51Z" level=error msg="Error getting backup metadata from backup store" backup=velero-scheduled-backup-20240320010022 backupLocation=velero/default controller=backup-sync error="yaml: line 3: mapping values are not allowed in this context" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/persistence/object_store.go:315" error.function="github.com/vmware-tanzu/velero/pkg/persistence.(objectBackupStore).GetBackupMetadata" logSource="pkg/controller/backup_sync_controller.go:158" time="2024-05-21T09:55:51Z" level=info msg="Attempting to sync backup into cluster" backup=velero-scheduled-backup-20240316010018 backupLocation=velero/default controller=backup-sync logSource="pkg/controller/backup_sync_controller.go:144" time="2024-05-21T09:55:51Z" level=error msg="Error getting backup metadata from backup store" backup=velero-scheduled-backup-20240316010018 backupLocation=velero/default controller=backup-sync error="yaml: line 3: mapping values are not allowed in this context" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/persistence/object_store.go:315" error.function="github.com/vmware-tanzu/velero/pkg/persistence.(objectBackupStore).GetBackupMetadata" logSource="pkg/controller/backup_sync_controller.go:158" time="2024-05-21T09:55:51Z" level=info msg="Attempting to sync backup into cluster" backup=velero-scheduled-backup-20240321010023 backupLocation=velero/default controller=backup-sync logSource="pkg/controller/backup_sync_controller.go:144" time="2024-05-21T09:55:51Z" level=error msg="Error getting backup metadata from backup store" backup=velero-scheduled-backup-20240321010023 backupLocation=velero/default controller=backup-sync error="yaml: line 3: mapping values are not allowed in this context" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/persistence/object_store.go:315" error.function="github.com/vmware-tanzu/velero/pkg/persistence.(objectBackupStore).GetBackupMetadata" logSource="pkg/controller/backup_sync_controller.go:158" time="2024-05-21T09:55:51Z" level=info msg="Attempting to sync backup into cluster" backup=velero-scheduled-backup-20240317010019 backupLocation=velero/default controller=backup-sync logSource="pkg/controller/backup_sync_controller.go:144" time="2024-05-21T09:55:51Z" level=error msg="Error getting backup metadata from backup store" backup=velero-scheduled-backup-20240317010019 backupLocation=velero/default controller=backup-sync error="yaml: line 3: mapping values are not allowed in this context" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/persistence/object_store.go:315" error.function="github.com/vmware-tanzu/velero/pkg/persistence.(objectBackupStore).GetBackupMetadata" logSource="pkg/controller/backup_sync_controller.go:158" time="2024-05-21T09:55:51Z" level=info msg="Attempting to sync backup into cluster" backup=velero-scheduled-backup-20240322010024 backupLocation=velero/default controller=backup-sync logSource="pkg/controller/backup_sync_controller.go:144" time="2024-05-21T09:55:51Z" level=error msg="Error getting backup metadata from backup store" backup=velero-scheduled-backup-20240322010024 backupLocation=velero/default controller=backup-sync error="yaml: line 3: mapping values are not allowed in this context" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/persistence/object_store.go:315" error.function="github.com/vmware-tanzu/velero/pkg/persistence.(objectBackupStore).GetBackupMetadata" logSource="pkg/controller/backup_sync_controller.go:158" time="2024-05-21T09:55:51Z" level=info msg="Attempting to sync backup into cluster" backup=velero-scheduled-backup-20240318010020 backupLocation=velero/default controller=backup-sync logSource="pkg/controller/backup_sync_controller.go:144" time="2024-05-21T09:55:51Z" level=error msg="Error getting backup metadata from backup store" backup=velero-scheduled-backup-20240318010020 backupLocation=velero/default controller=backup-sync error="yaml: line 3: mapping values are not allowed in this context" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/persistence/object_store.go:315" error.function="github.com/vmware-tanzu/velero/pkg/persistence.(objectBackupStore).GetBackupMetadata" logSource="pkg/controller/backup_sync_controller.go:158" time="2024-05-21T09:56:09Z" level=info msg="Validating BackupStorageLocation" backup-storage-location=velero/default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:141" time="2024-05-21T09:56:09Z" level=info msg="BackupStorageLocations is valid, marking as available" backup-storage-location=velero/default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:126" I0521 09:56:16.589529 1 request.go:697] Waited for 1.04589123s due to client-side throttling, not priority and fairness, request: GET:https://10.247.0.1:443/apis/config.k8s.io/v1beta1?timeout=32s time="2024-05-21T09:56:22Z" level=error msg="Error getting backup item operations" backup=velero/velero-scheduled-backup-20240520010049 controller=backup-finalizer error="gzip: invalid header" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/persistence/object_store.go:412" error.function=github.com/vmware-tanzu/velero/pkg/persistence.decode logSource="pkg/controller/backup_finalizer_controller.go:143" time="2024-05-21T09:56:22Z" level=error msg="Reconciler error" Backup="{\"name\":\"velero-scheduled-backup-20240520010049\",\"namespace\":\"velero\"}" controller=backup controllerGroup=velero.io controllerKind=Backup error="gzip: invalid header" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/persistence/object_store.go:412" error.function=github.com/vmware-tanzu/velero/pkg/persistence.decode logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:123" name=velero-scheduled-backup-20240520010049 namespace=velero reconcileID="\"7d04fff7-6b76-476c-af0d-46cbf09d2c5f\""

Environment: k8s

vmware-tanzu / helm-charts

After update velero-plugin-for-aws - backup status finalizing never completed #580