vmware-tanzu / velero

Backup and migrate Kubernetes applications and their persistent volumes
https://velero.io
Apache License 2.0
8.76k stars 1.41k forks source link

Velero backup failed on azure #4895

Closed chrislinan closed 2 years ago

chrislinan commented 2 years ago

What steps did you take and what happened: I create a backup from schedule, and velero log shows everythings gose well but the back up failed in the end. log: time="2022-05-08T14:10:54Z" level=error msg="backup failed" controller=backup error="rpc error: code = Unavailable desc = error reading from server: EOF" key=default/velero-backup-obj-3f6fffe7-867b-4609-bade-00e5ab32977d logSource="pkg/controller/backup_controller.go:282"

What did you expect to happen: backup succeeded

The following information will help us better understand what's going on:

If you are using velero v1.7.0+:
Please use velero debug --backup <backupname> --restore <restorename> to generate the support bundle, and attach to this issue, more options please refer to velero debug --help An error occurred: velero deployment does not exist in namespace: default

If you are using earlier versions:
Please provide the output of the following commands (Pasting long output into a GitHub gist or other pastebin is fine.)

Phase: Failed (run velero backup logs velero-backup-obj-a9badd03-192e-4123-8b92-ea5476d7bf84 for more information)

Errors: 0 Warnings: 0

Namespaces: Included: * Excluded: kube-system, kube-public, kube-node-lease, velert-restore-test

Resources: Included: * Excluded: pv, pvc Cluster-scoped: auto

Label selector:

Storage Location: default

Velero-Native Snapshot PVs: auto

TTL: 48h0m0s

Hooks:

Backup Format Version: 1.1.0

Started: 2022-05-08 14:54:13 +0000 UTC Completed: 2022-05-08 14:57:17 +0000 UTC

Expiration: 2022-05-10 14:54:13 +0000 UTC

Total items to be backed up: 1910 Items backed up: 1910

Velero-Native Snapshots:

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]

Environment:

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

reasonerjt commented 2 years ago

@chrislinan Have you tried manually run the backup (not via schedule)?

Based on the information you provided it's hard to tell if it's due to error in calling Azure's API We need more logs to debug. As for the error by velero debug, it should by default looking for velero deployment in ns velero not default. You may also customize the namespace by using --namespace in the command, check velero debug --help

Additionally, are you using a released? The commit ID should NOT be dev.

chrislinan commented 2 years ago

Hi @reasonerjt I create a backup manually by this command: velero backup create backup-test1 here is the full log of this backup, it looks like the error occurred when calling Azure api:

time="2022-05-09T06:30:44Z" level=info msg="Setting up backup log" backup=default/backup-test1 controller=backup logSource="pkg/controller/backup_controller.go:535"
time="2022-05-09T06:30:44Z" level=info msg="Setting up backup temp file" backup=default/backup-test1 logSource="pkg/controller/backup_controller.go:557"
time="2022-05-09T06:30:44Z" level=info msg="Setting up plugin manager" backup=default/backup-test1 logSource="pkg/controller/backup_controller.go:564"
time="2022-05-09T06:30:44Z" level=info msg="Getting backup item actions" backup=default/backup-test1 logSource="pkg/controller/backup_controller.go:568"
time="2022-05-09T06:30:46Z" level=info msg="Setting up backup store to check for backup existence" backup=default/backup-test1 logSource="pkg/controller/backup_controller.go:578"
time="2022-05-09T06:30:50Z" level=info msg="Writing backup version file" backup=default/backup-test1 logSource="pkg/backup/backup.go:191"
time="2022-05-09T06:30:50Z" level=info msg="Including namespaces: *" backup=default/backup-test1 logSource="pkg/backup/backup.go:197"
time="2022-05-09T06:30:50Z" level=info msg="Excluding namespaces: <none>" backup=default/backup-test1 logSource="pkg/backup/backup.go:198"
time="2022-05-09T06:30:50Z" level=info msg="Including resources: *" backup=default/backup-test1 logSource="pkg/backup/backup.go:201"
time="2022-05-09T06:30:50Z" level=info msg="Excluding resources: <none>" backup=default/backup-test1 logSource="pkg/backup/backup.go:202"
time="2022-05-09T06:30:50Z" level=info msg="Backing up all pod volumes using restic: false" backup=default/backup-test1 logSource="pkg/backup/backup.go:203"
time="2022-05-09T06:30:57Z" level=info msg="Getting items for group" backup=default/backup-test1 group=v1 logSource="pkg/backup/item_collector.go:81"
time="2022-05-09T06:30:57Z" level=info msg="Getting items for resource" backup=default/backup-test1 group=v1 logSource="pkg/backup/item_collector.go:170" resource=pods
time="2022-05-09T06:30:57Z" level=info msg="Listing items" backup=default/backup-test1 group=v1 logSource="pkg/backup/item_collector.go:299" namespace= resource=pods
time="2022-05-09T06:31:01Z" level=info msg="list for groupResource pods was not paginated" backup=default/backup-test1 group=v1 logSource="pkg/backup/item_collector.go:318" namespace= resource=pods
time="2022-05-09T06:31:01Z" level=info msg="Retrieved 135 items" backup=default/backup-test1 group=v1 logSource="pkg/backup/item_collector.go:343" namespace= resource=pods
time="2022-05-09T06:31:02Z" level=info msg="Getting items for resource" backup=default/backup-test1 group=v1 logSource="pkg/backup/item_collector.go:170" resource=persistentvolumeclaims
time="2022-05-09T06:31:02Z" level=info msg="Listing items" backup=default/backup-test1 group=v1 logSource="pkg/backup/item_collector.go:299" namespace= resource=persistentvolumeclaims
time="2022-05-09T06:31:02Z" level=info msg="list for groupResource persistentvolumeclaims was not paginated" backup=default/backup-test1 group=v1 logSource="pkg/backup/item_collector.go:318" namespace= resource=persistentvolumeclaims
time="2022-05-09T06:31:02Z" level=info msg="Retrieved 8 items" backup=default/backup-test1 group=v1 logSource="pkg/backup/item_collector.go:343" namespace= resource=persistentvolumeclaims
time="2022-05-09T06:31:02Z" level=info msg="Getting items for resource" backup=default/backup-test1 group=v1 logSource="pkg/backup/item_collector.go:170" resource=persistentvolumes
time="2022-05-09T06:31:02Z" level=info msg="Listing items" backup=default/backup-test1 group=v1 logSource="pkg/backup/item_collector.go:299" namespace= resource=persistentvolumes
time="2022-05-09T06:31:02Z" level=info msg="list for groupResource persistentvolumes was not paginated" backup=default/backup-test1 group=v1 logSource="pkg/backup/item_collector.go:318" namespace= resource=persistentvolumes
time="2022-05-09T06:31:02Z" level=info msg="Retrieved 8 items" backup=default/backup-test1 group=v1 logSource="pkg/backup/item_collector.go:343" namespace= resource=persistentvolumes
time="2022-05-09T06:31:02Z" level=info msg="Getting items for resource" backup=default/backup-test1 group=v1 logSource="pkg/backup/item_collector.go:170" resource=endpoints
time="2022-05-09T06:31:02Z" level=info msg="Listing items" backup=default/backup-test1 group=v1 logSource="pkg/backup/item_collector.go:299" namespace= resource=endpoints
time="2022-05-09T06:31:03Z" level=info msg="list for groupResource endpoints was not paginated" backup=default/backup-test1 group=v1 logSource="pkg/backup/item_collector.go:318" namespace= resource=endpoints
time="2022-05-09T06:31:03Z" level=info msg="Retrieved 65 items" backup=default/backup-test1 group=v1 logSource="pkg/backup/item_collector.go:343" namespace= resource=endpoints
time="2022-05-09T06:31:03Z" level=info msg="Getting items for resource" backup=default/backup-test1 group=v1 logSource="pkg/backup/item_collector.go:170" resource=events
time="2022-05-09T06:31:03Z" level=info msg="Listing items" backup=default/backup-test1 group=v1 logSource="pkg/backup/item_collector.go:299" namespace= resource=events
time="2022-05-09T06:31:04Z" level=info msg="list for groupResource events was not paginated" backup=default/backup-test1 group=v1 logSource="pkg/backup/item_collector.go:318" namespace= resource=events
**** too long to put the full log here *****
time="2022-05-09T06:35:05Z" level=info msg="Backed up 2756 items out of an estimated total of 2788 (estimate will change throughout the backup)" backup=default/backup-test1 logSource="pkg/backup/backup.go:379" name=hana-scale-out-mode namespace=default progress= resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Processing item" backup=default/backup-test1 logSource="pkg/backup/backup.go:339" name=hanaservice-topology-reconciler namespace=default progress= resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Backing up item" backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:121" name=hanaservice-topology-reconciler namespace=default resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Backed up 2757 items out of an estimated total of 2788 (estimate will change throughout the backup)" backup=default/backup-test1 logSource="pkg/backup/backup.go:379" name=hanaservice-topology-reconciler namespace=default progress= resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Processing item" backup=default/backup-test1 logSource="pkg/backup/backup.go:339" name=hanaservicetopology namespace=default progress= resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Backing up item" backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:121" name=hanaservicetopology namespace=default resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Backed up 2758 items out of an estimated total of 2788 (estimate will change throughout the backup)" backup=default/backup-test1 logSource="pkg/backup/backup.go:379" name=hanaservicetopology namespace=default progress= resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Processing item" backup=default/backup-test1 logSource="pkg/backup/backup.go:339" name=hc-fg-conditionaltoggles namespace=default progress= resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Backing up item" backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:121" name=hc-fg-conditionaltoggles namespace=default resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Backed up 2759 items out of an estimated total of 2788 (estimate will change throughout the backup)" backup=default/backup-test1 logSource="pkg/backup/backup.go:379" name=hc-fg-conditionaltoggles namespace=default progress= resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Processing item" backup=default/backup-test1 logSource="pkg/backup/backup.go:339" name=hc-global-sample-behavior namespace=default progress= resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Backing up item" backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:121" name=hc-global-sample-behavior namespace=default resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Backed up 2760 items out of an estimated total of 2788 (estimate will change throughout the backup)" backup=default/backup-test1 logSource="pkg/backup/backup.go:379" name=hc-global-sample-behavior namespace=default progress= resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Processing item" backup=default/backup-test1 logSource="pkg/backup/backup.go:339" name=hc-nzdu namespace=default progress= resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Backing up item" backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:121" name=hc-nzdu namespace=default resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Backed up 2761 items out of an estimated total of 2788 (estimate will change throughout the backup)" backup=default/backup-test1 logSource="pkg/backup/backup.go:379" name=hc-nzdu namespace=default progress= resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Processing item" backup=default/backup-test1 logSource="pkg/backup/backup.go:339" name=hc-sample-fancy-behavior namespace=default progress= resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Backing up item" backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:121" name=hc-sample-fancy-behavior namespace=default resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Backed up 2762 items out of an estimated total of 2788 (estimate will change throughout the backup)" backup=default/backup-test1 logSource="pkg/backup/backup.go:379" name=hc-sample-fancy-behavior namespace=default progress= resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Processing item" backup=default/backup-test1 logSource="pkg/backup/backup.go:339" name=hc-sample-planned-behavior namespace=default progress= resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Backing up item" backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:121" name=hc-sample-planned-behavior namespace=default resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Backed up 2763 items out of an estimated total of 2788 (estimate will change throughout the backup)" backup=default/backup-test1 logSource="pkg/backup/backup.go:379" name=hc-sample-planned-behavior namespace=default progress= resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Processing item" backup=default/backup-test1 logSource="pkg/backup/backup.go:339" name=hdl-dioperator-di-tenant-status namespace=default progress= resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Backing up item" backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:121" name=hdl-dioperator-di-tenant-status namespace=default resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Backed up 2764 items out of an estimated total of 2788 (estimate will change throughout the backup)" backup=default/backup-test1 logSource="pkg/backup/backup.go:379" name=hdl-dioperator-di-tenant-status namespace=default progress= resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Processing item" backup=default/backup-test1 logSource="pkg/backup/backup.go:339" name=hdl-iqserver-sapdba-uid namespace=default progress= resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Backing up item" backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:121" name=hdl-iqserver-sapdba-uid namespace=default resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Backed up 2765 items out of an estimated total of 2788 (estimate will change throughout the backup)" backup=default/backup-test1 logSource="pkg/backup/backup.go:379" name=hdl-iqserver-sapdba-uid namespace=default progress= resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Processing item" backup=default/backup-test1 logSource="pkg/backup/backup.go:339" name=hdl-uidscheme namespace=default progress= resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Backing up item" backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:121" name=hdl-uidscheme namespace=default resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Backed up 2766 items out of an estimated total of 2788 (estimate will change throughout the backup)" backup=default/backup-test1 logSource="pkg/backup/backup.go:379" name=hdl-uidscheme namespace=default progress= resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Processing item" backup=default/backup-test1 logSource="pkg/backup/backup.go:339" name=hxtp-ase-dedicated-users-enabled namespace=default progress= resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Backing up item" backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:121" name=hxtp-ase-dedicated-users-enabled namespace=default resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Backed up 2767 items out of an estimated total of 2788 (estimate will change throughout the backup)" backup=default/backup-test1 logSource="pkg/backup/backup.go:379" name=hxtp-ase-dedicated-users-enabled namespace=default progress= resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Processing item" backup=default/backup-test1 logSource="pkg/backup/backup.go:339" name=hxtp-ase-job-scheduler-enabled namespace=default progress= resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Backing up item" backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:121" name=hxtp-ase-job-scheduler-enabled namespace=default resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Backed up 2768 items out of an estimated total of 2788 (estimate will change throughout the backup)" backup=default/backup-test1 logSource="pkg/backup/backup.go:379" name=hxtp-ase-job-scheduler-enabled namespace=default progress= resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Processing item" backup=default/backup-test1 logSource="pkg/backup/backup.go:339" name=hxtp-ase-operator-backup-server-logging namespace=default progress= resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Backing up item" backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:121" name=hxtp-ase-operator-backup-server-logging namespace=default resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Backed up 2769 items out of an estimated total of 2788 (estimate will change throughout the backup)" backup=default/backup-test1 logSource="pkg/backup/backup.go:379" name=hxtp-ase-operator-backup-server-logging namespace=default progress= resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Processing item" backup=default/backup-test1 logSource="pkg/backup/backup.go:339" name=hxtp-ase-operator-backup-storage-policy namespace=default progress= resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Backing up item" backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:121" name=hxtp-ase-operator-backup-storage-policy namespace=default resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Backed up 2770 items out of an estimated total of 2788 (estimate will change throughout the backup)" backup=default/backup-test1 logSource="pkg/backup/backup.go:379" name=hxtp-ase-operator-backup-storage-policy namespace=default progress= resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Processing item" backup=default/backup-test1 logSource="pkg/backup/backup.go:339" name=hxtp-ase-operator-enable-minor-version-upgrade namespace=default progress= resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Backing up item" backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:121" name=hxtp-ase-operator-enable-minor-version-upgrade namespace=default resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Backed up 2771 items out of an estimated total of 2788 (estimate will change throughout the backup)" backup=default/backup-test1 logSource="pkg/backup/backup.go:379" name=hxtp-ase-operator-enable-minor-version-upgrade namespace=default progress= resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Processing item" backup=default/backup-test1 logSource="pkg/backup/backup.go:339" name=hxtp-ase-operator-pre-upgrade-backup-enabled namespace=default progress= resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Backing up item" backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:121" name=hxtp-ase-operator-pre-upgrade-backup-enabled namespace=default resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Backed up 2772 items out of an estimated total of 2788 (estimate will change throughout the backup)" backup=default/backup-test1 logSource="pkg/backup/backup.go:379" name=hxtp-ase-operator-pre-upgrade-backup-enabled namespace=default progress= resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Processing item" backup=default/backup-test1 logSource="pkg/backup/backup.go:339" name=hxtp-external-dataserver-port-enabled namespace=default progress= resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Backing up item" backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:121" name=hxtp-external-dataserver-port-enabled namespace=default resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Backed up 2773 items out of an estimated total of 2788 (estimate will change throughout the backup)" backup=default/backup-test1 logSource="pkg/backup/backup.go:379" name=hxtp-external-dataserver-port-enabled namespace=default progress= resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Processing item" backup=default/backup-test1 logSource="pkg/backup/backup.go:339" name=hxtp-external-repserver-port-enabled namespace=default progress= resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Backing up item" backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:121" name=hxtp-external-repserver-port-enabled namespace=default resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Backed up 2774 items out of an estimated total of 2788 (estimate will change throughout the backup)" backup=default/backup-test1 logSource="pkg/backup/backup.go:379" name=hxtp-external-repserver-port-enabled namespace=default progress= resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Processing item" backup=default/backup-test1 logSource="pkg/backup/backup.go:339" name=hxtp-srs-operator-pre-upgrade-backup-enabled namespace=default progress= resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Backing up item" backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:121" name=hxtp-srs-operator-pre-upgrade-backup-enabled namespace=default resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Backed up 2775 items out of an estimated total of 2788 (estimate will change throughout the backup)" backup=default/backup-test1 logSource="pkg/backup/backup.go:379" name=hxtp-srs-operator-pre-upgrade-backup-enabled namespace=default progress= resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Processing item" backup=default/backup-test1 logSource="pkg/backup/backup.go:339" name=kms-mode namespace=default progress= resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Backing up item" backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:121" name=kms-mode namespace=default resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Backed up 2776 items out of an estimated total of 2788 (estimate will change throughout the backup)" backup=default/backup-test1 logSource="pkg/backup/backup.go:379" name=kms-mode namespace=default progress= resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Processing item" backup=default/backup-test1 logSource="pkg/backup/backup.go:339" name=system-pki-disabled namespace=default progress= resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Backing up item" backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:121" name=system-pki-disabled namespace=default resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Backed up 2777 items out of an estimated total of 2788 (estimate will change throughout the backup)" backup=default/backup-test1 logSource="pkg/backup/backup.go:379" name=system-pki-disabled namespace=default progress= resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Processing item" backup=default/backup-test1 logSource="pkg/backup/backup.go:339" name=tenant-system-user-password-rotator namespace=default progress= resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Backing up item" backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:121" name=tenant-system-user-password-rotator namespace=default resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Backed up 2778 items out of an estimated total of 2788 (estimate will change throughout the backup)" backup=default/backup-test1 logSource="pkg/backup/backup.go:379" name=tenant-system-user-password-rotator namespace=default progress= resource=behaviors.toggles.hc.sap.com
time="2022-05-09T06:35:05Z" level=info msg="Processing item" backup=default/backup-test1 logSource="pkg/backup/backup.go:339" name=alt-domain-ingress namespace=default progress= resource=ingresses.extensions
time="2022-05-09T06:35:05Z" level=info msg="Backing up item" backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:121" name=alt-domain-ingress namespace=default resource=ingresses.extensions
time="2022-05-09T06:35:05Z" level=info msg="Backed up 2779 items out of an estimated total of 2788 (estimate will change throughout the backup)" backup=default/backup-test1 logSource="pkg/backup/backup.go:379" name=alt-domain-ingress namespace=default progress= resource=ingresses.extensions
time="2022-05-09T06:35:05Z" level=info msg="Processing item" backup=default/backup-test1 logSource="pkg/backup/backup.go:339" name=central-connection-mgmt-sync-ingress namespace=default progress= resource=ingresses.extensions
time="2022-05-09T06:35:05Z" level=info msg="Backing up item" backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:121" name=central-connection-mgmt-sync-ingress namespace=default resource=ingresses.extensions
time="2022-05-09T06:35:05Z" level=info msg="Backed up 2780 items out of an estimated total of 2788 (estimate will change throughout the backup)" backup=default/backup-test1 logSource="pkg/backup/backup.go:379" name=central-connection-mgmt-sync-ingress namespace=default progress= resource=ingresses.extensions
time="2022-05-09T06:35:05Z" level=info msg="Processing item" backup=default/backup-test1 logSource="pkg/backup/backup.go:339" name=api namespace=hc-idm progress= resource=ingresses.extensions
time="2022-05-09T06:35:05Z" level=info msg="Backing up item" backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:121" name=api namespace=hc-idm resource=ingresses.extensions
time="2022-05-09T06:35:05Z" level=info msg="Backed up 2781 items out of an estimated total of 2788 (estimate will change throughout the backup)" backup=default/backup-test1 logSource="pkg/backup/backup.go:379" name=api namespace=hc-idm progress= resource=ingresses.extensions
time="2022-05-09T06:35:05Z" level=info msg="Processing item" backup=default/backup-test1 logSource="pkg/backup/backup.go:339" name=external-auth-oauth2 namespace=monitoring progress= resource=ingresses.extensions
time="2022-05-09T06:35:05Z" level=info msg="Backing up item" backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:121" name=external-auth-oauth2 namespace=monitoring resource=ingresses.extensions
time="2022-05-09T06:35:06Z" level=info msg="Backed up 2782 items out of an estimated total of 2788 (estimate will change throughout the backup)" backup=default/backup-test1 logSource="pkg/backup/backup.go:379" name=external-auth-oauth2 namespace=monitoring progress= resource=ingresses.extensions
time="2022-05-09T06:35:06Z" level=info msg="Processing item" backup=default/backup-test1 logSource="pkg/backup/backup.go:339" name=grafana namespace=monitoring progress= resource=ingresses.extensions
time="2022-05-09T06:35:06Z" level=info msg="Backing up item" backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:121" name=grafana namespace=monitoring resource=ingresses.extensions
time="2022-05-09T06:35:06Z" level=info msg="Backed up 2783 items out of an estimated total of 2788 (estimate will change throughout the backup)" backup=default/backup-test1 logSource="pkg/backup/backup.go:379" name=grafana namespace=monitoring progress= resource=ingresses.extensions
time="2022-05-09T06:35:06Z" level=info msg="Processing item" backup=default/backup-test1 logSource="pkg/backup/backup.go:339" name=loki namespace=monitoring progress= resource=ingresses.extensions
time="2022-05-09T06:35:06Z" level=info msg="Backing up item" backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:121" name=loki namespace=monitoring resource=ingresses.extensions
time="2022-05-09T06:35:06Z" level=info msg="Backed up 2784 items out of an estimated total of 2788 (estimate will change throughout the backup)" backup=default/backup-test1 logSource="pkg/backup/backup.go:379" name=loki namespace=monitoring progress= resource=ingresses.extensions
time="2022-05-09T06:35:06Z" level=info msg="Processing item" backup=default/backup-test1 logSource="pkg/backup/backup.go:339" name=loki-longterm namespace=monitoring progress= resource=ingresses.extensions
time="2022-05-09T06:35:06Z" level=info msg="Backing up item" backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:121" name=loki-longterm namespace=monitoring resource=ingresses.extensions
time="2022-05-09T06:35:06Z" level=info msg="Backed up 2785 items out of an estimated total of 2788 (estimate will change throughout the backup)" backup=default/backup-test1 logSource="pkg/backup/backup.go:379" name=loki-longterm namespace=monitoring progress= resource=ingresses.extensions
time="2022-05-09T06:35:06Z" level=info msg="Processing item" backup=default/backup-test1 logSource="pkg/backup/backup.go:339" name=oauth2-proxy namespace=monitoring progress= resource=ingresses.extensions
time="2022-05-09T06:35:06Z" level=info msg="Backing up item" backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:121" name=oauth2-proxy namespace=monitoring resource=ingresses.extensions
time="2022-05-09T06:35:06Z" level=info msg="Backed up 2786 items out of an estimated total of 2788 (estimate will change throughout the backup)" backup=default/backup-test1 logSource="pkg/backup/backup.go:379" name=oauth2-proxy namespace=monitoring progress= resource=ingresses.extensions
time="2022-05-09T06:35:06Z" level=info msg="Processing item" backup=default/backup-test1 logSource="pkg/backup/backup.go:339" name=prometheus-federation-tech namespace=monitoring progress= resource=ingresses.extensions
time="2022-05-09T06:35:06Z" level=info msg="Backing up item" backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:121" name=prometheus-federation-tech namespace=monitoring resource=ingresses.extensions
time="2022-05-09T06:35:06Z" level=info msg="Backed up 2787 items out of an estimated total of 2788 (estimate will change throughout the backup)" backup=default/backup-test1 logSource="pkg/backup/backup.go:379" name=prometheus-federation-tech namespace=monitoring progress= resource=ingresses.extensions
time="2022-05-09T06:35:06Z" level=info msg="Processing item" backup=default/backup-test1 logSource="pkg/backup/backup.go:339" name=prometheus-tech namespace=monitoring progress= resource=ingresses.extensions
time="2022-05-09T06:35:06Z" level=info msg="Backing up item" backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:121" name=prometheus-tech namespace=monitoring resource=ingresses.extensions
time="2022-05-09T06:35:06Z" level=info msg="Backed up 2788 items out of an estimated total of 2788 (estimate will change throughout the backup)" backup=default/backup-test1 logSource="pkg/backup/backup.go:379" name=prometheus-tech namespace=monitoring progress= resource=ingresses.extensions
time="2022-05-09T06:35:06Z" level=info msg="Found associated CRD kongclusterplugins.configuration.konghq.com to add to backup" backup=default/backup-test1 logSource="pkg/backup/backup.go:461"
time="2022-05-09T06:35:06Z" level=info msg="Skipping item because it's already been backed up." backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:115" name=kongclusterplugins.configuration.konghq.com namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2022-05-09T06:35:06Z" level=info msg="Found associated CRD upgradestrategies.products.hc.sap.com to add to backup" backup=default/backup-test1 logSource="pkg/backup/backup.go:461"
time="2022-05-09T06:35:06Z" level=info msg="Skipping item because it's already been backed up." backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:115" name=upgradestrategies.products.hc.sap.com namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2022-05-09T06:35:06Z" level=info msg="Found associated CRD kongplugins.configuration.konghq.com to add to backup" backup=default/backup-test1 logSource="pkg/backup/backup.go:461"
time="2022-05-09T06:35:06Z" level=info msg="Skipping item because it's already been backed up." backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:115" name=kongplugins.configuration.konghq.com namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2022-05-09T06:35:06Z" level=info msg="Found associated CRD workflowtemplates.argoproj.io to add to backup" backup=default/backup-test1 logSource="pkg/backup/backup.go:461"
time="2022-05-09T06:35:06Z" level=info msg="Skipping item because it's already been backed up." backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:115" name=workflowtemplates.argoproj.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2022-05-09T06:35:06Z" level=info msg="Found associated CRD productconfignotifiers.products.hc.sap.com to add to backup" backup=default/backup-test1 logSource="pkg/backup/backup.go:461"
time="2022-05-09T06:35:06Z" level=info msg="Skipping item because it's already been backed up." backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:115" name=productconfignotifiers.products.hc.sap.com namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2022-05-09T06:35:06Z" level=info msg="Found associated CRD verticalpodautoscalercheckpoints.autoscaling.k8s.io to add to backup" backup=default/backup-test1 logSource="pkg/backup/backup.go:461"
time="2022-05-09T06:35:06Z" level=info msg="Skipping item because it's already been backed up." backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:115" name=verticalpodautoscalercheckpoints.autoscaling.k8s.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2022-05-09T06:35:06Z" level=info msg="Found associated CRD issuers.cert-manager.io to add to backup" backup=default/backup-test1 logSource="pkg/backup/backup.go:461"
time="2022-05-09T06:35:06Z" level=info msg="Skipping item because it's already been backed up." backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:115" name=issuers.cert-manager.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2022-05-09T06:35:06Z" level=info msg="Found associated CRD certificaterequests.cert-manager.io to add to backup" backup=default/backup-test1 logSource="pkg/backup/backup.go:461"
time="2022-05-09T06:35:06Z" level=info msg="Skipping item because it's already been backed up." backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:115" name=certificaterequests.cert-manager.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2022-05-09T06:35:06Z" level=info msg="Found associated CRD productplugins.products.hc.sap.com to add to backup" backup=default/backup-test1 logSource="pkg/backup/backup.go:461"
time="2022-05-09T06:35:06Z" level=info msg="Skipping item because it's already been backed up." backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:115" name=productplugins.products.hc.sap.com namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2022-05-09T06:35:06Z" level=info msg="Found associated CRD felixconfigurations.crd.projectcalico.org to add to backup" backup=default/backup-test1 logSource="pkg/backup/backup.go:461"
time="2022-05-09T06:35:06Z" level=info msg="Skipping item because it's already been backed up." backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:115" name=felixconfigurations.crd.projectcalico.org namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2022-05-09T06:35:07Z" level=info msg="Found associated CRD backups.velero.io to add to backup" backup=default/backup-test1 logSource="pkg/backup/backup.go:461"
time="2022-05-09T06:35:07Z" level=info msg="Skipping item because it's already been backed up." backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:115" name=backups.velero.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2022-05-09T06:35:07Z" level=info msg="Found associated CRD clusterissuers.cert-manager.io to add to backup" backup=default/backup-test1 logSource="pkg/backup/backup.go:461"
time="2022-05-09T06:35:07Z" level=info msg="Skipping item because it's already been backed up." backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:115" name=clusterissuers.cert-manager.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2022-05-09T06:35:07Z" level=info msg="Found associated CRD kongconsumers.configuration.konghq.com to add to backup" backup=default/backup-test1 logSource="pkg/backup/backup.go:461"
time="2022-05-09T06:35:07Z" level=info msg="Skipping item because it's already been backed up." backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:115" name=kongconsumers.configuration.konghq.com namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2022-05-09T06:35:07Z" level=info msg="Found associated CRD servicemonitors.monitoring.coreos.com to add to backup" backup=default/backup-test1 logSource="pkg/backup/backup.go:461"
time="2022-05-09T06:35:07Z" level=info msg="Skipping item because it's already been backed up." backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:115" name=servicemonitors.monitoring.coreos.com namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2022-05-09T06:35:07Z" level=info msg="Found associated CRD alertmanagerconfigs.monitoring.coreos.com to add to backup" backup=default/backup-test1 logSource="pkg/backup/backup.go:461"
time="2022-05-09T06:35:07Z" level=info msg="Skipping item because it's already been backed up." backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:115" name=alertmanagerconfigs.monitoring.coreos.com namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2022-05-09T06:35:08Z" level=info msg="Found associated CRD certificates.cert-manager.io to add to backup" backup=default/backup-test1 logSource="pkg/backup/backup.go:461"
time="2022-05-09T06:35:08Z" level=info msg="Skipping item because it's already been backed up." backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:115" name=certificates.cert-manager.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2022-05-09T06:35:08Z" level=info msg="Found associated CRD volumesnapshotlocations.velero.io to add to backup" backup=default/backup-test1 logSource="pkg/backup/backup.go:461"
time="2022-05-09T06:35:08Z" level=info msg="Skipping item because it's already been backed up." backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:115" name=volumesnapshotlocations.velero.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2022-05-09T06:35:08Z" level=info msg="Found associated CRD productversionmaps.products.hc.sap.com to add to backup" backup=default/backup-test1 logSource="pkg/backup/backup.go:461"
time="2022-05-09T06:35:08Z" level=info msg="Skipping item because it's already been backed up." backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:115" name=productversionmaps.products.hc.sap.com namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2022-05-09T06:35:08Z" level=info msg="Found associated CRD clusterworkflowtemplates.argoproj.io to add to backup" backup=default/backup-test1 logSource="pkg/backup/backup.go:461"
time="2022-05-09T06:35:08Z" level=info msg="Skipping item because it's already been backed up." backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:115" name=clusterworkflowtemplates.argoproj.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2022-05-09T06:35:08Z" level=info msg="Found associated CRD behaviors.toggles.hc.sap.com to add to backup" backup=default/backup-test1 logSource="pkg/backup/backup.go:461"
time="2022-05-09T06:35:08Z" level=info msg="Skipping item because it's already been backed up." backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:115" name=behaviors.toggles.hc.sap.com namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2022-05-09T06:35:08Z" level=info msg="Found associated CRD ippools.crd.projectcalico.org to add to backup" backup=default/backup-test1 logSource="pkg/backup/backup.go:461"
time="2022-05-09T06:35:08Z" level=info msg="Skipping item because it's already been backed up." backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:115" name=ippools.crd.projectcalico.org namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2022-05-09T06:35:08Z" level=info msg="Found associated CRD podmonitors.monitoring.coreos.com to add to backup" backup=default/backup-test1 logSource="pkg/backup/backup.go:461"
time="2022-05-09T06:35:08Z" level=info msg="Skipping item because it's already been backed up." backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:115" name=podmonitors.monitoring.coreos.com namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2022-05-09T06:35:08Z" level=info msg="Found associated CRD alertmanagers.monitoring.coreos.com to add to backup" backup=default/backup-test1 logSource="pkg/backup/backup.go:461"
time="2022-05-09T06:35:08Z" level=info msg="Skipping item because it's already been backed up." backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:115" name=alertmanagers.monitoring.coreos.com namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2022-05-09T06:35:08Z" level=info msg="Found associated CRD schedules.velero.io to add to backup" backup=default/backup-test1 logSource="pkg/backup/backup.go:461"
time="2022-05-09T06:35:08Z" level=info msg="Skipping item because it's already been backed up." backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:115" name=schedules.velero.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2022-05-09T06:35:08Z" level=info msg="Found associated CRD dnsentries.dns.gardener.cloud to add to backup" backup=default/backup-test1 logSource="pkg/backup/backup.go:461"
time="2022-05-09T06:35:08Z" level=info msg="Skipping item because it's already been backed up." backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:115" name=dnsentries.dns.gardener.cloud namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2022-05-09T06:35:08Z" level=info msg="Found associated CRD dnsproviders.dns.gardener.cloud to add to backup" backup=default/backup-test1 logSource="pkg/backup/backup.go:461"
time="2022-05-09T06:35:08Z" level=info msg="Skipping item because it's already been backed up." backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:115" name=dnsproviders.dns.gardener.cloud namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2022-05-09T06:35:08Z" level=info msg="Found associated CRD affinities.hana.sap.com to add to backup" backup=default/backup-test1 logSource="pkg/backup/backup.go:461"
time="2022-05-09T06:35:08Z" level=info msg="Skipping item because it's already been backed up." backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:115" name=affinities.hana.sap.com namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2022-05-09T06:35:08Z" level=info msg="Found associated CRD cronworkflows.argoproj.io to add to backup" backup=default/backup-test1 logSource="pkg/backup/backup.go:461"
time="2022-05-09T06:35:08Z" level=info msg="Skipping item because it's already been backed up." backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:115" name=cronworkflows.argoproj.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2022-05-09T06:35:09Z" level=info msg="Found associated CRD verticalpodautoscalers.autoscaling.k8s.io to add to backup" backup=default/backup-test1 logSource="pkg/backup/backup.go:461"
time="2022-05-09T06:35:09Z" level=info msg="Skipping item because it's already been backed up." backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:115" name=verticalpodautoscalers.autoscaling.k8s.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2022-05-09T06:35:09Z" level=info msg="Found associated CRD backupstoragelocations.velero.io to add to backup" backup=default/backup-test1 logSource="pkg/backup/backup.go:461"
time="2022-05-09T06:35:09Z" level=info msg="Skipping item because it's already been backed up." backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:115" name=backupstoragelocations.velero.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2022-05-09T06:35:09Z" level=info msg="Found associated CRD prometheuses.monitoring.coreos.com to add to backup" backup=default/backup-test1 logSource="pkg/backup/backup.go:461"
time="2022-05-09T06:35:09Z" level=info msg="Skipping item because it's already been backed up." backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:115" name=prometheuses.monitoring.coreos.com namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2022-05-09T06:35:09Z" level=info msg="Found associated CRD prometheusrules.monitoring.coreos.com to add to backup" backup=default/backup-test1 logSource="pkg/backup/backup.go:461"
time="2022-05-09T06:35:09Z" level=info msg="Skipping item because it's already been backed up." backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:115" name=prometheusrules.monitoring.coreos.com namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2022-05-09T06:35:09Z" level=info msg="Found associated CRD rolloutstrategies.enablements.hc.sap.com to add to backup" backup=default/backup-test1 logSource="pkg/backup/backup.go:461"
time="2022-05-09T06:35:09Z" level=info msg="Skipping item because it's already been backed up." backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:115" name=rolloutstrategies.enablements.hc.sap.com namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2022-05-09T06:35:09Z" level=info msg="Found associated CRD features.toggles.hc.sap.com to add to backup" backup=default/backup-test1 logSource="pkg/backup/backup.go:461"
time="2022-05-09T06:35:09Z" level=info msg="Skipping item because it's already been backed up." backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:115" name=features.toggles.hc.sap.com namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2022-05-09T06:35:09Z" level=info msg="Found associated CRD clusterinformations.crd.projectcalico.org to add to backup" backup=default/backup-test1 logSource="pkg/backup/backup.go:461"
time="2022-05-09T06:35:09Z" level=info msg="Skipping item because it's already been backed up." backup=default/backup-test1 logSource="pkg/backup/item_backupper.go:115" name=clusterinformations.crd.projectcalico.org namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2022-05-09T06:35:09Z" level=info msg="Backed up a total of 2788 items" backup=default/backup-test1 logSource="pkg/backup/backup.go:404" progress=
time="2022-05-09T06:35:09Z" level=info msg="Setting up backup store to persist the backup" backup=default/backup-test1 logSource="pkg/controller/backup_controller.go:663"
time="2022-05-09T06:35:16Z" level=info msg="Plugin process exited - restarting." backup=default/backup-test1 logSource="pkg/plugin/clientmgmt/restartable_process.go:155"
time="2022-05-09T06:35:21Z" level=info msg="Backup completed" backup=default/backup-test1 controller=backup logSource="pkg/controller/backup_controller.go:673"
time="2022-05-09T06:35:21Z" level=error msg="backup failed" controller=backup error="[rpc error: code = Unavailable desc = error reading from server: EOF, rpc error: code = Unavailable desc = connection error: desc = \"transport: error while dialing: dial unix /tmp/plugin1201909614: connect: connection refused\"]" key=default/backup-test1 logSource="pkg/controller/backup_controller.go:282"

My velero is deployed in 'default' ns, and velero debug --help shows: -n, --namespace string The namespace in which Velero should operate (default "default")

But when I run :velero debug --backup backup-test1 --namespace default. it shows : An error occurred: velero deployment does not exist in namespace: default

chrislinan commented 2 years ago

Maybe there is something wrong with my configration? I put my configration here: backupstoragelocation:

apiVersion: velero.io/v1
kind: BackupStorageLocation
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"velero.io/v1","kind":"BackupStorageLocation","metadata":{"annotations":{},"labels":{"app.kubernetes.io/name":"velero","hc.sap.com/hc-provisioned":"True","helm.sh/chart":"hc-disaster-recovery-2.9.15"},"name":"default","namespace":"default"},"spec":{"backupSyncPeriod":"6h","config":{"resourceGroup":"disaster-recovery","storageAccount":"disasterrecdevazrhna","subscriptionId":"a5dc0748-xxxx-001eb9ca016a"},"credential":{"key":"cloud","name":"velero-object-store-credential"},"objectStorage":{"bucket":"disaster-recovery-dev-azure-hana","prefix":"non-prod/shoot--xxxx-haas"},"provider":"azure"}}
  creationTimestamp: "2022-05-08T10:11:01Z"
  generation: 3
  labels:
    app.kubernetes.io/name: velero
    hc.sap.com/hc-provisioned: "True"
    helm.sh/chart: hc-disaster-recovery-2.9.15
  name: default
  namespace: default
  resourceVersion: "655883"
  uid: 665c562f-8ec2-4f30-9ad0-a9b0153a976e
spec:
  backupSyncPeriod: 6h
  config:
    resourceGroup: disaster-recovery
    storageAccount: disasterrecdevazrhna
    subscriptionId: a5dc0748xxxxxx01eb9ca016a
  credential:
    key: cloud
    name: velero-object-store-credential
  default: true
  objectStorage:
    bucket: disaster-recovery-dev-azure-hana
    prefix: non-prod/shoot--xxxx-haas
  provider: azure
status:
  lastSyncedTime: "2022-05-09T04:12:30Z"
  lastValidationTime: "2022-05-09T06:44:03Z"
  phase: Available

volumesnapshotlocation:

apiVersion: velero.io/v1
kind: VolumeSnapshotLocation
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"velero.io/v1","kind":"VolumeSnapshotLocation","metadata":{"annotations":{},"labels":{"app.kubernetes.io/name":"velero","hc.sap.com/hc-provisioned":"True","helm.sh/chart":"hc-disaster-recovery-2.9.15"},"name":"default","namespace":"default"},"spec":{"config":{"incremental":"true","resourceGroup":"disaster-recovery","subscriptionId":"a68ae472xxxxxxxx070acd2d"},"provider":"azure"}}
  creationTimestamp: "2022-05-08T10:11:01Z"
  generation: 2
  labels:
    app.kubernetes.io/name: velero
    hc.sap.com/hc-provisioned: "True"
    helm.sh/chart: hc-disaster-recovery-2.9.15
  name: default
  namespace: default
  resourceVersion: "66043"
  uid: 2839c9f0-b410-41a5-80ce-88cb5ab958e4
spec:
  config:
    incremental: "true"
    resourceGroup: disaster-recovery
    subscriptionId: a68ae472-xxxxxx-24f5070acd2d
  provider: azure

Secret:

apiVersion: v1
data:
  cloud: ICAgIEFaVVJ5kcWlvxxxjQ2xvdWQK
kind: Secret
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"cloud":"xxxxvdWQK"},"kind":"Secret","metadata":{"annotations":{},"labels":{"app.kubernetes.io/name":"velero","hc.sap.com/hc-provisioned":"True","helm.sh/chart":"hc-disaster-recovery-2.9.15"},"name":"velero-object-store-credential","namespace":"default"},"type":"Opaque"}
  creationTimestamp: "2022-05-08T10:03:30Z"
  labels:
    app.kubernetes.io/name: velero
    hc.sap.com/hc-provisioned: "True"
    helm.sh/chart: hc-disaster-recovery-2.9.15
  name: velero-object-store-credential
  namespace: default
  resourceVersion: "65966"
  uid: 666f9279-1cc4-48e0-ba8a-d684db165ad5
type: Opaque

The secret is the credential for Azure, it looks like:

    AZURE_SUBSCRIPTION_ID=a5dc0748-xxxxxx-001eb9ca016a
    AZURE_TENANT_ID=42f7676c-xxxxxx-dc2d99791af7
    AZURE_CLIENT_ID=9d398e-xxxxxx-ae4f10d83a01
    AZURE_CLIENT_SECRET=kHxxxxxxxxxxGaRN
    AZURE_RESOURCE_GROUP=shoot--xxxxxx-haas
    AZURE_CLOUD_NAME=AzurePublicCloud
chrislinan commented 2 years ago

I download the velero source code of version 1.8, and build it by myself, and I deploy velero by helm chart. Maybe it's the reason that commit ID is dev.

ywk253100 commented 2 years ago

According to the log time="2022-05-09T06:35:21Z" level=error msg="backup failed" controller=backup error="[rpc error: code = Unavailable desc = error reading from server: EOF, rpc error: code = Unavailable desc = connection error: desc = \"transport: error while dialing: dial unix /tmp/plugin1201909614: connect: connection refused\"]" key=default/backup-test1 logSource="pkg/controller/backup_controller.go:282"

it could be caused by some issues with the communication between the velero and azure plugin, is it possible for you to do some further debug to locate the root cause? It's hard for us without your env to debug the issue