search

vmware-tanzu / velero

Backup and migrate Kubernetes applications and their persistent volumes
https://velero.io
Apache License 2.0
8.64k stars 1.39k forks source link

Velero sometimes does not backup API resources installed up to five minutes before backup #3471

Open codegold79 opened 3 years ago

codegold79 commented 3 years ago

I've filed this issue as a bug, but it may just be how Velero is designed to work. If that's the case, I suggest updating the documentation to say that resources need up to five minutes to be recogized by Velero. Suggest ways to make the process faster in the docs as well, such as deleting a Velero pod after installing a resource and Velero will pick it up right away.

@lighting-wings suggested in the Discussions #3295 refreshing discovery just before a backup is performed. It might be better than refreshing every 5 minutes. Only update before a backup.

What steps did you take and what happened:

  1. install Velero
  2. install cert-manager with kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.0.3/cert-manager.yaml
  3. install CRD with kubectl apply -f https://raw.githubusercontent.com/brito-rafa/k8s-webhooks/master/examples-for-projectvelero/case-a/source/case-a-source.yaml and wait for pods to be running.
  4. create namespace with kubectl create namespace rockbands-v1
  5. install a custom resource with kubectl create -n rockbands-v1 -f https://raw.githubusercontent.com/brito-rafa/k8s-webhooks/master/examples-for-projectvelero/case-a/source/music/config/samples/music_v1_rockband.yaml
  6. run velero backup using --include-namespaces rockbands-v1
  7. verify CRs are there with kubectl get rockbands -A -o yaml
  8. checked backup tarball, and the installed CR wasn't there

What did you expect to happen: I expected the backup tarball to contain the custom resource that was on the cluster during backup.

The output of the following commands will help us better understand what's going on:

Phase: Completed

Errors: 0 Warnings: 0

Namespaces: Included: rockbands-v1 Excluded:

Resources: Included: * Excluded: Cluster-scoped: auto

Label selector:

Storage Location: default

Velero-Native Snapshot PVs: auto

TTL: 720h0m0s

Hooks:

Backup Format Version: 1.1.0

Started: 2021-02-17 11:41:40 -0800 PST Completed: 2021-02-17 11:41:41 -0800 PST

Expiration: 2021-03-19 12:41:40 -0700 PDT

Total items to be backed up: 3 Items backed up: 3

Velero-Native Snapshots: 

- `velero backup logs <backupname>`

time="2021-02-17T19:41:40Z" level=info msg="Setting up backup temp file" backup=velero/fgold-backup-021701 logSource="pkg/controller/backup_controller.go:534" time="2021-02-17T19:41:40Z" level=info msg="Setting up plugin manager" backup=velero/fgold-backup-021701 logSource="pkg/controller/backup_controller.go:541" time="2021-02-17T19:41:40Z" level=info msg="Getting backup item actions" backup=velero/fgold-backup-021701 logSource="pkg/controller/backup_controller.go:545" time="2021-02-17T19:41:40Z" level=info msg="Setting up backup store to check for backup existence" backup=velero/fgold-backup-021701 logSource="pkg/controller/backup_controller.go:551" time="2021-02-17T19:41:40Z" level=info msg="Writing backup version file" backup=velero/fgold-backup-021701 logSource="pkg/backup/backup.go:212" time="2021-02-17T19:41:40Z" level=info msg="Including namespaces: rockbands-v1" backup=velero/fgold-backup-021701 logSource="pkg/backup/backup.go:218" time="2021-02-17T19:41:40Z" level=info msg="Excluding namespaces: " backup=velero/fgold-backup-021701 logSource="pkg/backup/backup.go:219" time="2021-02-17T19:41:40Z" level=info msg="Including resources: *" backup=velero/fgold-backup-021701 logSource="pkg/backup/backup.go:222" time="2021-02-17T19:41:40Z" level=info msg="Excluding resources: " backup=velero/fgold-backup-021701 logSource="pkg/backup/backup.go:223" time="2021-02-17T19:41:40Z" level=info msg="Backing up all pod volumes using restic: false" backup=velero/fgold-backup-021701 logSource="pkg/backup/backup.go:224" time="2021-02-17T19:41:40Z" level=info msg="Getting items for group" backup=velero/fgold-backup-021701 group=v1 logSource="pkg/backup/item_collector.go:76" time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=v1 logSource="pkg/backup/item_collector.go:165" resource=pods time="2021-02-17T19:41:40Z" level=info msg="Listing items" backup=velero/fgold-backup-021701 group=v1 logSource="pkg/backup/item_collector.go:291" namespace=rockbands-v1 resource=pods time="2021-02-17T19:41:40Z" level=info msg="Retrieved 0 items" backup=velero/fgold-backup-021701 group=v1 logSource="pkg/backup/item_collector.go:297" namespace=rockbands-v1 resource=pods time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=v1 logSource="pkg/backup/item_collector.go:165" resource=persistentvolumeclaims time="2021-02-17T19:41:40Z" level=info msg="Listing items" backup=velero/fgold-backup-021701 group=v1 logSource="pkg/backup/item_collector.go:291" namespace=rockbands-v1 resource=persistentvolumeclaims time="2021-02-17T19:41:40Z" level=info msg="Retrieved 0 items" backup=velero/fgold-backup-021701 group=v1 logSource="pkg/backup/item_collector.go:297" namespace=rockbands-v1 resource=persistentvolumeclaims time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=v1 logSource="pkg/backup/item_collector.go:165" resource=persistentvolumes time="2021-02-17T19:41:40Z" level=info msg="Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup" backup=velero/fgold-backup-021701 group=v1 logSource="pkg/backup/item_collector.go:192" resource=persistentvolumes time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=v1 logSource="pkg/backup/item_collector.go:165" resource=configmaps time="2021-02-17T19:41:40Z" level=info msg="Listing items" backup=velero/fgold-backup-021701 group=v1 logSource="pkg/backup/item_collector.go:291" namespace=rockbands-v1 resource=configmaps time="2021-02-17T19:41:40Z" level=info msg="Retrieved 0 items" backup=velero/fgold-backup-021701 group=v1 logSource="pkg/backup/item_collector.go:297" namespace=rockbands-v1 resource=configmaps time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=v1 logSource="pkg/backup/item_collector.go:165" resource=endpoints time="2021-02-17T19:41:40Z" level=info msg="Listing items" backup=velero/fgold-backup-021701 group=v1 logSource="pkg/backup/item_collector.go:291" namespace=rockbands-v1 resource=endpoints time="2021-02-17T19:41:40Z" level=info msg="Retrieved 0 items" backup=velero/fgold-backup-021701 group=v1 logSource="pkg/backup/item_collector.go:297" namespace=rockbands-v1 resource=endpoints time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=v1 logSource="pkg/backup/item_collector.go:165" resource=events time="2021-02-17T19:41:40Z" level=info msg="Listing items" backup=velero/fgold-backup-021701 group=v1 logSource="pkg/backup/item_collector.go:291" namespace=rockbands-v1 resource=events time="2021-02-17T19:41:40Z" level=info msg="Retrieved 0 items" backup=velero/fgold-backup-021701 group=v1 logSource="pkg/backup/item_collector.go:297" namespace=rockbands-v1 resource=events time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=v1 logSource="pkg/backup/item_collector.go:165" resource=limitranges time="2021-02-17T19:41:40Z" level=info msg="Listing items" backup=velero/fgold-backup-021701 group=v1 logSource="pkg/backup/item_collector.go:291" namespace=rockbands-v1 resource=limitranges time="2021-02-17T19:41:40Z" level=info msg="Retrieved 0 items" backup=velero/fgold-backup-021701 group=v1 logSource="pkg/backup/item_collector.go:297" namespace=rockbands-v1 resource=limitranges time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=v1 logSource="pkg/backup/item_collector.go:165" resource=namespaces time="2021-02-17T19:41:40Z" level=info msg="Getting namespace" backup=velero/fgold-backup-021701 group=v1 logSource="pkg/backup/item_collector.go:239" namespace=rockbands-v1 resource=namespaces time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=v1 logSource="pkg/backup/item_collector.go:165" resource=nodes time="2021-02-17T19:41:40Z" level=info msg="Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup" backup=velero/fgold-backup-021701 group=v1 logSource="pkg/backup/item_collector.go:192" resource=nodes time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=v1 logSource="pkg/backup/item_collector.go:165" resource=podtemplates time="2021-02-17T19:41:40Z" level=info msg="Listing items" backup=velero/fgold-backup-021701 group=v1 logSource="pkg/backup/item_collector.go:291" namespace=rockbands-v1 resource=podtemplates time="2021-02-17T19:41:40Z" level=info msg="Retrieved 0 items" backup=velero/fgold-backup-021701 group=v1 logSource="pkg/backup/item_collector.go:297" namespace=rockbands-v1 resource=podtemplates time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=v1 logSource="pkg/backup/item_collector.go:165" resource=replicationcontrollers time="2021-02-17T19:41:40Z" level=info msg="Listing items" backup=velero/fgold-backup-021701 group=v1 logSource="pkg/backup/item_collector.go:291" namespace=rockbands-v1 resource=replicationcontrollers time="2021-02-17T19:41:40Z" level=info msg="Retrieved 0 items" backup=velero/fgold-backup-021701 group=v1 logSource="pkg/backup/item_collector.go:297" namespace=rockbands-v1 resource=replicationcontrollers time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=v1 logSource="pkg/backup/item_collector.go:165" resource=resourcequotas time="2021-02-17T19:41:40Z" level=info msg="Listing items" backup=velero/fgold-backup-021701 group=v1 logSource="pkg/backup/item_collector.go:291" namespace=rockbands-v1 resource=resourcequotas time="2021-02-17T19:41:40Z" level=info msg="Retrieved 0 items" backup=velero/fgold-backup-021701 group=v1 logSource="pkg/backup/item_collector.go:297" namespace=rockbands-v1 resource=resourcequotas time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=v1 logSource="pkg/backup/item_collector.go:165" resource=secrets time="2021-02-17T19:41:40Z" level=info msg="Listing items" backup=velero/fgold-backup-021701 group=v1 logSource="pkg/backup/item_collector.go:291" namespace=rockbands-v1 resource=secrets time="2021-02-17T19:41:40Z" level=info msg="Retrieved 1 items" backup=velero/fgold-backup-021701 group=v1 logSource="pkg/backup/item_collector.go:297" namespace=rockbands-v1 resource=secrets time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=v1 logSource="pkg/backup/item_collector.go:165" resource=serviceaccounts time="2021-02-17T19:41:40Z" level=info msg="Listing items" backup=velero/fgold-backup-021701 group=v1 logSource="pkg/backup/item_collector.go:291" namespace=rockbands-v1 resource=serviceaccounts time="2021-02-17T19:41:40Z" level=info msg="Retrieved 1 items" backup=velero/fgold-backup-021701 group=v1 logSource="pkg/backup/item_collector.go:297" namespace=rockbands-v1 resource=serviceaccounts time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=v1 logSource="pkg/backup/item_collector.go:165" resource=services time="2021-02-17T19:41:40Z" level=info msg="Listing items" backup=velero/fgold-backup-021701 group=v1 logSource="pkg/backup/item_collector.go:291" namespace=rockbands-v1 resource=services time="2021-02-17T19:41:40Z" level=info msg="Retrieved 0 items" backup=velero/fgold-backup-021701 group=v1 logSource="pkg/backup/item_collector.go:297" namespace=rockbands-v1 resource=services time="2021-02-17T19:41:40Z" level=info msg="Getting items for group" backup=velero/fgold-backup-021701 group=apiregistration.k8s.io/v1 logSource="pkg/backup/item_collector.go:76" time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=apiregistration.k8s.io/v1 logSource="pkg/backup/item_collector.go:165" resource=apiservices time="2021-02-17T19:41:40Z" level=info msg="Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup" backup=velero/fgold-backup-021701 group=apiregistration.k8s.io/v1 logSource="pkg/backup/item_collector.go:192" resource=apiservices time="2021-02-17T19:41:40Z" level=info msg="Getting items for group" backup=velero/fgold-backup-021701 group=apiregistration.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:76" time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=apiregistration.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:165" resource=apiservices time="2021-02-17T19:41:40Z" level=info msg="Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup" backup=velero/fgold-backup-021701 group=apiregistration.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:192" resource=apiservices time="2021-02-17T19:41:40Z" level=info msg="Getting items for group" backup=velero/fgold-backup-021701 group=apps/v1 logSource="pkg/backup/item_collector.go:76" time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=apps/v1 logSource="pkg/backup/item_collector.go:165" resource=controllerrevisions time="2021-02-17T19:41:40Z" level=info msg="Listing items" backup=velero/fgold-backup-021701 group=apps/v1 logSource="pkg/backup/item_collector.go:291" namespace=rockbands-v1 resource=controllerrevisions time="2021-02-17T19:41:40Z" level=info msg="Retrieved 0 items" backup=velero/fgold-backup-021701 group=apps/v1 logSource="pkg/backup/item_collector.go:297" namespace=rockbands-v1 resource=controllerrevisions time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=apps/v1 logSource="pkg/backup/item_collector.go:165" resource=daemonsets time="2021-02-17T19:41:40Z" level=info msg="Listing items" backup=velero/fgold-backup-021701 group=apps/v1 logSource="pkg/backup/item_collector.go:291" namespace=rockbands-v1 resource=daemonsets time="2021-02-17T19:41:40Z" level=info msg="Retrieved 0 items" backup=velero/fgold-backup-021701 group=apps/v1 logSource="pkg/backup/item_collector.go:297" namespace=rockbands-v1 resource=daemonsets time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=apps/v1 logSource="pkg/backup/item_collector.go:165" resource=deployments time="2021-02-17T19:41:40Z" level=info msg="Listing items" backup=velero/fgold-backup-021701 group=apps/v1 logSource="pkg/backup/item_collector.go:291" namespace=rockbands-v1 resource=deployments time="2021-02-17T19:41:40Z" level=info msg="Retrieved 0 items" backup=velero/fgold-backup-021701 group=apps/v1 logSource="pkg/backup/item_collector.go:297" namespace=rockbands-v1 resource=deployments time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=apps/v1 logSource="pkg/backup/item_collector.go:165" resource=replicasets time="2021-02-17T19:41:40Z" level=info msg="Listing items" backup=velero/fgold-backup-021701 group=apps/v1 logSource="pkg/backup/item_collector.go:291" namespace=rockbands-v1 resource=replicasets time="2021-02-17T19:41:40Z" level=info msg="Retrieved 0 items" backup=velero/fgold-backup-021701 group=apps/v1 logSource="pkg/backup/item_collector.go:297" namespace=rockbands-v1 resource=replicasets time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=apps/v1 logSource="pkg/backup/item_collector.go:165" resource=statefulsets time="2021-02-17T19:41:40Z" level=info msg="Listing items" backup=velero/fgold-backup-021701 group=apps/v1 logSource="pkg/backup/item_collector.go:291" namespace=rockbands-v1 resource=statefulsets time="2021-02-17T19:41:40Z" level=info msg="Retrieved 0 items" backup=velero/fgold-backup-021701 group=apps/v1 logSource="pkg/backup/item_collector.go:297" namespace=rockbands-v1 resource=statefulsets time="2021-02-17T19:41:40Z" level=info msg="Getting items for group" backup=velero/fgold-backup-021701 group=events.k8s.io/v1 logSource="pkg/backup/item_collector.go:76" time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=events.k8s.io/v1 logSource="pkg/backup/item_collector.go:165" resource=events time="2021-02-17T19:41:40Z" level=info msg="Skipping resource because it cohabitates and we've already processed it" backup=velero/fgold-backup-021701 cohabitatingResource1=events cohabitatingResource2=events.events.k8s.io group=events.k8s.io/v1 logSource="pkg/backup/item_collector.go:213" resource=events time="2021-02-17T19:41:40Z" level=info msg="Getting items for group" backup=velero/fgold-backup-021701 group=events.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:76" time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=events.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:165" resource=events time="2021-02-17T19:41:40Z" level=info msg="Skipping resource because it cohabitates and we've already processed it" backup=velero/fgold-backup-021701 cohabitatingResource1=events cohabitatingResource2=events.events.k8s.io group=events.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:213" resource=events time="2021-02-17T19:41:40Z" level=info msg="Getting items for group" backup=velero/fgold-backup-021701 group=autoscaling/v1 logSource="pkg/backup/item_collector.go:76" time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=autoscaling/v1 logSource="pkg/backup/item_collector.go:165" resource=horizontalpodautoscalers time="2021-02-17T19:41:40Z" level=info msg="Listing items" backup=velero/fgold-backup-021701 group=autoscaling/v1 logSource="pkg/backup/item_collector.go:291" namespace=rockbands-v1 resource=horizontalpodautoscalers time="2021-02-17T19:41:40Z" level=info msg="Retrieved 0 items" backup=velero/fgold-backup-021701 group=autoscaling/v1 logSource="pkg/backup/item_collector.go:297" namespace=rockbands-v1 resource=horizontalpodautoscalers time="2021-02-17T19:41:40Z" level=info msg="Getting items for group" backup=velero/fgold-backup-021701 group=autoscaling/v2beta1 logSource="pkg/backup/item_collector.go:76" time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=autoscaling/v2beta1 logSource="pkg/backup/item_collector.go:165" resource=horizontalpodautoscalers time="2021-02-17T19:41:40Z" level=info msg="Listing items" backup=velero/fgold-backup-021701 group=autoscaling/v2beta1 logSource="pkg/backup/item_collector.go:291" namespace=rockbands-v1 resource=horizontalpodautoscalers time="2021-02-17T19:41:40Z" level=info msg="Retrieved 0 items" backup=velero/fgold-backup-021701 group=autoscaling/v2beta1 logSource="pkg/backup/item_collector.go:297" namespace=rockbands-v1 resource=horizontalpodautoscalers time="2021-02-17T19:41:40Z" level=info msg="Getting items for group" backup=velero/fgold-backup-021701 group=autoscaling/v2beta2 logSource="pkg/backup/item_collector.go:76" time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=autoscaling/v2beta2 logSource="pkg/backup/item_collector.go:165" resource=horizontalpodautoscalers time="2021-02-17T19:41:40Z" level=info msg="Listing items" backup=velero/fgold-backup-021701 group=autoscaling/v2beta2 logSource="pkg/backup/item_collector.go:291" namespace=rockbands-v1 resource=horizontalpodautoscalers time="2021-02-17T19:41:40Z" level=info msg="Retrieved 0 items" backup=velero/fgold-backup-021701 group=autoscaling/v2beta2 logSource="pkg/backup/item_collector.go:297" namespace=rockbands-v1 resource=horizontalpodautoscalers time="2021-02-17T19:41:40Z" level=info msg="Getting items for group" backup=velero/fgold-backup-021701 group=batch/v1 logSource="pkg/backup/item_collector.go:76" time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=batch/v1 logSource="pkg/backup/item_collector.go:165" resource=jobs time="2021-02-17T19:41:40Z" level=info msg="Listing items" backup=velero/fgold-backup-021701 group=batch/v1 logSource="pkg/backup/item_collector.go:291" namespace=rockbands-v1 resource=jobs time="2021-02-17T19:41:40Z" level=info msg="Retrieved 0 items" backup=velero/fgold-backup-021701 group=batch/v1 logSource="pkg/backup/item_collector.go:297" namespace=rockbands-v1 resource=jobs time="2021-02-17T19:41:40Z" level=info msg="Getting items for group" backup=velero/fgold-backup-021701 group=batch/v1beta1 logSource="pkg/backup/item_collector.go:76" time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=batch/v1beta1 logSource="pkg/backup/item_collector.go:165" resource=cronjobs time="2021-02-17T19:41:40Z" level=info msg="Listing items" backup=velero/fgold-backup-021701 group=batch/v1beta1 logSource="pkg/backup/item_collector.go:291" namespace=rockbands-v1 resource=cronjobs time="2021-02-17T19:41:40Z" level=info msg="Retrieved 0 items" backup=velero/fgold-backup-021701 group=batch/v1beta1 logSource="pkg/backup/item_collector.go:297" namespace=rockbands-v1 resource=cronjobs time="2021-02-17T19:41:40Z" level=info msg="Getting items for group" backup=velero/fgold-backup-021701 group=certificates.k8s.io/v1 logSource="pkg/backup/item_collector.go:76" time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=certificates.k8s.io/v1 logSource="pkg/backup/item_collector.go:165" resource=certificatesigningrequests time="2021-02-17T19:41:40Z" level=info msg="Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup" backup=velero/fgold-backup-021701 group=certificates.k8s.io/v1 logSource="pkg/backup/item_collector.go:192" resource=certificatesigningrequests time="2021-02-17T19:41:40Z" level=info msg="Getting items for group" backup=velero/fgold-backup-021701 group=certificates.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:76" time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=certificates.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:165" resource=certificatesigningrequests time="2021-02-17T19:41:40Z" level=info msg="Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup" backup=velero/fgold-backup-021701 group=certificates.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:192" resource=certificatesigningrequests time="2021-02-17T19:41:40Z" level=info msg="Getting items for group" backup=velero/fgold-backup-021701 group=networking.k8s.io/v1 logSource="pkg/backup/item_collector.go:76" time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=networking.k8s.io/v1 logSource="pkg/backup/item_collector.go:165" resource=ingressclasses time="2021-02-17T19:41:40Z" level=info msg="Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup" backup=velero/fgold-backup-021701 group=networking.k8s.io/v1 logSource="pkg/backup/item_collector.go:192" resource=ingressclasses time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=networking.k8s.io/v1 logSource="pkg/backup/item_collector.go:165" resource=ingresses time="2021-02-17T19:41:40Z" level=info msg="Listing items" backup=velero/fgold-backup-021701 group=networking.k8s.io/v1 logSource="pkg/backup/item_collector.go:291" namespace=rockbands-v1 resource=ingresses time="2021-02-17T19:41:40Z" level=info msg="Retrieved 0 items" backup=velero/fgold-backup-021701 group=networking.k8s.io/v1 logSource="pkg/backup/item_collector.go:297" namespace=rockbands-v1 resource=ingresses time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=networking.k8s.io/v1 logSource="pkg/backup/item_collector.go:165" resource=networkpolicies time="2021-02-17T19:41:40Z" level=info msg="Listing items" backup=velero/fgold-backup-021701 group=networking.k8s.io/v1 logSource="pkg/backup/item_collector.go:291" namespace=rockbands-v1 resource=networkpolicies time="2021-02-17T19:41:40Z" level=info msg="Retrieved 0 items" backup=velero/fgold-backup-021701 group=networking.k8s.io/v1 logSource="pkg/backup/item_collector.go:297" namespace=rockbands-v1 resource=networkpolicies time="2021-02-17T19:41:40Z" level=info msg="Getting items for group" backup=velero/fgold-backup-021701 group=networking.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:76" time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=networking.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:165" resource=ingressclasses time="2021-02-17T19:41:40Z" level=info msg="Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup" backup=velero/fgold-backup-021701 group=networking.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:192" resource=ingressclasses time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=networking.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:165" resource=ingresses time="2021-02-17T19:41:40Z" level=info msg="Listing items" backup=velero/fgold-backup-021701 group=networking.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:291" namespace=rockbands-v1 resource=ingresses time="2021-02-17T19:41:40Z" level=info msg="Retrieved 0 items" backup=velero/fgold-backup-021701 group=networking.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:297" namespace=rockbands-v1 resource=ingresses time="2021-02-17T19:41:40Z" level=info msg="Getting items for group" backup=velero/fgold-backup-021701 group=policy/v1beta1 logSource="pkg/backup/item_collector.go:76" time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=policy/v1beta1 logSource="pkg/backup/item_collector.go:165" resource=poddisruptionbudgets time="2021-02-17T19:41:40Z" level=info msg="Listing items" backup=velero/fgold-backup-021701 group=policy/v1beta1 logSource="pkg/backup/item_collector.go:291" namespace=rockbands-v1 resource=poddisruptionbudgets time="2021-02-17T19:41:40Z" level=info msg="Retrieved 0 items" backup=velero/fgold-backup-021701 group=policy/v1beta1 logSource="pkg/backup/item_collector.go:297" namespace=rockbands-v1 resource=poddisruptionbudgets time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=policy/v1beta1 logSource="pkg/backup/item_collector.go:165" resource=podsecuritypolicies time="2021-02-17T19:41:40Z" level=info msg="Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup" backup=velero/fgold-backup-021701 group=policy/v1beta1 logSource="pkg/backup/item_collector.go:192" resource=podsecuritypolicies time="2021-02-17T19:41:40Z" level=info msg="Getting items for group" backup=velero/fgold-backup-021701 group=rbac.authorization.k8s.io/v1 logSource="pkg/backup/item_collector.go:76" time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=rbac.authorization.k8s.io/v1 logSource="pkg/backup/item_collector.go:165" resource=clusterrolebindings time="2021-02-17T19:41:40Z" level=info msg="Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup" backup=velero/fgold-backup-021701 group=rbac.authorization.k8s.io/v1 logSource="pkg/backup/item_collector.go:192" resource=clusterrolebindings time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=rbac.authorization.k8s.io/v1 logSource="pkg/backup/item_collector.go:165" resource=clusterroles time="2021-02-17T19:41:40Z" level=info msg="Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup" backup=velero/fgold-backup-021701 group=rbac.authorization.k8s.io/v1 logSource="pkg/backup/item_collector.go:192" resource=clusterroles time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=rbac.authorization.k8s.io/v1 logSource="pkg/backup/item_collector.go:165" resource=rolebindings time="2021-02-17T19:41:40Z" level=info msg="Listing items" backup=velero/fgold-backup-021701 group=rbac.authorization.k8s.io/v1 logSource="pkg/backup/item_collector.go:291" namespace=rockbands-v1 resource=rolebindings time="2021-02-17T19:41:40Z" level=info msg="Retrieved 0 items" backup=velero/fgold-backup-021701 group=rbac.authorization.k8s.io/v1 logSource="pkg/backup/item_collector.go:297" namespace=rockbands-v1 resource=rolebindings time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=rbac.authorization.k8s.io/v1 logSource="pkg/backup/item_collector.go:165" resource=roles time="2021-02-17T19:41:40Z" level=info msg="Listing items" backup=velero/fgold-backup-021701 group=rbac.authorization.k8s.io/v1 logSource="pkg/backup/item_collector.go:291" namespace=rockbands-v1 resource=roles time="2021-02-17T19:41:40Z" level=info msg="Retrieved 0 items" backup=velero/fgold-backup-021701 group=rbac.authorization.k8s.io/v1 logSource="pkg/backup/item_collector.go:297" namespace=rockbands-v1 resource=roles time="2021-02-17T19:41:40Z" level=info msg="Getting items for group" backup=velero/fgold-backup-021701 group=rbac.authorization.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:76" time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=rbac.authorization.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:165" resource=clusterrolebindings time="2021-02-17T19:41:40Z" level=info msg="Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup" backup=velero/fgold-backup-021701 group=rbac.authorization.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:192" resource=clusterrolebindings time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=rbac.authorization.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:165" resource=clusterroles time="2021-02-17T19:41:40Z" level=info msg="Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup" backup=velero/fgold-backup-021701 group=rbac.authorization.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:192" resource=clusterroles time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=rbac.authorization.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:165" resource=rolebindings time="2021-02-17T19:41:40Z" level=info msg="Listing items" backup=velero/fgold-backup-021701 group=rbac.authorization.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:291" namespace=rockbands-v1 resource=rolebindings time="2021-02-17T19:41:40Z" level=info msg="Retrieved 0 items" backup=velero/fgold-backup-021701 group=rbac.authorization.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:297" namespace=rockbands-v1 resource=rolebindings time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=rbac.authorization.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:165" resource=roles time="2021-02-17T19:41:40Z" level=info msg="Listing items" backup=velero/fgold-backup-021701 group=rbac.authorization.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:291" namespace=rockbands-v1 resource=roles time="2021-02-17T19:41:40Z" level=info msg="Retrieved 0 items" backup=velero/fgold-backup-021701 group=rbac.authorization.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:297" namespace=rockbands-v1 resource=roles time="2021-02-17T19:41:40Z" level=info msg="Getting items for group" backup=velero/fgold-backup-021701 group=storage.k8s.io/v1 logSource="pkg/backup/item_collector.go:76" time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=storage.k8s.io/v1 logSource="pkg/backup/item_collector.go:165" resource=csidrivers time="2021-02-17T19:41:40Z" level=info msg="Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup" backup=velero/fgold-backup-021701 group=storage.k8s.io/v1 logSource="pkg/backup/item_collector.go:192" resource=csidrivers time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=storage.k8s.io/v1 logSource="pkg/backup/item_collector.go:165" resource=csinodes time="2021-02-17T19:41:40Z" level=info msg="Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup" backup=velero/fgold-backup-021701 group=storage.k8s.io/v1 logSource="pkg/backup/item_collector.go:192" resource=csinodes time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=storage.k8s.io/v1 logSource="pkg/backup/item_collector.go:165" resource=storageclasses time="2021-02-17T19:41:40Z" level=info msg="Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup" backup=velero/fgold-backup-021701 group=storage.k8s.io/v1 logSource="pkg/backup/item_collector.go:192" resource=storageclasses time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=storage.k8s.io/v1 logSource="pkg/backup/item_collector.go:165" resource=volumeattachments time="2021-02-17T19:41:40Z" level=info msg="Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup" backup=velero/fgold-backup-021701 group=storage.k8s.io/v1 logSource="pkg/backup/item_collector.go:192" resource=volumeattachments time="2021-02-17T19:41:40Z" level=info msg="Getting items for group" backup=velero/fgold-backup-021701 group=storage.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:76" time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=storage.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:165" resource=csidrivers time="2021-02-17T19:41:40Z" level=info msg="Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup" backup=velero/fgold-backup-021701 group=storage.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:192" resource=csidrivers time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=storage.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:165" resource=csinodes time="2021-02-17T19:41:40Z" level=info msg="Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup" backup=velero/fgold-backup-021701 group=storage.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:192" resource=csinodes time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=storage.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:165" resource=storageclasses time="2021-02-17T19:41:40Z" level=info msg="Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup" backup=velero/fgold-backup-021701 group=storage.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:192" resource=storageclasses time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=storage.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:165" resource=volumeattachments time="2021-02-17T19:41:40Z" level=info msg="Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup" backup=velero/fgold-backup-021701 group=storage.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:192" resource=volumeattachments time="2021-02-17T19:41:40Z" level=info msg="Getting items for group" backup=velero/fgold-backup-021701 group=admissionregistration.k8s.io/v1 logSource="pkg/backup/item_collector.go:76" time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=admissionregistration.k8s.io/v1 logSource="pkg/backup/item_collector.go:165" resource=mutatingwebhookconfigurations time="2021-02-17T19:41:40Z" level=info msg="Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup" backup=velero/fgold-backup-021701 group=admissionregistration.k8s.io/v1 logSource="pkg/backup/item_collector.go:192" resource=mutatingwebhookconfigurations time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=admissionregistration.k8s.io/v1 logSource="pkg/backup/item_collector.go:165" resource=validatingwebhookconfigurations time="2021-02-17T19:41:40Z" level=info msg="Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup" backup=velero/fgold-backup-021701 group=admissionregistration.k8s.io/v1 logSource="pkg/backup/item_collector.go:192" resource=validatingwebhookconfigurations time="2021-02-17T19:41:40Z" level=info msg="Getting items for group" backup=velero/fgold-backup-021701 group=admissionregistration.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:76" time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=admissionregistration.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:165" resource=mutatingwebhookconfigurations time="2021-02-17T19:41:40Z" level=info msg="Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup" backup=velero/fgold-backup-021701 group=admissionregistration.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:192" resource=mutatingwebhookconfigurations time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=admissionregistration.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:165" resource=validatingwebhookconfigurations time="2021-02-17T19:41:40Z" level=info msg="Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup" backup=velero/fgold-backup-021701 group=admissionregistration.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:192" resource=validatingwebhookconfigurations time="2021-02-17T19:41:40Z" level=info msg="Getting items for group" backup=velero/fgold-backup-021701 group=apiextensions.k8s.io/v1 logSource="pkg/backup/item_collector.go:76" time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=apiextensions.k8s.io/v1 logSource="pkg/backup/item_collector.go:165" resource=customresourcedefinitions time="2021-02-17T19:41:40Z" level=info msg="Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup" backup=velero/fgold-backup-021701 group=apiextensions.k8s.io/v1 logSource="pkg/backup/item_collector.go:192" resource=customresourcedefinitions time="2021-02-17T19:41:40Z" level=info msg="Getting items for group" backup=velero/fgold-backup-021701 group=apiextensions.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:76" time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=apiextensions.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:165" resource=customresourcedefinitions time="2021-02-17T19:41:40Z" level=info msg="Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup" backup=velero/fgold-backup-021701 group=apiextensions.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:192" resource=customresourcedefinitions time="2021-02-17T19:41:40Z" level=info msg="Getting items for group" backup=velero/fgold-backup-021701 group=scheduling.k8s.io/v1 logSource="pkg/backup/item_collector.go:76" time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=scheduling.k8s.io/v1 logSource="pkg/backup/item_collector.go:165" resource=priorityclasses time="2021-02-17T19:41:40Z" level=info msg="Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup" backup=velero/fgold-backup-021701 group=scheduling.k8s.io/v1 logSource="pkg/backup/item_collector.go:192" resource=priorityclasses time="2021-02-17T19:41:40Z" level=info msg="Getting items for group" backup=velero/fgold-backup-021701 group=scheduling.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:76" time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=scheduling.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:165" resource=priorityclasses time="2021-02-17T19:41:40Z" level=info msg="Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup" backup=velero/fgold-backup-021701 group=scheduling.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:192" resource=priorityclasses time="2021-02-17T19:41:40Z" level=info msg="Getting items for group" backup=velero/fgold-backup-021701 group=coordination.k8s.io/v1 logSource="pkg/backup/item_collector.go:76" time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=coordination.k8s.io/v1 logSource="pkg/backup/item_collector.go:165" resource=leases time="2021-02-17T19:41:40Z" level=info msg="Listing items" backup=velero/fgold-backup-021701 group=coordination.k8s.io/v1 logSource="pkg/backup/item_collector.go:291" namespace=rockbands-v1 resource=leases time="2021-02-17T19:41:40Z" level=info msg="Retrieved 0 items" backup=velero/fgold-backup-021701 group=coordination.k8s.io/v1 logSource="pkg/backup/item_collector.go:297" namespace=rockbands-v1 resource=leases time="2021-02-17T19:41:40Z" level=info msg="Getting items for group" backup=velero/fgold-backup-021701 group=coordination.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:76" time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=coordination.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:165" resource=leases time="2021-02-17T19:41:40Z" level=info msg="Listing items" backup=velero/fgold-backup-021701 group=coordination.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:291" namespace=rockbands-v1 resource=leases time="2021-02-17T19:41:40Z" level=info msg="Retrieved 0 items" backup=velero/fgold-backup-021701 group=coordination.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:297" namespace=rockbands-v1 resource=leases time="2021-02-17T19:41:40Z" level=info msg="Getting items for group" backup=velero/fgold-backup-021701 group=node.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:76" time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=node.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:165" resource=runtimeclasses time="2021-02-17T19:41:40Z" level=info msg="Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup" backup=velero/fgold-backup-021701 group=node.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:192" resource=runtimeclasses time="2021-02-17T19:41:40Z" level=info msg="Getting items for group" backup=velero/fgold-backup-021701 group=discovery.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:76" time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=discovery.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:165" resource=endpointslices time="2021-02-17T19:41:40Z" level=info msg="Listing items" backup=velero/fgold-backup-021701 group=discovery.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:291" namespace=rockbands-v1 resource=endpointslices time="2021-02-17T19:41:40Z" level=info msg="Retrieved 0 items" backup=velero/fgold-backup-021701 group=discovery.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:297" namespace=rockbands-v1 resource=endpointslices time="2021-02-17T19:41:40Z" level=info msg="Getting items for group" backup=velero/fgold-backup-021701 group=velero.io/v1 logSource="pkg/backup/item_collector.go:76" time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=velero.io/v1 logSource="pkg/backup/item_collector.go:165" resource=backupstoragelocations time="2021-02-17T19:41:40Z" level=info msg="Listing items" backup=velero/fgold-backup-021701 group=velero.io/v1 logSource="pkg/backup/item_collector.go:291" namespace=rockbands-v1 resource=backupstoragelocations time="2021-02-17T19:41:40Z" level=info msg="Retrieved 0 items" backup=velero/fgold-backup-021701 group=velero.io/v1 logSource="pkg/backup/item_collector.go:297" namespace=rockbands-v1 resource=backupstoragelocations time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=velero.io/v1 logSource="pkg/backup/item_collector.go:165" resource=podvolumebackups time="2021-02-17T19:41:40Z" level=info msg="Listing items" backup=velero/fgold-backup-021701 group=velero.io/v1 logSource="pkg/backup/item_collector.go:291" namespace=rockbands-v1 resource=podvolumebackups time="2021-02-17T19:41:40Z" level=info msg="Retrieved 0 items" backup=velero/fgold-backup-021701 group=velero.io/v1 logSource="pkg/backup/item_collector.go:297" namespace=rockbands-v1 resource=podvolumebackups time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=velero.io/v1 logSource="pkg/backup/item_collector.go:165" resource=podvolumerestores time="2021-02-17T19:41:40Z" level=info msg="Listing items" backup=velero/fgold-backup-021701 group=velero.io/v1 logSource="pkg/backup/item_collector.go:291" namespace=rockbands-v1 resource=podvolumerestores time="2021-02-17T19:41:40Z" level=info msg="Retrieved 0 items" backup=velero/fgold-backup-021701 group=velero.io/v1 logSource="pkg/backup/item_collector.go:297" namespace=rockbands-v1 resource=podvolumerestores time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=velero.io/v1 logSource="pkg/backup/item_collector.go:165" resource=restores time="2021-02-17T19:41:40Z" level=info msg="Listing items" backup=velero/fgold-backup-021701 group=velero.io/v1 logSource="pkg/backup/item_collector.go:291" namespace=rockbands-v1 resource=restores time="2021-02-17T19:41:40Z" level=info msg="Retrieved 0 items" backup=velero/fgold-backup-021701 group=velero.io/v1 logSource="pkg/backup/item_collector.go:297" namespace=rockbands-v1 resource=restores time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=velero.io/v1 logSource="pkg/backup/item_collector.go:165" resource=schedules time="2021-02-17T19:41:40Z" level=info msg="Listing items" backup=velero/fgold-backup-021701 group=velero.io/v1 logSource="pkg/backup/item_collector.go:291" namespace=rockbands-v1 resource=schedules time="2021-02-17T19:41:40Z" level=info msg="Retrieved 0 items" backup=velero/fgold-backup-021701 group=velero.io/v1 logSource="pkg/backup/item_collector.go:297" namespace=rockbands-v1 resource=schedules time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=velero.io/v1 logSource="pkg/backup/item_collector.go:165" resource=serverstatusrequests time="2021-02-17T19:41:40Z" level=info msg="Listing items" backup=velero/fgold-backup-021701 group=velero.io/v1 logSource="pkg/backup/item_collector.go:291" namespace=rockbands-v1 resource=serverstatusrequests time="2021-02-17T19:41:40Z" level=info msg="Retrieved 0 items" backup=velero/fgold-backup-021701 group=velero.io/v1 logSource="pkg/backup/item_collector.go:297" namespace=rockbands-v1 resource=serverstatusrequests time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=velero.io/v1 logSource="pkg/backup/item_collector.go:165" resource=backups time="2021-02-17T19:41:40Z" level=info msg="Listing items" backup=velero/fgold-backup-021701 group=velero.io/v1 logSource="pkg/backup/item_collector.go:291" namespace=rockbands-v1 resource=backups time="2021-02-17T19:41:40Z" level=info msg="Retrieved 0 items" backup=velero/fgold-backup-021701 group=velero.io/v1 logSource="pkg/backup/item_collector.go:297" namespace=rockbands-v1 resource=backups time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=velero.io/v1 logSource="pkg/backup/item_collector.go:165" resource=deletebackuprequests time="2021-02-17T19:41:40Z" level=info msg="Listing items" backup=velero/fgold-backup-021701 group=velero.io/v1 logSource="pkg/backup/item_collector.go:291" namespace=rockbands-v1 resource=deletebackuprequests time="2021-02-17T19:41:40Z" level=info msg="Retrieved 0 items" backup=velero/fgold-backup-021701 group=velero.io/v1 logSource="pkg/backup/item_collector.go:297" namespace=rockbands-v1 resource=deletebackuprequests time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=velero.io/v1 logSource="pkg/backup/item_collector.go:165" resource=downloadrequests time="2021-02-17T19:41:40Z" level=info msg="Listing items" backup=velero/fgold-backup-021701 group=velero.io/v1 logSource="pkg/backup/item_collector.go:291" namespace=rockbands-v1 resource=downloadrequests time="2021-02-17T19:41:40Z" level=info msg="Retrieved 0 items" backup=velero/fgold-backup-021701 group=velero.io/v1 logSource="pkg/backup/item_collector.go:297" namespace=rockbands-v1 resource=downloadrequests time="2021-02-17T19:41:40Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=velero.io/v1 logSource="pkg/backup/item_collector.go:165" resource=resticrepositories time="2021-02-17T19:41:40Z" level=info msg="Listing items" backup=velero/fgold-backup-021701 group=velero.io/v1 logSource="pkg/backup/item_collector.go:291" namespace=rockbands-v1 resource=resticrepositories time="2021-02-17T19:41:41Z" level=info msg="Retrieved 0 items" backup=velero/fgold-backup-021701 group=velero.io/v1 logSource="pkg/backup/item_collector.go:297" namespace=rockbands-v1 resource=resticrepositories time="2021-02-17T19:41:41Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=velero.io/v1 logSource="pkg/backup/item_collector.go:165" resource=volumesnapshotlocations time="2021-02-17T19:41:41Z" level=info msg="Listing items" backup=velero/fgold-backup-021701 group=velero.io/v1 logSource="pkg/backup/item_collector.go:291" namespace=rockbands-v1 resource=volumesnapshotlocations time="2021-02-17T19:41:41Z" level=info msg="Retrieved 0 items" backup=velero/fgold-backup-021701 group=velero.io/v1 logSource="pkg/backup/item_collector.go:297" namespace=rockbands-v1 resource=volumesnapshotlocations time="2021-02-17T19:41:41Z" level=info msg="Getting items for group" backup=velero/fgold-backup-021701 group=extensions/v1beta1 logSource="pkg/backup/item_collector.go:76" time="2021-02-17T19:41:41Z" level=info msg="Getting items for resource" backup=velero/fgold-backup-021701 group=extensions/v1beta1 logSource="pkg/backup/item_collector.go:165" resource=ingresses time="2021-02-17T19:41:41Z" level=info msg="Listing items" backup=velero/fgold-backup-021701 group=extensions/v1beta1 logSource="pkg/backup/item_collector.go:291" namespace=rockbands-v1 resource=ingresses time="2021-02-17T19:41:41Z" level=info msg="Retrieved 0 items" backup=velero/fgold-backup-021701 group=extensions/v1beta1 logSource="pkg/backup/item_collector.go:297" namespace=rockbands-v1 resource=ingresses time="2021-02-17T19:41:41Z" level=info msg="Collected 3 items matching the backup spec from the Kubernetes API (actual number of items backed up may be more or less depending on velero.io/exclude-from-backup annotation, plugins returning additional related items to back up, etc.)" backup=velero/fgold-backup-021701 logSource="pkg/backup/backup.go:278" progress= time="2021-02-17T19:41:41Z" level=info msg="Processing item" backup=velero/fgold-backup-021701 logSource="pkg/backup/backup.go:354" name=rockbands-v1 namespace= progress= resource=namespaces time="2021-02-17T19:41:41Z" level=info msg="Backing up item" backup=velero/fgold-backup-021701 logSource="pkg/backup/item_backupper.go:121" name=rockbands-v1 namespace= resource=namespaces time="2021-02-17T19:41:41Z" level=info msg="Backed up 1 items out of an estimated total of 3 (estimate will change throughout the backup)" backup=velero/fgold-backup-021701 logSource="pkg/backup/backup.go:394" name=rockbands-v1 namespace= progress= resource=namespaces time="2021-02-17T19:41:41Z" level=info msg="Processing item" backup=velero/fgold-backup-021701 logSource="pkg/backup/backup.go:354" name=default-token-c84rp namespace=rockbands-v1 progress= resource=secrets time="2021-02-17T19:41:41Z" level=info msg="Backing up item" backup=velero/fgold-backup-021701 logSource="pkg/backup/item_backupper.go:121" name=default-token-c84rp namespace=rockbands-v1 resource=secrets time="2021-02-17T19:41:41Z" level=info msg="Backed up 2 items out of an estimated total of 3 (estimate will change throughout the backup)" backup=velero/fgold-backup-021701 logSource="pkg/backup/backup.go:394" name=default-token-c84rp namespace=rockbands-v1 progress= resource=secrets time="2021-02-17T19:41:41Z" level=info msg="Processing item" backup=velero/fgold-backup-021701 logSource="pkg/backup/backup.go:354" name=default namespace=rockbands-v1 progress= resource=serviceaccounts time="2021-02-17T19:41:41Z" level=info msg="Backing up item" backup=velero/fgold-backup-021701 logSource="pkg/backup/item_backupper.go:121" name=default namespace=rockbands-v1 resource=serviceaccounts time="2021-02-17T19:41:41Z" level=info msg="Executing custom action" backup=velero/fgold-backup-021701 logSource="pkg/backup/item_backupper.go:327" name=default namespace=rockbands-v1 resource=serviceaccounts time="2021-02-17T19:41:41Z" level=info msg="Running ServiceAccountAction" backup=velero/fgold-backup-021701 cmd=/velero logSource="pkg/backup/service_account_action.go:77" pluginName=velero time="2021-02-17T19:41:41Z" level=info msg="Done running ServiceAccountAction" backup=velero/fgold-backup-021701 cmd=/velero logSource="pkg/backup/service_account_action.go:120" pluginName=velero time="2021-02-17T19:41:41Z" level=info msg="Backed up 3 items out of an estimated total of 3 (estimate will change throughout the backup)" backup=velero/fgold-backup-021701 logSource="pkg/backup/backup.go:394" name=default namespace=rockbands-v1 progress= resource=serviceaccounts time="2021-02-17T19:41:41Z" level=info msg="Backed up a total of 3 items" backup=velero/fgold-backup-021701 logSource="pkg/backup/backup.go:419" progress=

- `velero restore describe <restorename>` or `kubectl get restore/<restorename> -n velero -o yaml`
n/a
- `velero restore logs <restorename>`
n/a

**Anything else you would like to add:**
The cause might be that it can take up to five minutes for Velero to call discovery and realize that the new CRDs are there.

See [line 381 in server.go](https://github.com/vmware-tanzu/velero/blob/6bdd4ac1922ebee440257ae5ab21e7b7ce12e60b/pkg/cmd/server/server.go#L381) where discovery is set to look for new objects every five minutes.

**Environment:**

- Velero version (use `velero version`): `v1.5.3`
- Velero features (use `velero client config get features`): `features: <NOT SET>`
- Kubernetes version (use `kubectl version`):

Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"06ad960bfd03b39c8310aaf92d1e7c12ce618213", GitTreeState:"clean", BuildDate:"2020-02-13T18:08:14Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.1", GitCommit:"206bcadf021e76c27513500ca24182692aabd17e", GitTreeState:"clean", BuildDate:"2020-09-14T07:30:52Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"} 


- Kubernetes installer & version: `kind v0.9.0 go1.15.2 darwin/amd64`
- Cloud provider or hardware configuration: `using local instance of Minio`
- OS (e.g. from `/etc/os-release`): `MacOS 10.15.7`

**Vote on this issue!**

This is an invitation to the Velero community to vote on issues, you can see the project's [top voted issues listed here](https://github.com/vmware-tanzu/velero/issues?q=is%3Aissue+is%3Aopen+sort%3Areactions-%2B1-desc).  
Use the "reaction smiley face" up to the right of this comment to vote.

- :+1: for "I would like to see this bug fixed as soon as possible"
- :-1: for "There are more important bugs to focus on right now"
dsu-igeek commented 3 years ago

When this is done test/e2e/enable_api_group_versions_test.go should be fixed to install Velero BEFORE installing the CRDs. Currently it installs the CRDs first to get around this issue.