search

vmware-tanzu / velero

Backup and migrate Kubernetes applications and their persistent volumes
https://velero.io
Apache License 2.0
8.6k stars 1.39k forks source link

restic: unable to restore "Local" Persistent Volumes since they're statically provisioned #2520

Open edmund-nelson opened 4 years ago

edmund-nelson commented 4 years ago

What steps did you take and what happened:

I backed up a namespace with pods and PVC's which included Persistent Volumes with a "local" spec (which the documentation suggests is supported - https://github.com/vmware-tanzu/velero/blob/master/site/docs/master/restic.md #), and associated pods annotated with the volume names.

The backup appears to work - and as far as I can tell, Restic backs up the Persistent Volume.

However when attempting to restore the namespace the PV with "local" storage never seems to get restored. Sometimes Restic hangs for an hour before timing out waiting for the restore to complete ie:-

level=info msg="Waiting for all restic restores to complete" logSource="pkg/restore/restore.go:545" restore=velero/nginx-example3-20200507164146
level=error msg="unable to successfully complete restic restores of pod's volumes" error="timed out waiting for all PodVolumeRestores to complete"

Other times the restore completes without error - but no PV is brought down (though the namespace, pods and PVC's are restored - though in a forever "pending" state due to the missing PV)

In the "successful" restores - despite the PV being deleted from the cluster (along with namespace, pod deployments etc.) The logs seem to imply that restic/velero think the PV has already been restored and is thus skipped

time="2020-05-07T16:32:50Z" level=info msg="Skipping persistentvolumes/nginx-logs-volume because it's already been restored." logSource="pkg/restore/restore.go:910" restore=velero/nginx-example2-20200507163249
time="2020-05-07T16:32:50Z" level=info msg="Executing item action for persistentvolumeclaims" logSource="pkg/restore/restore.go:1030" restore=velero/nginx-example2-20200507163249
time="2020-05-07T16:32:50Z" level=info msg="Executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:63" pluginName=velero restore=velero/nginx-example2-20200507163249
time="2020-05-07T16:32:50Z" level=info msg="Done executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:74" pluginName=velero restore=velero/nginx-example2-20200507163249
time="2020-05-07T16:32:50Z" level=info msg="Resetting PersistentVolumeClaim nginx-example/nginx-logs for dynamic provisioning because its PV nginx-logs-volume has a reclaim policy of Delete" logSource="pkg/restore/restore.go:1103" restore=velero/nginx-example2-20200507163249

I have also tried to just restore the PV's on its own (with no preexisting pods or pod deployments) and this hasn't worked, both with a reclaim policy of Delete and Retain.

The reason I'm trying to use "local" storage is that this is a environment that is on prem and not in the cloud - the existing PV's use "hostpath" which restic doesnt support - so attempting to prove backups and restores using "local" - with a view of eventually changing the existing PV's to "local" from "hostpath" if this works.

For context - I also backed up a namespace which had a PV under it using "rook-ceph-block" storageclass - deleted all resources - and restored - and it worked fine, bringing the PV back along with associated data - so I can only surmise that restic isnt liking my PV's with "local" persistent volumes, or I am doing something wrong.

What did you expect to happen:

The Persistent volume(s) to restore along with data - and pods/PVC's to come up healthily.

Below is a quick demo of what I did.


velero install \
> --provider aws \
> --use-restic \
> --bucket cva-builder-velero-backups-test \
> --secret-file ./credentials-velero \
> --backup-location-config region=us-east-1 \
> --plugins velero/velero-plugin-for-aws:v1.0.1 \
> --snapshot-location-config region=us-east-1 \
> --use-volume-snapshots=true

kubectl get pods -n velero
NAME                     READY   STATUS    RESTARTS   AGE
restic-469kh             1/1     Running   0          18s
restic-bclvc             1/1     Running   0          18s
velero-84b675ddc-p6ft7   1/1     Running   0          18s
kubectl apply -f nginx_example.yml
namespace/nginx-example created
persistentvolume/nginx-logs-volume created
persistentvolumeclaim/nginx-logs created
deployment.apps/nginx-deployment created
service/my-nginx created
kubectl get pv | grep nginx
nginx-logs-volume                          10Gi       RWO            Delete           Bound    nginx-example/nginx-logs             local-storage                 4m40s
[centos@cva-builder:4.11.4-PR-1712-29 SANDBOX2 CVA-PR-1712 tmp]$ kubectl get pvc -n nginx-example
NAME         STATUS   VOLUME              CAPACITY   ACCESS MODES   STORAGECLASS    AGE
nginx-logs   Bound    nginx-logs-volume   10Gi       RWO            local-storage   4m53s
[centos@cva-builder:4.11.4-PR-1712-29 SANDBOX2 CVA-PR-1712 tmp]$ kubectl get pod -n nginx-example
NAME                                READY   STATUS    RESTARTS   AGE
nginx-deployment-6fbd5f64c4-fps5c   1/1     Running   0          108s
kubectl get pv nginx-logs-volume -o yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"PersistentVolume","metadata":{"annotations":{},"labels":{"app":"nginx","type":"local"},"name":"nginx-logs-volume"},"spec":{"accessModes":["ReadWriteOnce"],"capacity":{"storage":"10Gi"},"local":{"path":"/mnt/data"},"nodeAffinity":{"required":{"nodeSelectorTerms":[{"matchExpressions":[{"key":"workerNodeName","operator":"In","values":["node-1"]}]}]}},"persistentVolumeReclaimPolicy":"Delete","storageClassName":"local-storage","volumeMode":"Filesystem"}}
    pv.kubernetes.io/bound-by-controller: "yes"
  creationTimestamp: "2020-05-08T08:56:36Z"
  finalizers:
  - kubernetes.io/pv-protection
  labels:
    app: nginx
    type: local
  name: nginx-logs-volume
  resourceVersion: "298587"
  selfLink: /api/v1/persistentvolumes/nginx-logs-volume
  uid: d2b6957c-9109-11ea-a626-02021a19689b
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 10Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: nginx-logs
    namespace: nginx-example
    resourceVersion: "298572"
    uid: d2bc87d4-9109-11ea-a626-02021a19689b
  local:
    path: /mnt/data
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: workerNodeName
          operator: In
          values:
          - node-1
  persistentVolumeReclaimPolicy: Delete
  storageClassName: local-storage
  volumeMode: Filesystem
status:
  phase: Bound
kubectl get pod nginx-deployment-6fbd5f64c4-fps5c -n nginx-example -o yaml
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: "2020-05-08T08:59:50Z"
  generateName: nginx-deployment-6fbd5f64c4-
  labels:
    app: nginx
    pod-template-hash: 6fbd5f64c4
  name: nginx-deployment-6fbd5f64c4-fps5c
  namespace: nginx-example
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: nginx-deployment-6fbd5f64c4
    uid: d2c226da-9109-11ea-ac8a-028358a66085
  resourceVersion: "299852"
  selfLink: /api/v1/namespaces/nginx-example/pods/nginx-deployment-6fbd5f64c4-fps5c
  uid: 4640dfa5-910a-11ea-ac8a-028358a66085
spec:
  containers:
  - image: nginx:1.7.9
    imagePullPolicy: IfNotPresent
    name: nginx
    ports:
    - containerPort: 80
      protocol: TCP
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/log/nginx
      name: nginx-logs
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-zmrr5
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  nodeName: ip-10-165-138-230.ec2.internal
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: nginx-logs
    persistentVolumeClaim:
      claimName: nginx-logs
  - name: default-token-zmrr5
    secret:
      defaultMode: 420
      secretName: default-token-zmrr5
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2020-05-08T08:59:50Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2020-05-08T09:00:54Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2020-05-08T09:00:54Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2020-05-08T08:59:50Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: docker://735b18c8379ee6b452e95c6c2ab45d1c82a9086598300de7fd90374336bac13f
    image: nginx:1.7.9
    imageID: docker-pullable://nginx@sha256:e3456c851a152494c3e4ff5fcc26f240206abac0c9d794affb40e0714846c451
    lastState: {}
    name: nginx
    ready: true
    restartCount: 0
    state:
      running:
        startedAt: "2020-05-08T09:00:54Z"
  hostIP: 10.165.138.230
  phase: Running
  podIP: 130.174.242.199
  qosClass: BestEffort
  startTime: "2020-05-08T08:59:50Z"
kubectl get pvc -n nginx-example -o yaml
apiVersion: v1
items:
- apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"labels":{"app":"nginx"},"name":"nginx-logs","namespace":"nginx-example"},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"50Mi"}},"storageClassName":"local-storage"}}
      pv.kubernetes.io/bind-completed: "yes"
      pv.kubernetes.io/bound-by-controller: "yes"
    creationTimestamp: "2020-05-08T08:56:36Z"
    finalizers:
    - kubernetes.io/pvc-protection
    labels:
      app: nginx
    name: nginx-logs
    namespace: nginx-example
    resourceVersion: "298592"
    selfLink: /api/v1/namespaces/nginx-example/persistentvolumeclaims/nginx-logs
    uid: d2bc87d4-9109-11ea-a626-02021a19689b
  spec:
    accessModes:
    - ReadWriteOnce
    dataSource: null
    resources:
      requests:
        storage: 50Mi
    storageClassName: local-storage
    volumeMode: Filesystem
    volumeName: nginx-logs-volume
  status:
    accessModes:
    - ReadWriteOnce
    capacity:
      storage: 10Gi
    phase: Bound
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""
kubectl -n nginx-example annotate pod/nginx-deployment-6fbd5f64c4-fps5c backup.velero.io/backup-volumes=nginx-logs
pod/nginx-deployment-6fbd5f64c4-fps5c annotated
velero create backup nginx-example-backup-new --include-namespaces nginx-example --snapshot-volumes
Backup request "nginx-example-backup-new" submitted successfully.
Run `velero backup describe nginx-example-backup-new` or `velero backup logs nginx-example-backup-new` for more details.
velero backup describe nginx-example-backup-new --details
Name:         nginx-example-backup-new
Namespace:    velero
Labels:       velero.io/storage-location=default
Annotations:  <none>

Phase:  Completed

Namespaces:
  Included:  nginx-example
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  auto

Label selector:  <none>

Storage Location:  default

Snapshot PVs:  true

TTL:  720h0m0s

Hooks:  <none>

Backup Format Version:  1

Started:    2020-05-08 09:33:25 +0000 UTC
Completed:  2020-05-08 09:33:44 +0000 UTC

Expiration:  2020-06-07 09:33:25 +0000 UTC

Resource List:
  apps/v1/Deployment:
    - nginx-example/nginx-deployment
  apps/v1/ReplicaSet:
    - nginx-example/nginx-deployment-6fbd5f64c4
  v1/Endpoints:
    - nginx-example/my-nginx
  v1/Event:
    - nginx-example/nginx-deployment-6fbd5f64c4-2r7zk.160d01c6f35fac21
    - nginx-example/nginx-deployment-6fbd5f64c4-2r7zk.160d01c6ffb7dd12
    - nginx-example/nginx-deployment-6fbd5f64c4-2r7zk.160d01e397c7f57f
    - nginx-example/nginx-deployment-6fbd5f64c4-2r7zk.160d020352295182
    - nginx-example/nginx-deployment-6fbd5f64c4-fps5c.160d01f3d4be1543
    - nginx-example/nginx-deployment-6fbd5f64c4-fps5c.160d02014a99421f
    - nginx-example/nginx-deployment-6fbd5f64c4-fps5c.160d020281b1aab3
    - nginx-example/nginx-deployment-6fbd5f64c4-fps5c.160d0202a4ed5f11
    - nginx-example/nginx-deployment-6fbd5f64c4-fps5c.160d0202ae7423c0
    - nginx-example/nginx-deployment-6fbd5f64c4.160d01c6b69e8011
    - nginx-example/nginx-deployment-6fbd5f64c4.160d01f3d43d2119
    - nginx-example/nginx-deployment.160d01c6b5b35eb3
    - nginx-example/nginx-logs.160d01c6b399f74e
  v1/Namespace:
    - nginx-example
  v1/PersistentVolume:
    - nginx-logs-volume
  v1/PersistentVolumeClaim:
    - nginx-example/nginx-logs
  v1/Pod:
    - nginx-example/nginx-deployment-6fbd5f64c4-fps5c
  v1/Secret:
    - nginx-example/default-token-zmrr5
  v1/Service:
    - nginx-example/my-nginx
  v1/ServiceAccount:
    - nginx-example/default

Persistent Volumes: <none included>

Restic Backups:
  Completed:
    nginx-example/nginx-deployment-6fbd5f64c4-fps5c: nginx-logs
velero backup logs nginx-example-backup-new
time="2020-05-08T09:33:25Z" level=info msg="Setting up backup temp file" backup=velero/nginx-example-backup-new logSource="pkg/controller/backup_controller.go:494"
time="2020-05-08T09:33:25Z" level=info msg="Setting up plugin manager" backup=velero/nginx-example-backup-new logSource="pkg/controller/backup_controller.go:501"
time="2020-05-08T09:33:25Z" level=info msg="Getting backup item actions" backup=velero/nginx-example-backup-new logSource="pkg/controller/backup_controller.go:505"
time="2020-05-08T09:33:25Z" level=info msg="Setting up backup store" backup=velero/nginx-example-backup-new logSource="pkg/controller/backup_controller.go:511"
time="2020-05-08T09:33:25Z" level=info msg="Writing backup version file" backup=velero/nginx-example-backup-new logSource="pkg/backup/backup.go:213"
time="2020-05-08T09:33:25Z" level=info msg="Including namespaces: nginx-example" backup=velero/nginx-example-backup-new logSource="pkg/backup/backup.go:219"
time="2020-05-08T09:33:25Z" level=info msg="Excluding namespaces: <none>" backup=velero/nginx-example-backup-new logSource="pkg/backup/backup.go:220"
time="2020-05-08T09:33:25Z" level=info msg="Including resources: *" backup=velero/nginx-example-backup-new logSource="pkg/backup/backup.go:223"
time="2020-05-08T09:33:25Z" level=info msg="Excluding resources: <none>" backup=velero/nginx-example-backup-new logSource="pkg/backup/backup.go:224"
time="2020-05-08T09:33:40Z" level=info msg="Backing up group" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/group_backupper.go:101"
time="2020-05-08T09:33:40Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/resource_backupper.go:105" resource=pods
time="2020-05-08T09:33:40Z" level=info msg="Listing items" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/resource_backupper.go:226" namespace=nginx-example resource=pods
time="2020-05-08T09:33:40Z" level=info msg="Retrieved 1 items" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/resource_backupper.go:240" namespace=nginx-example resource=pods
time="2020-05-08T09:33:40Z" level=info msg="Backing up item" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/item_backupper.go:169" name=nginx-deployment-6fbd5f64c4-fps5c namespace=nginx-example resource=pods
time="2020-05-08T09:33:40Z" level=info msg="Executing custom action" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/item_backupper.go:330" name=nginx-deployment-6fbd5f64c4-fps5c namespace=nginx-example resource=pods
time="2020-05-08T09:33:40Z" level=info msg="Executing podAction" backup=velero/nginx-example-backup-new cmd=/velero logSource="pkg/backup/pod_action.go:51" pluginName=velero
time="2020-05-08T09:33:40Z" level=info msg="Adding pvc nginx-logs to additionalItems" backup=velero/nginx-example-backup-new cmd=/velero logSource="pkg/backup/pod_action.go:67" pluginName=velero
time="2020-05-08T09:33:40Z" level=info msg="Done executing podAction" backup=velero/nginx-example-backup-new cmd=/velero logSource="pkg/backup/pod_action.go:77" pluginName=velero
time="2020-05-08T09:33:40Z" level=info msg="Backing up item" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/item_backupper.go:169" name=nginx-logs namespace=nginx-example resource=persistentvolumeclaims
time="2020-05-08T09:33:40Z" level=info msg="Executing custom action" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/item_backupper.go:330" name=nginx-logs namespace=nginx-example resource=persistentvolumeclaims
time="2020-05-08T09:33:40Z" level=info msg="Executing PVCAction" backup=velero/nginx-example-backup-new cmd=/velero logSource="pkg/backup/backup_pv_action.go:49" pluginName=velero
time="2020-05-08T09:33:40Z" level=info msg="Backing up item" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/item_backupper.go:169" name=nginx-logs-volume namespace= resource=persistentvolumes
time="2020-05-08T09:33:40Z" level=info msg="Executing takePVSnapshot" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/item_backupper.go:400" name=nginx-logs-volume namespace= resource=persistentvolumes
time="2020-05-08T09:33:40Z" level=info msg="Skipping snapshot of persistent volume because volume is being backed up with restic." backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/item_backupper.go:418" name=nginx-logs-volume namespace= persistentVolume=nginx-logs-volume resource=persistentvolumes
time="2020-05-08T09:33:42Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/resource_backupper.go:105" resource=persistentvolumeclaims
time="2020-05-08T09:33:42Z" level=info msg="Listing items" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/resource_backupper.go:226" namespace=nginx-example resource=persistentvolumeclaims
time="2020-05-08T09:33:42Z" level=info msg="Retrieved 1 items" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/resource_backupper.go:240" namespace=nginx-example resource=persistentvolumeclaims
time="2020-05-08T09:33:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/item_backupper.go:163" name=nginx-logs namespace=nginx-example resource=persistentvolumeclaims
time="2020-05-08T09:33:42Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/resource_backupper.go:105" resource=persistentvolumes
time="2020-05-08T09:33:42Z" level=info msg="Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/resource_backupper.go:127" resource=persistentvolumes
time="2020-05-08T09:33:42Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/resource_backupper.go:105" resource=events
time="2020-05-08T09:33:42Z" level=info msg="Listing items" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/resource_backupper.go:226" namespace=nginx-example resource=events
time="2020-05-08T09:33:42Z" level=info msg="Retrieved 13 items" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/resource_backupper.go:240" namespace=nginx-example resource=events
time="2020-05-08T09:33:42Z" level=info msg="Backing up item" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/item_backupper.go:169" name=nginx-deployment-6fbd5f64c4-2r7zk.160d01c6f35fac21 namespace=nginx-example resource=events
time="2020-05-08T09:33:42Z" level=info msg="Backing up item" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/item_backupper.go:169" name=nginx-deployment-6fbd5f64c4-2r7zk.160d01c6ffb7dd12 namespace=nginx-example resource=events
time="2020-05-08T09:33:42Z" level=info msg="Backing up item" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/item_backupper.go:169" name=nginx-deployment-6fbd5f64c4-2r7zk.160d01e397c7f57f namespace=nginx-example resource=events
time="2020-05-08T09:33:42Z" level=info msg="Backing up item" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/item_backupper.go:169" name=nginx-deployment-6fbd5f64c4-2r7zk.160d020352295182 namespace=nginx-example resource=events
time="2020-05-08T09:33:42Z" level=info msg="Backing up item" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/item_backupper.go:169" name=nginx-deployment-6fbd5f64c4-fps5c.160d01f3d4be1543 namespace=nginx-example resource=events
time="2020-05-08T09:33:42Z" level=info msg="Backing up item" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/item_backupper.go:169" name=nginx-deployment-6fbd5f64c4-fps5c.160d02014a99421f namespace=nginx-example resource=events
time="2020-05-08T09:33:42Z" level=info msg="Backing up item" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/item_backupper.go:169" name=nginx-deployment-6fbd5f64c4-fps5c.160d020281b1aab3 namespace=nginx-example resource=events
time="2020-05-08T09:33:42Z" level=info msg="Backing up item" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/item_backupper.go:169" name=nginx-deployment-6fbd5f64c4-fps5c.160d0202a4ed5f11 namespace=nginx-example resource=events
time="2020-05-08T09:33:42Z" level=info msg="Backing up item" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/item_backupper.go:169" name=nginx-deployment-6fbd5f64c4-fps5c.160d0202ae7423c0 namespace=nginx-example resource=events
time="2020-05-08T09:33:42Z" level=info msg="Backing up item" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/item_backupper.go:169" name=nginx-deployment-6fbd5f64c4.160d01c6b69e8011 namespace=nginx-example resource=events
time="2020-05-08T09:33:42Z" level=info msg="Backing up item" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/item_backupper.go:169" name=nginx-deployment-6fbd5f64c4.160d01f3d43d2119 namespace=nginx-example resource=events
time="2020-05-08T09:33:42Z" level=info msg="Backing up item" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/item_backupper.go:169" name=nginx-deployment.160d01c6b5b35eb3 namespace=nginx-example resource=events
time="2020-05-08T09:33:42Z" level=info msg="Backing up item" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/item_backupper.go:169" name=nginx-logs.160d01c6b399f74e namespace=nginx-example resource=events
time="2020-05-08T09:33:42Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/resource_backupper.go:105" resource=limitranges
time="2020-05-08T09:33:42Z" level=info msg="Listing items" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/resource_backupper.go:226" namespace=nginx-example resource=limitranges
time="2020-05-08T09:33:42Z" level=info msg="Retrieved 0 items" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/resource_backupper.go:240" namespace=nginx-example resource=limitranges
time="2020-05-08T09:33:42Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/resource_backupper.go:105" resource=services
time="2020-05-08T09:33:42Z" level=info msg="Listing items" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/resource_backupper.go:226" namespace=nginx-example resource=services
time="2020-05-08T09:33:42Z" level=info msg="Retrieved 1 items" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/resource_backupper.go:240" namespace=nginx-example resource=services
time="2020-05-08T09:33:42Z" level=info msg="Backing up item" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/item_backupper.go:169" name=my-nginx namespace=nginx-example resource=services
time="2020-05-08T09:33:42Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/resource_backupper.go:105" resource=configmaps
time="2020-05-08T09:33:42Z" level=info msg="Listing items" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/resource_backupper.go:226" namespace=nginx-example resource=configmaps
time="2020-05-08T09:33:42Z" level=info msg="Retrieved 0 items" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/resource_backupper.go:240" namespace=nginx-example resource=configmaps
time="2020-05-08T09:33:42Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/resource_backupper.go:105" resource=resourcequotas
time="2020-05-08T09:33:42Z" level=info msg="Listing items" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/resource_backupper.go:226" namespace=nginx-example resource=resourcequotas
time="2020-05-08T09:33:42Z" level=info msg="Retrieved 0 items" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/resource_backupper.go:240" namespace=nginx-example resource=resourcequotas
time="2020-05-08T09:33:42Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/resource_backupper.go:105" resource=serviceaccounts
time="2020-05-08T09:33:42Z" level=info msg="Listing items" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/resource_backupper.go:226" namespace=nginx-example resource=serviceaccounts
time="2020-05-08T09:33:42Z" level=info msg="Retrieved 1 items" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/resource_backupper.go:240" namespace=nginx-example resource=serviceaccounts
time="2020-05-08T09:33:42Z" level=info msg="Backing up item" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/item_backupper.go:169" name=default namespace=nginx-example resource=serviceaccounts
time="2020-05-08T09:33:42Z" level=info msg="Executing custom action" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/item_backupper.go:330" name=default namespace=nginx-example resource=serviceaccounts
time="2020-05-08T09:33:42Z" level=info msg="Running ServiceAccountAction" backup=velero/nginx-example-backup-new cmd=/velero logSource="pkg/backup/service_account_action.go:77" pluginName=velero
time="2020-05-08T09:33:42Z" level=info msg="Done running ServiceAccountAction" backup=velero/nginx-example-backup-new cmd=/velero logSource="pkg/backup/service_account_action.go:120" pluginName=velero
time="2020-05-08T09:33:42Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/resource_backupper.go:105" resource=nodes
time="2020-05-08T09:33:42Z" level=info msg="Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/resource_backupper.go:127" resource=nodes
time="2020-05-08T09:33:42Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/resource_backupper.go:105" resource=secrets
time="2020-05-08T09:33:42Z" level=info msg="Listing items" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/resource_backupper.go:226" namespace=nginx-example resource=secrets
time="2020-05-08T09:33:42Z" level=info msg="Retrieved 1 items" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/resource_backupper.go:240" namespace=nginx-example resource=secrets
time="2020-05-08T09:33:42Z" level=info msg="Backing up item" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/item_backupper.go:169" name=default-token-zmrr5 namespace=nginx-example resource=secrets
time="2020-05-08T09:33:42Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/resource_backupper.go:105" resource=namespaces
time="2020-05-08T09:33:42Z" level=info msg="Getting namespace" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/resource_backupper.go:184" namespace=nginx-example resource=namespaces
time="2020-05-08T09:33:42Z" level=info msg="Backing up item" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/item_backupper.go:169" name=nginx-example namespace= resource=namespaces
time="2020-05-08T09:33:42Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/resource_backupper.go:105" resource=endpoints
time="2020-05-08T09:33:42Z" level=info msg="Listing items" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/resource_backupper.go:226" namespace=nginx-example resource=endpoints
time="2020-05-08T09:33:42Z" level=info msg="Retrieved 1 items" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/resource_backupper.go:240" namespace=nginx-example resource=endpoints
time="2020-05-08T09:33:42Z" level=info msg="Backing up item" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/item_backupper.go:169" name=my-nginx namespace=nginx-example resource=endpoints
time="2020-05-08T09:33:42Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/resource_backupper.go:105" resource=podtemplates
time="2020-05-08T09:33:42Z" level=info msg="Listing items" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/resource_backupper.go:226" namespace=nginx-example resource=podtemplates
time="2020-05-08T09:33:42Z" level=info msg="Retrieved 0 items" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/resource_backupper.go:240" namespace=nginx-example resource=podtemplates
time="2020-05-08T09:33:42Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/resource_backupper.go:105" resource=replicationcontrollers
time="2020-05-08T09:33:42Z" level=info msg="Listing items" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/resource_backupper.go:226" namespace=nginx-example resource=replicationcontrollers
time="2020-05-08T09:33:42Z" level=info msg="Retrieved 0 items" backup=velero/nginx-example-backup-new group=v1 logSource="pkg/backup/resource_backupper.go:240" namespace=nginx-example resource=replicationcontrollers
time="2020-05-08T09:33:42Z" level=info msg="Backing up group" backup=velero/nginx-example-backup-new group=apiregistration.k8s.io/v1 logSource="pkg/backup/group_backupper.go:101"
time="2020-05-08T09:33:42Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=apiregistration.k8s.io/v1 logSource="pkg/backup/resource_backupper.go:105" resource=apiservices
time="2020-05-08T09:33:42Z" level=info msg="Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup" backup=velero/nginx-example-backup-new group=apiregistration.k8s.io/v1 logSource="pkg/backup/resource_backupper.go:127" resource=apiservices
time="2020-05-08T09:33:42Z" level=info msg="Backing up group" backup=velero/nginx-example-backup-new group=apps/v1 logSource="pkg/backup/group_backupper.go:101"
time="2020-05-08T09:33:42Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=apps/v1 logSource="pkg/backup/resource_backupper.go:105" resource=controllerrevisions
time="2020-05-08T09:33:42Z" level=info msg="Listing items" backup=velero/nginx-example-backup-new group=apps/v1 logSource="pkg/backup/resource_backupper.go:226" namespace=nginx-example resource=controllerrevisions
time="2020-05-08T09:33:42Z" level=info msg="Retrieved 0 items" backup=velero/nginx-example-backup-new group=apps/v1 logSource="pkg/backup/resource_backupper.go:240" namespace=nginx-example resource=controllerrevisions
time="2020-05-08T09:33:42Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=apps/v1 logSource="pkg/backup/resource_backupper.go:105" resource=daemonsets
time="2020-05-08T09:33:42Z" level=info msg="Listing items" backup=velero/nginx-example-backup-new group=apps/v1 logSource="pkg/backup/resource_backupper.go:226" namespace=nginx-example resource=daemonsets
time="2020-05-08T09:33:42Z" level=info msg="Retrieved 0 items" backup=velero/nginx-example-backup-new group=apps/v1 logSource="pkg/backup/resource_backupper.go:240" namespace=nginx-example resource=daemonsets
time="2020-05-08T09:33:42Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=apps/v1 logSource="pkg/backup/resource_backupper.go:105" resource=statefulsets
time="2020-05-08T09:33:42Z" level=info msg="Listing items" backup=velero/nginx-example-backup-new group=apps/v1 logSource="pkg/backup/resource_backupper.go:226" namespace=nginx-example resource=statefulsets
time="2020-05-08T09:33:42Z" level=info msg="Retrieved 0 items" backup=velero/nginx-example-backup-new group=apps/v1 logSource="pkg/backup/resource_backupper.go:240" namespace=nginx-example resource=statefulsets
time="2020-05-08T09:33:42Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=apps/v1 logSource="pkg/backup/resource_backupper.go:105" resource=deployments
time="2020-05-08T09:33:42Z" level=info msg="Listing items" backup=velero/nginx-example-backup-new group=apps/v1 logSource="pkg/backup/resource_backupper.go:226" namespace=nginx-example resource=deployments
time="2020-05-08T09:33:42Z" level=info msg="Retrieved 1 items" backup=velero/nginx-example-backup-new group=apps/v1 logSource="pkg/backup/resource_backupper.go:240" namespace=nginx-example resource=deployments
time="2020-05-08T09:33:42Z" level=info msg="Backing up item" backup=velero/nginx-example-backup-new group=apps/v1 logSource="pkg/backup/item_backupper.go:169" name=nginx-deployment namespace=nginx-example resource=deployments.apps
time="2020-05-08T09:33:42Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=apps/v1 logSource="pkg/backup/resource_backupper.go:105" resource=replicasets
time="2020-05-08T09:33:42Z" level=info msg="Listing items" backup=velero/nginx-example-backup-new group=apps/v1 logSource="pkg/backup/resource_backupper.go:226" namespace=nginx-example resource=replicasets
time="2020-05-08T09:33:42Z" level=info msg="Retrieved 1 items" backup=velero/nginx-example-backup-new group=apps/v1 logSource="pkg/backup/resource_backupper.go:240" namespace=nginx-example resource=replicasets
time="2020-05-08T09:33:42Z" level=info msg="Backing up item" backup=velero/nginx-example-backup-new group=apps/v1 logSource="pkg/backup/item_backupper.go:169" name=nginx-deployment-6fbd5f64c4 namespace=nginx-example resource=replicasets.apps
time="2020-05-08T09:33:42Z" level=info msg="Backing up group" backup=velero/nginx-example-backup-new group=events.k8s.io/v1beta1 logSource="pkg/backup/group_backupper.go:101"
time="2020-05-08T09:33:42Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=events.k8s.io/v1beta1 logSource="pkg/backup/resource_backupper.go:105" resource=events
time="2020-05-08T09:33:42Z" level=info msg="Skipping resource because it cohabitates and we've already processed it" backup=velero/nginx-example-backup-new cohabitatingResource1=events cohabitatingResource2=events.events.k8s.io group=events.k8s.io/v1beta1 logSource="pkg/backup/resource_backupper.go:148" resource=events
time="2020-05-08T09:33:42Z" level=info msg="Backing up group" backup=velero/nginx-example-backup-new group=autoscaling/v1 logSource="pkg/backup/group_backupper.go:101"
time="2020-05-08T09:33:42Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=autoscaling/v1 logSource="pkg/backup/resource_backupper.go:105" resource=horizontalpodautoscalers
time="2020-05-08T09:33:42Z" level=info msg="Listing items" backup=velero/nginx-example-backup-new group=autoscaling/v1 logSource="pkg/backup/resource_backupper.go:226" namespace=nginx-example resource=horizontalpodautoscalers
time="2020-05-08T09:33:42Z" level=info msg="Retrieved 0 items" backup=velero/nginx-example-backup-new group=autoscaling/v1 logSource="pkg/backup/resource_backupper.go:240" namespace=nginx-example resource=horizontalpodautoscalers
time="2020-05-08T09:33:42Z" level=info msg="Backing up group" backup=velero/nginx-example-backup-new group=batch/v1 logSource="pkg/backup/group_backupper.go:101"
time="2020-05-08T09:33:42Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=batch/v1 logSource="pkg/backup/resource_backupper.go:105" resource=jobs
time="2020-05-08T09:33:42Z" level=info msg="Listing items" backup=velero/nginx-example-backup-new group=batch/v1 logSource="pkg/backup/resource_backupper.go:226" namespace=nginx-example resource=jobs
time="2020-05-08T09:33:42Z" level=info msg="Retrieved 0 items" backup=velero/nginx-example-backup-new group=batch/v1 logSource="pkg/backup/resource_backupper.go:240" namespace=nginx-example resource=jobs
time="2020-05-08T09:33:42Z" level=info msg="Backing up group" backup=velero/nginx-example-backup-new group=batch/v1beta1 logSource="pkg/backup/group_backupper.go:101"
time="2020-05-08T09:33:42Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=batch/v1beta1 logSource="pkg/backup/resource_backupper.go:105" resource=cronjobs
time="2020-05-08T09:33:42Z" level=info msg="Listing items" backup=velero/nginx-example-backup-new group=batch/v1beta1 logSource="pkg/backup/resource_backupper.go:226" namespace=nginx-example resource=cronjobs
time="2020-05-08T09:33:42Z" level=info msg="Retrieved 0 items" backup=velero/nginx-example-backup-new group=batch/v1beta1 logSource="pkg/backup/resource_backupper.go:240" namespace=nginx-example resource=cronjobs
time="2020-05-08T09:33:42Z" level=info msg="Backing up group" backup=velero/nginx-example-backup-new group=certificates.k8s.io/v1beta1 logSource="pkg/backup/group_backupper.go:101"
time="2020-05-08T09:33:42Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=certificates.k8s.io/v1beta1 logSource="pkg/backup/resource_backupper.go:105" resource=certificatesigningrequests
time="2020-05-08T09:33:42Z" level=info msg="Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup" backup=velero/nginx-example-backup-new group=certificates.k8s.io/v1beta1 logSource="pkg/backup/resource_backupper.go:127" resource=certificatesigningrequests
time="2020-05-08T09:33:42Z" level=info msg="Backing up group" backup=velero/nginx-example-backup-new group=networking.k8s.io/v1 logSource="pkg/backup/group_backupper.go:101"
time="2020-05-08T09:33:42Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=networking.k8s.io/v1 logSource="pkg/backup/resource_backupper.go:105" resource=networkpolicies
time="2020-05-08T09:33:42Z" level=info msg="Listing items" backup=velero/nginx-example-backup-new group=networking.k8s.io/v1 logSource="pkg/backup/resource_backupper.go:226" namespace=nginx-example resource=networkpolicies
time="2020-05-08T09:33:42Z" level=info msg="Retrieved 0 items" backup=velero/nginx-example-backup-new group=networking.k8s.io/v1 logSource="pkg/backup/resource_backupper.go:240" namespace=nginx-example resource=networkpolicies
time="2020-05-08T09:33:42Z" level=info msg="Backing up group" backup=velero/nginx-example-backup-new group=policy/v1beta1 logSource="pkg/backup/group_backupper.go:101"
time="2020-05-08T09:33:42Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=policy/v1beta1 logSource="pkg/backup/resource_backupper.go:105" resource=podsecuritypolicies
time="2020-05-08T09:33:42Z" level=info msg="Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup" backup=velero/nginx-example-backup-new group=policy/v1beta1 logSource="pkg/backup/resource_backupper.go:127" resource=podsecuritypolicies
time="2020-05-08T09:33:42Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=policy/v1beta1 logSource="pkg/backup/resource_backupper.go:105" resource=poddisruptionbudgets
time="2020-05-08T09:33:42Z" level=info msg="Listing items" backup=velero/nginx-example-backup-new group=policy/v1beta1 logSource="pkg/backup/resource_backupper.go:226" namespace=nginx-example resource=poddisruptionbudgets
time="2020-05-08T09:33:42Z" level=info msg="Retrieved 0 items" backup=velero/nginx-example-backup-new group=policy/v1beta1 logSource="pkg/backup/resource_backupper.go:240" namespace=nginx-example resource=poddisruptionbudgets
time="2020-05-08T09:33:42Z" level=info msg="Backing up group" backup=velero/nginx-example-backup-new group=rbac.authorization.k8s.io/v1 logSource="pkg/backup/group_backupper.go:101"
time="2020-05-08T09:33:42Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=rbac.authorization.k8s.io/v1 logSource="pkg/backup/resource_backupper.go:105" resource=roles
time="2020-05-08T09:33:42Z" level=info msg="Listing items" backup=velero/nginx-example-backup-new group=rbac.authorization.k8s.io/v1 logSource="pkg/backup/resource_backupper.go:226" namespace=nginx-example resource=roles
time="2020-05-08T09:33:42Z" level=info msg="Retrieved 0 items" backup=velero/nginx-example-backup-new group=rbac.authorization.k8s.io/v1 logSource="pkg/backup/resource_backupper.go:240" namespace=nginx-example resource=roles
time="2020-05-08T09:33:42Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=rbac.authorization.k8s.io/v1 logSource="pkg/backup/resource_backupper.go:105" resource=clusterrolebindings
time="2020-05-08T09:33:42Z" level=info msg="Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup" backup=velero/nginx-example-backup-new group=rbac.authorization.k8s.io/v1 logSource="pkg/backup/resource_backupper.go:127" resource=clusterrolebindings
time="2020-05-08T09:33:42Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=rbac.authorization.k8s.io/v1 logSource="pkg/backup/resource_backupper.go:105" resource=clusterroles
time="2020-05-08T09:33:42Z" level=info msg="Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup" backup=velero/nginx-example-backup-new group=rbac.authorization.k8s.io/v1 logSource="pkg/backup/resource_backupper.go:127" resource=clusterroles
time="2020-05-08T09:33:42Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=rbac.authorization.k8s.io/v1 logSource="pkg/backup/resource_backupper.go:105" resource=rolebindings
time="2020-05-08T09:33:42Z" level=info msg="Listing items" backup=velero/nginx-example-backup-new group=rbac.authorization.k8s.io/v1 logSource="pkg/backup/resource_backupper.go:226" namespace=nginx-example resource=rolebindings
time="2020-05-08T09:33:42Z" level=info msg="Retrieved 0 items" backup=velero/nginx-example-backup-new group=rbac.authorization.k8s.io/v1 logSource="pkg/backup/resource_backupper.go:240" namespace=nginx-example resource=rolebindings
time="2020-05-08T09:33:42Z" level=info msg="Backing up group" backup=velero/nginx-example-backup-new group=storage.k8s.io/v1 logSource="pkg/backup/group_backupper.go:101"
time="2020-05-08T09:33:42Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=storage.k8s.io/v1 logSource="pkg/backup/resource_backupper.go:105" resource=volumeattachments
time="2020-05-08T09:33:42Z" level=info msg="Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup" backup=velero/nginx-example-backup-new group=storage.k8s.io/v1 logSource="pkg/backup/resource_backupper.go:127" resource=volumeattachments
time="2020-05-08T09:33:42Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=storage.k8s.io/v1 logSource="pkg/backup/resource_backupper.go:105" resource=storageclasses
time="2020-05-08T09:33:42Z" level=info msg="Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup" backup=velero/nginx-example-backup-new group=storage.k8s.io/v1 logSource="pkg/backup/resource_backupper.go:127" resource=storageclasses
time="2020-05-08T09:33:42Z" level=info msg="Backing up group" backup=velero/nginx-example-backup-new group=admissionregistration.k8s.io/v1beta1 logSource="pkg/backup/group_backupper.go:101"
time="2020-05-08T09:33:42Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=admissionregistration.k8s.io/v1beta1 logSource="pkg/backup/resource_backupper.go:105" resource=mutatingwebhookconfigurations
time="2020-05-08T09:33:42Z" level=info msg="Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup" backup=velero/nginx-example-backup-new group=admissionregistration.k8s.io/v1beta1 logSource="pkg/backup/resource_backupper.go:127" resource=mutatingwebhookconfigurations
time="2020-05-08T09:33:42Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=admissionregistration.k8s.io/v1beta1 logSource="pkg/backup/resource_backupper.go:105" resource=validatingwebhookconfigurations
time="2020-05-08T09:33:42Z" level=info msg="Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup" backup=velero/nginx-example-backup-new group=admissionregistration.k8s.io/v1beta1 logSource="pkg/backup/resource_backupper.go:127" resource=validatingwebhookconfigurations
time="2020-05-08T09:33:42Z" level=info msg="Backing up group" backup=velero/nginx-example-backup-new group=admissionregistration.k8s.io/v1alpha1 logSource="pkg/backup/group_backupper.go:101"
time="2020-05-08T09:33:42Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=admissionregistration.k8s.io/v1alpha1 logSource="pkg/backup/resource_backupper.go:105" resource=initializerconfigurations
time="2020-05-08T09:33:42Z" level=info msg="Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup" backup=velero/nginx-example-backup-new group=admissionregistration.k8s.io/v1alpha1 logSource="pkg/backup/resource_backupper.go:127" resource=initializerconfigurations
time="2020-05-08T09:33:42Z" level=info msg="Backing up group" backup=velero/nginx-example-backup-new group=apiextensions.k8s.io/v1beta1 logSource="pkg/backup/group_backupper.go:101"
time="2020-05-08T09:33:42Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=apiextensions.k8s.io/v1beta1 logSource="pkg/backup/resource_backupper.go:105" resource=customresourcedefinitions
time="2020-05-08T09:33:42Z" level=info msg="Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup" backup=velero/nginx-example-backup-new group=apiextensions.k8s.io/v1beta1 logSource="pkg/backup/resource_backupper.go:127" resource=customresourcedefinitions
time="2020-05-08T09:33:42Z" level=info msg="Backing up group" backup=velero/nginx-example-backup-new group=scheduling.k8s.io/v1beta1 logSource="pkg/backup/group_backupper.go:101"
time="2020-05-08T09:33:42Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=scheduling.k8s.io/v1beta1 logSource="pkg/backup/resource_backupper.go:105" resource=priorityclasses
time="2020-05-08T09:33:42Z" level=info msg="Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup" backup=velero/nginx-example-backup-new group=scheduling.k8s.io/v1beta1 logSource="pkg/backup/resource_backupper.go:127" resource=priorityclasses
time="2020-05-08T09:33:42Z" level=info msg="Backing up group" backup=velero/nginx-example-backup-new group=coordination.k8s.io/v1beta1 logSource="pkg/backup/group_backupper.go:101"
time="2020-05-08T09:33:42Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=coordination.k8s.io/v1beta1 logSource="pkg/backup/resource_backupper.go:105" resource=leases
time="2020-05-08T09:33:42Z" level=info msg="Listing items" backup=velero/nginx-example-backup-new group=coordination.k8s.io/v1beta1 logSource="pkg/backup/resource_backupper.go:226" namespace=nginx-example resource=leases
time="2020-05-08T09:33:42Z" level=info msg="Retrieved 0 items" backup=velero/nginx-example-backup-new group=coordination.k8s.io/v1beta1 logSource="pkg/backup/resource_backupper.go:240" namespace=nginx-example resource=leases
time="2020-05-08T09:33:42Z" level=info msg="Backing up group" backup=velero/nginx-example-backup-new group=ceph.rook.io/v1 logSource="pkg/backup/group_backupper.go:101"
time="2020-05-08T09:33:42Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=ceph.rook.io/v1 logSource="pkg/backup/resource_backupper.go:105" resource=cephnfses
time="2020-05-08T09:33:42Z" level=info msg="Listing items" backup=velero/nginx-example-backup-new group=ceph.rook.io/v1 logSource="pkg/backup/resource_backupper.go:226" namespace=nginx-example resource=cephnfses
time="2020-05-08T09:33:42Z" level=info msg="Retrieved 0 items" backup=velero/nginx-example-backup-new group=ceph.rook.io/v1 logSource="pkg/backup/resource_backupper.go:240" namespace=nginx-example resource=cephnfses
time="2020-05-08T09:33:42Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=ceph.rook.io/v1 logSource="pkg/backup/resource_backupper.go:105" resource=cephblockpools
time="2020-05-08T09:33:42Z" level=info msg="Listing items" backup=velero/nginx-example-backup-new group=ceph.rook.io/v1 logSource="pkg/backup/resource_backupper.go:226" namespace=nginx-example resource=cephblockpools
time="2020-05-08T09:33:43Z" level=info msg="Retrieved 0 items" backup=velero/nginx-example-backup-new group=ceph.rook.io/v1 logSource="pkg/backup/resource_backupper.go:240" namespace=nginx-example resource=cephblockpools
time="2020-05-08T09:33:43Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=ceph.rook.io/v1 logSource="pkg/backup/resource_backupper.go:105" resource=cephfilesystems
time="2020-05-08T09:33:43Z" level=info msg="Listing items" backup=velero/nginx-example-backup-new group=ceph.rook.io/v1 logSource="pkg/backup/resource_backupper.go:226" namespace=nginx-example resource=cephfilesystems
time="2020-05-08T09:33:43Z" level=info msg="Retrieved 0 items" backup=velero/nginx-example-backup-new group=ceph.rook.io/v1 logSource="pkg/backup/resource_backupper.go:240" namespace=nginx-example resource=cephfilesystems
time="2020-05-08T09:33:43Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=ceph.rook.io/v1 logSource="pkg/backup/resource_backupper.go:105" resource=cephclusters
time="2020-05-08T09:33:43Z" level=info msg="Listing items" backup=velero/nginx-example-backup-new group=ceph.rook.io/v1 logSource="pkg/backup/resource_backupper.go:226" namespace=nginx-example resource=cephclusters
time="2020-05-08T09:33:43Z" level=info msg="Retrieved 0 items" backup=velero/nginx-example-backup-new group=ceph.rook.io/v1 logSource="pkg/backup/resource_backupper.go:240" namespace=nginx-example resource=cephclusters
time="2020-05-08T09:33:43Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=ceph.rook.io/v1 logSource="pkg/backup/resource_backupper.go:105" resource=cephobjectstoreusers
time="2020-05-08T09:33:43Z" level=info msg="Listing items" backup=velero/nginx-example-backup-new group=ceph.rook.io/v1 logSource="pkg/backup/resource_backupper.go:226" namespace=nginx-example resource=cephobjectstoreusers
time="2020-05-08T09:33:43Z" level=info msg="Retrieved 0 items" backup=velero/nginx-example-backup-new group=ceph.rook.io/v1 logSource="pkg/backup/resource_backupper.go:240" namespace=nginx-example resource=cephobjectstoreusers
time="2020-05-08T09:33:43Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=ceph.rook.io/v1 logSource="pkg/backup/resource_backupper.go:105" resource=cephobjectstores
time="2020-05-08T09:33:43Z" level=info msg="Listing items" backup=velero/nginx-example-backup-new group=ceph.rook.io/v1 logSource="pkg/backup/resource_backupper.go:226" namespace=nginx-example resource=cephobjectstores
time="2020-05-08T09:33:43Z" level=info msg="Retrieved 0 items" backup=velero/nginx-example-backup-new group=ceph.rook.io/v1 logSource="pkg/backup/resource_backupper.go:240" namespace=nginx-example resource=cephobjectstores
time="2020-05-08T09:33:43Z" level=info msg="Backing up group" backup=velero/nginx-example-backup-new group=monitoring.coreos.com/v1 logSource="pkg/backup/group_backupper.go:101"
time="2020-05-08T09:33:43Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=monitoring.coreos.com/v1 logSource="pkg/backup/resource_backupper.go:105" resource=prometheusrules
time="2020-05-08T09:33:43Z" level=info msg="Listing items" backup=velero/nginx-example-backup-new group=monitoring.coreos.com/v1 logSource="pkg/backup/resource_backupper.go:226" namespace=nginx-example resource=prometheusrules
time="2020-05-08T09:33:43Z" level=info msg="Retrieved 0 items" backup=velero/nginx-example-backup-new group=monitoring.coreos.com/v1 logSource="pkg/backup/resource_backupper.go:240" namespace=nginx-example resource=prometheusrules
time="2020-05-08T09:33:43Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=monitoring.coreos.com/v1 logSource="pkg/backup/resource_backupper.go:105" resource=servicemonitors
time="2020-05-08T09:33:43Z" level=info msg="Listing items" backup=velero/nginx-example-backup-new group=monitoring.coreos.com/v1 logSource="pkg/backup/resource_backupper.go:226" namespace=nginx-example resource=servicemonitors
time="2020-05-08T09:33:43Z" level=info msg="Retrieved 0 items" backup=velero/nginx-example-backup-new group=monitoring.coreos.com/v1 logSource="pkg/backup/resource_backupper.go:240" namespace=nginx-example resource=servicemonitors
time="2020-05-08T09:33:43Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=monitoring.coreos.com/v1 logSource="pkg/backup/resource_backupper.go:105" resource=prometheuses
time="2020-05-08T09:33:43Z" level=info msg="Listing items" backup=velero/nginx-example-backup-new group=monitoring.coreos.com/v1 logSource="pkg/backup/resource_backupper.go:226" namespace=nginx-example resource=prometheuses
time="2020-05-08T09:33:43Z" level=info msg="Retrieved 0 items" backup=velero/nginx-example-backup-new group=monitoring.coreos.com/v1 logSource="pkg/backup/resource_backupper.go:240" namespace=nginx-example resource=prometheuses
time="2020-05-08T09:33:43Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=monitoring.coreos.com/v1 logSource="pkg/backup/resource_backupper.go:105" resource=alertmanagers
time="2020-05-08T09:33:43Z" level=info msg="Listing items" backup=velero/nginx-example-backup-new group=monitoring.coreos.com/v1 logSource="pkg/backup/resource_backupper.go:226" namespace=nginx-example resource=alertmanagers
time="2020-05-08T09:33:43Z" level=info msg="Retrieved 0 items" backup=velero/nginx-example-backup-new group=monitoring.coreos.com/v1 logSource="pkg/backup/resource_backupper.go:240" namespace=nginx-example resource=alertmanagers
time="2020-05-08T09:33:43Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=monitoring.coreos.com/v1 logSource="pkg/backup/resource_backupper.go:105" resource=podmonitors
time="2020-05-08T09:33:43Z" level=info msg="Listing items" backup=velero/nginx-example-backup-new group=monitoring.coreos.com/v1 logSource="pkg/backup/resource_backupper.go:226" namespace=nginx-example resource=podmonitors
time="2020-05-08T09:33:43Z" level=info msg="Retrieved 0 items" backup=velero/nginx-example-backup-new group=monitoring.coreos.com/v1 logSource="pkg/backup/resource_backupper.go:240" namespace=nginx-example resource=podmonitors
time="2020-05-08T09:33:43Z" level=info msg="Backing up group" backup=velero/nginx-example-backup-new group=velero.io/v1 logSource="pkg/backup/group_backupper.go:101"
time="2020-05-08T09:33:43Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=velero.io/v1 logSource="pkg/backup/resource_backupper.go:105" resource=restores
time="2020-05-08T09:33:43Z" level=info msg="Listing items" backup=velero/nginx-example-backup-new group=velero.io/v1 logSource="pkg/backup/resource_backupper.go:226" namespace=nginx-example resource=restores
time="2020-05-08T09:33:43Z" level=info msg="Retrieved 0 items" backup=velero/nginx-example-backup-new group=velero.io/v1 logSource="pkg/backup/resource_backupper.go:240" namespace=nginx-example resource=restores
time="2020-05-08T09:33:43Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=velero.io/v1 logSource="pkg/backup/resource_backupper.go:105" resource=schedules
time="2020-05-08T09:33:43Z" level=info msg="Listing items" backup=velero/nginx-example-backup-new group=velero.io/v1 logSource="pkg/backup/resource_backupper.go:226" namespace=nginx-example resource=schedules
time="2020-05-08T09:33:43Z" level=info msg="Retrieved 0 items" backup=velero/nginx-example-backup-new group=velero.io/v1 logSource="pkg/backup/resource_backupper.go:240" namespace=nginx-example resource=schedules
time="2020-05-08T09:33:43Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=velero.io/v1 logSource="pkg/backup/resource_backupper.go:105" resource=podvolumebackups
time="2020-05-08T09:33:43Z" level=info msg="Listing items" backup=velero/nginx-example-backup-new group=velero.io/v1 logSource="pkg/backup/resource_backupper.go:226" namespace=nginx-example resource=podvolumebackups
time="2020-05-08T09:33:43Z" level=info msg="Retrieved 0 items" backup=velero/nginx-example-backup-new group=velero.io/v1 logSource="pkg/backup/resource_backupper.go:240" namespace=nginx-example resource=podvolumebackups
time="2020-05-08T09:33:43Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=velero.io/v1 logSource="pkg/backup/resource_backupper.go:105" resource=podvolumerestores
time="2020-05-08T09:33:43Z" level=info msg="Listing items" backup=velero/nginx-example-backup-new group=velero.io/v1 logSource="pkg/backup/resource_backupper.go:226" namespace=nginx-example resource=podvolumerestores
time="2020-05-08T09:33:43Z" level=info msg="Retrieved 0 items" backup=velero/nginx-example-backup-new group=velero.io/v1 logSource="pkg/backup/resource_backupper.go:240" namespace=nginx-example resource=podvolumerestores
time="2020-05-08T09:33:43Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=velero.io/v1 logSource="pkg/backup/resource_backupper.go:105" resource=resticrepositories
time="2020-05-08T09:33:43Z" level=info msg="Listing items" backup=velero/nginx-example-backup-new group=velero.io/v1 logSource="pkg/backup/resource_backupper.go:226" namespace=nginx-example resource=resticrepositories
time="2020-05-08T09:33:43Z" level=info msg="Retrieved 0 items" backup=velero/nginx-example-backup-new group=velero.io/v1 logSource="pkg/backup/resource_backupper.go:240" namespace=nginx-example resource=resticrepositories
time="2020-05-08T09:33:43Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=velero.io/v1 logSource="pkg/backup/resource_backupper.go:105" resource=backups
time="2020-05-08T09:33:43Z" level=info msg="Listing items" backup=velero/nginx-example-backup-new group=velero.io/v1 logSource="pkg/backup/resource_backupper.go:226" namespace=nginx-example resource=backups
time="2020-05-08T09:33:43Z" level=info msg="Retrieved 0 items" backup=velero/nginx-example-backup-new group=velero.io/v1 logSource="pkg/backup/resource_backupper.go:240" namespace=nginx-example resource=backups
time="2020-05-08T09:33:43Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=velero.io/v1 logSource="pkg/backup/resource_backupper.go:105" resource=downloadrequests
time="2020-05-08T09:33:43Z" level=info msg="Listing items" backup=velero/nginx-example-backup-new group=velero.io/v1 logSource="pkg/backup/resource_backupper.go:226" namespace=nginx-example resource=downloadrequests
time="2020-05-08T09:33:43Z" level=info msg="Retrieved 0 items" backup=velero/nginx-example-backup-new group=velero.io/v1 logSource="pkg/backup/resource_backupper.go:240" namespace=nginx-example resource=downloadrequests
time="2020-05-08T09:33:43Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=velero.io/v1 logSource="pkg/backup/resource_backupper.go:105" resource=deletebackuprequests
time="2020-05-08T09:33:43Z" level=info msg="Listing items" backup=velero/nginx-example-backup-new group=velero.io/v1 logSource="pkg/backup/resource_backupper.go:226" namespace=nginx-example resource=deletebackuprequests
time="2020-05-08T09:33:43Z" level=info msg="Retrieved 0 items" backup=velero/nginx-example-backup-new group=velero.io/v1 logSource="pkg/backup/resource_backupper.go:240" namespace=nginx-example resource=deletebackuprequests
time="2020-05-08T09:33:43Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=velero.io/v1 logSource="pkg/backup/resource_backupper.go:105" resource=serverstatusrequests
time="2020-05-08T09:33:43Z" level=info msg="Listing items" backup=velero/nginx-example-backup-new group=velero.io/v1 logSource="pkg/backup/resource_backupper.go:226" namespace=nginx-example resource=serverstatusrequests
time="2020-05-08T09:33:43Z" level=info msg="Retrieved 0 items" backup=velero/nginx-example-backup-new group=velero.io/v1 logSource="pkg/backup/resource_backupper.go:240" namespace=nginx-example resource=serverstatusrequests
time="2020-05-08T09:33:43Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=velero.io/v1 logSource="pkg/backup/resource_backupper.go:105" resource=backupstoragelocations
time="2020-05-08T09:33:43Z" level=info msg="Listing items" backup=velero/nginx-example-backup-new group=velero.io/v1 logSource="pkg/backup/resource_backupper.go:226" namespace=nginx-example resource=backupstoragelocations
time="2020-05-08T09:33:43Z" level=info msg="Retrieved 0 items" backup=velero/nginx-example-backup-new group=velero.io/v1 logSource="pkg/backup/resource_backupper.go:240" namespace=nginx-example resource=backupstoragelocations
time="2020-05-08T09:33:43Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=velero.io/v1 logSource="pkg/backup/resource_backupper.go:105" resource=volumesnapshotlocations
time="2020-05-08T09:33:43Z" level=info msg="Listing items" backup=velero/nginx-example-backup-new group=velero.io/v1 logSource="pkg/backup/resource_backupper.go:226" namespace=nginx-example resource=volumesnapshotlocations
time="2020-05-08T09:33:44Z" level=info msg="Retrieved 0 items" backup=velero/nginx-example-backup-new group=velero.io/v1 logSource="pkg/backup/resource_backupper.go:240" namespace=nginx-example resource=volumesnapshotlocations
time="2020-05-08T09:33:44Z" level=info msg="Backing up group" backup=velero/nginx-example-backup-new group=certmanager.k8s.io/v1alpha1 logSource="pkg/backup/group_backupper.go:101"
time="2020-05-08T09:33:44Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=certmanager.k8s.io/v1alpha1 logSource="pkg/backup/resource_backupper.go:105" resource=certificates
time="2020-05-08T09:33:44Z" level=info msg="Listing items" backup=velero/nginx-example-backup-new group=certmanager.k8s.io/v1alpha1 logSource="pkg/backup/resource_backupper.go:226" namespace=nginx-example resource=certificates
time="2020-05-08T09:33:44Z" level=info msg="Retrieved 0 items" backup=velero/nginx-example-backup-new group=certmanager.k8s.io/v1alpha1 logSource="pkg/backup/resource_backupper.go:240" namespace=nginx-example resource=certificates
time="2020-05-08T09:33:44Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=certmanager.k8s.io/v1alpha1 logSource="pkg/backup/resource_backupper.go:105" resource=issuers
time="2020-05-08T09:33:44Z" level=info msg="Listing items" backup=velero/nginx-example-backup-new group=certmanager.k8s.io/v1alpha1 logSource="pkg/backup/resource_backupper.go:226" namespace=nginx-example resource=issuers
time="2020-05-08T09:33:44Z" level=info msg="Retrieved 0 items" backup=velero/nginx-example-backup-new group=certmanager.k8s.io/v1alpha1 logSource="pkg/backup/resource_backupper.go:240" namespace=nginx-example resource=issuers
time="2020-05-08T09:33:44Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=certmanager.k8s.io/v1alpha1 logSource="pkg/backup/resource_backupper.go:105" resource=clusterissuers
time="2020-05-08T09:33:44Z" level=info msg="Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup" backup=velero/nginx-example-backup-new group=certmanager.k8s.io/v1alpha1 logSource="pkg/backup/resource_backupper.go:127" resource=clusterissuers
time="2020-05-08T09:33:44Z" level=info msg="Backing up group" backup=velero/nginx-example-backup-new group=nfs.rook.io/v1alpha1 logSource="pkg/backup/group_backupper.go:101"
time="2020-05-08T09:33:44Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=nfs.rook.io/v1alpha1 logSource="pkg/backup/resource_backupper.go:105" resource=nfsservers
time="2020-05-08T09:33:44Z" level=info msg="Listing items" backup=velero/nginx-example-backup-new group=nfs.rook.io/v1alpha1 logSource="pkg/backup/resource_backupper.go:226" namespace=nginx-example resource=nfsservers
time="2020-05-08T09:33:44Z" level=info msg="Retrieved 0 items" backup=velero/nginx-example-backup-new group=nfs.rook.io/v1alpha1 logSource="pkg/backup/resource_backupper.go:240" namespace=nginx-example resource=nfsservers
time="2020-05-08T09:33:44Z" level=info msg="Backing up group" backup=velero/nginx-example-backup-new group=objectbucket.io/v1alpha1 logSource="pkg/backup/group_backupper.go:101"
time="2020-05-08T09:33:44Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=objectbucket.io/v1alpha1 logSource="pkg/backup/resource_backupper.go:105" resource=objectbuckets
time="2020-05-08T09:33:44Z" level=info msg="Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup" backup=velero/nginx-example-backup-new group=objectbucket.io/v1alpha1 logSource="pkg/backup/resource_backupper.go:127" resource=objectbuckets
time="2020-05-08T09:33:44Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=objectbucket.io/v1alpha1 logSource="pkg/backup/resource_backupper.go:105" resource=objectbucketclaims
time="2020-05-08T09:33:44Z" level=info msg="Listing items" backup=velero/nginx-example-backup-new group=objectbucket.io/v1alpha1 logSource="pkg/backup/resource_backupper.go:226" namespace=nginx-example resource=objectbucketclaims
time="2020-05-08T09:33:44Z" level=info msg="Retrieved 0 items" backup=velero/nginx-example-backup-new group=objectbucket.io/v1alpha1 logSource="pkg/backup/resource_backupper.go:240" namespace=nginx-example resource=objectbucketclaims
time="2020-05-08T09:33:44Z" level=info msg="Backing up group" backup=velero/nginx-example-backup-new group=snapshot.storage.k8s.io/v1alpha1 logSource="pkg/backup/group_backupper.go:101"
time="2020-05-08T09:33:44Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=snapshot.storage.k8s.io/v1alpha1 logSource="pkg/backup/resource_backupper.go:105" resource=volumesnapshotcontents
time="2020-05-08T09:33:44Z" level=info msg="Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup" backup=velero/nginx-example-backup-new group=snapshot.storage.k8s.io/v1alpha1 logSource="pkg/backup/resource_backupper.go:127" resource=volumesnapshotcontents
time="2020-05-08T09:33:44Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=snapshot.storage.k8s.io/v1alpha1 logSource="pkg/backup/resource_backupper.go:105" resource=volumesnapshots
time="2020-05-08T09:33:44Z" level=info msg="Listing items" backup=velero/nginx-example-backup-new group=snapshot.storage.k8s.io/v1alpha1 logSource="pkg/backup/resource_backupper.go:226" namespace=nginx-example resource=volumesnapshots
time="2020-05-08T09:33:44Z" level=info msg="Retrieved 0 items" backup=velero/nginx-example-backup-new group=snapshot.storage.k8s.io/v1alpha1 logSource="pkg/backup/resource_backupper.go:240" namespace=nginx-example resource=volumesnapshots
time="2020-05-08T09:33:44Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=snapshot.storage.k8s.io/v1alpha1 logSource="pkg/backup/resource_backupper.go:105" resource=volumesnapshotclasses
time="2020-05-08T09:33:44Z" level=info msg="Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup" backup=velero/nginx-example-backup-new group=snapshot.storage.k8s.io/v1alpha1 logSource="pkg/backup/resource_backupper.go:127" resource=volumesnapshotclasses
time="2020-05-08T09:33:44Z" level=info msg="Backing up group" backup=velero/nginx-example-backup-new group=rook.io/v1alpha2 logSource="pkg/backup/group_backupper.go:101"
time="2020-05-08T09:33:44Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=rook.io/v1alpha2 logSource="pkg/backup/resource_backupper.go:105" resource=volumes
time="2020-05-08T09:33:44Z" level=info msg="Listing items" backup=velero/nginx-example-backup-new group=rook.io/v1alpha2 logSource="pkg/backup/resource_backupper.go:226" namespace=nginx-example resource=volumes
time="2020-05-08T09:33:44Z" level=info msg="Retrieved 0 items" backup=velero/nginx-example-backup-new group=rook.io/v1alpha2 logSource="pkg/backup/resource_backupper.go:240" namespace=nginx-example resource=volumes
time="2020-05-08T09:33:44Z" level=info msg="Backing up group" backup=velero/nginx-example-backup-new group=kubeless.io/v1beta1 logSource="pkg/backup/group_backupper.go:101"
time="2020-05-08T09:33:44Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=kubeless.io/v1beta1 logSource="pkg/backup/resource_backupper.go:105" resource=cronjobtriggers
time="2020-05-08T09:33:44Z" level=info msg="Listing items" backup=velero/nginx-example-backup-new group=kubeless.io/v1beta1 logSource="pkg/backup/resource_backupper.go:226" namespace=nginx-example resource=cronjobtriggers
time="2020-05-08T09:33:44Z" level=info msg="Retrieved 0 items" backup=velero/nginx-example-backup-new group=kubeless.io/v1beta1 logSource="pkg/backup/resource_backupper.go:240" namespace=nginx-example resource=cronjobtriggers
time="2020-05-08T09:33:44Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=kubeless.io/v1beta1 logSource="pkg/backup/resource_backupper.go:105" resource=httptriggers
time="2020-05-08T09:33:44Z" level=info msg="Listing items" backup=velero/nginx-example-backup-new group=kubeless.io/v1beta1 logSource="pkg/backup/resource_backupper.go:226" namespace=nginx-example resource=httptriggers
time="2020-05-08T09:33:44Z" level=info msg="Retrieved 0 items" backup=velero/nginx-example-backup-new group=kubeless.io/v1beta1 logSource="pkg/backup/resource_backupper.go:240" namespace=nginx-example resource=httptriggers
time="2020-05-08T09:33:44Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=kubeless.io/v1beta1 logSource="pkg/backup/resource_backupper.go:105" resource=functions
time="2020-05-08T09:33:44Z" level=info msg="Listing items" backup=velero/nginx-example-backup-new group=kubeless.io/v1beta1 logSource="pkg/backup/resource_backupper.go:226" namespace=nginx-example resource=functions
time="2020-05-08T09:33:44Z" level=info msg="Retrieved 0 items" backup=velero/nginx-example-backup-new group=kubeless.io/v1beta1 logSource="pkg/backup/resource_backupper.go:240" namespace=nginx-example resource=functions
time="2020-05-08T09:33:44Z" level=info msg="Backing up group" backup=velero/nginx-example-backup-new group=extensions/v1beta1 logSource="pkg/backup/group_backupper.go:101"
time="2020-05-08T09:33:44Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=extensions/v1beta1 logSource="pkg/backup/resource_backupper.go:105" resource=podsecuritypolicies
time="2020-05-08T09:33:44Z" level=info msg="Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup" backup=velero/nginx-example-backup-new group=extensions/v1beta1 logSource="pkg/backup/resource_backupper.go:127" resource=podsecuritypolicies
time="2020-05-08T09:33:44Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=extensions/v1beta1 logSource="pkg/backup/resource_backupper.go:105" resource=networkpolicies
time="2020-05-08T09:33:44Z" level=info msg="Skipping resource because it cohabitates and we've already processed it" backup=velero/nginx-example-backup-new cohabitatingResource1=networkpolicies.extensions cohabitatingResource2=networkpolicies.networking.k8s.io group=extensions/v1beta1 logSource="pkg/backup/resource_backupper.go:148" resource=networkpolicies
time="2020-05-08T09:33:44Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=extensions/v1beta1 logSource="pkg/backup/resource_backupper.go:105" resource=replicasets
time="2020-05-08T09:33:44Z" level=info msg="Skipping resource because it cohabitates and we've already processed it" backup=velero/nginx-example-backup-new cohabitatingResource1=replicasets.extensions cohabitatingResource2=replicasets.apps group=extensions/v1beta1 logSource="pkg/backup/resource_backupper.go:148" resource=replicasets
time="2020-05-08T09:33:44Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=extensions/v1beta1 logSource="pkg/backup/resource_backupper.go:105" resource=deployments
time="2020-05-08T09:33:44Z" level=info msg="Skipping resource because it cohabitates and we've already processed it" backup=velero/nginx-example-backup-new cohabitatingResource1=deployments.extensions cohabitatingResource2=deployments.apps group=extensions/v1beta1 logSource="pkg/backup/resource_backupper.go:148" resource=deployments
time="2020-05-08T09:33:44Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=extensions/v1beta1 logSource="pkg/backup/resource_backupper.go:105" resource=ingresses
time="2020-05-08T09:33:44Z" level=info msg="Listing items" backup=velero/nginx-example-backup-new group=extensions/v1beta1 logSource="pkg/backup/resource_backupper.go:226" namespace=nginx-example resource=ingresses
time="2020-05-08T09:33:44Z" level=info msg="Retrieved 0 items" backup=velero/nginx-example-backup-new group=extensions/v1beta1 logSource="pkg/backup/resource_backupper.go:240" namespace=nginx-example resource=ingresses
time="2020-05-08T09:33:44Z" level=info msg="Backing up resource" backup=velero/nginx-example-backup-new group=extensions/v1beta1 logSource="pkg/backup/resource_backupper.go:105" resource=daemonsets
time="2020-05-08T09:33:44Z" level=info msg="Skipping resource because it cohabitates and we've already processed it" backup=velero/nginx-example-backup-new cohabitatingResource1=daemonsets.extensions cohabitatingResource2=daemonsets.apps group=extensions/v1beta1 logSource="pkg/backup/resource_backupper.go:148" resource=daemonsets
kubectl -n velero get podvolumebackups -l velero.io/backup-name=nginx-example-backup-new -o yaml
apiVersion: v1
items:
- apiVersion: velero.io/v1
  kind: PodVolumeBackup
  metadata:
    annotations:
      velero.io/pvc-name: nginx-logs
    creationTimestamp: "2020-05-08T09:33:40Z"
    generateName: nginx-example-backup-new-
    generation: 3
    labels:
      velero.io/backup-name: nginx-example-backup-new
      velero.io/backup-uid: f7226a8c-910e-11ea-ac8a-028358a66085
      velero.io/pvc-uid: d2bc87d4-9109-11ea-a626-02021a19689b
    name: nginx-example-backup-new-wc88w
    namespace: velero
    ownerReferences:
    - apiVersion: velero.io/v1
      controller: true
      kind: Backup
      name: nginx-example-backup-new
      uid: f7226a8c-910e-11ea-ac8a-028358a66085
    resourceVersion: "309164"
    selfLink: /apis/velero.io/v1/namespaces/velero/podvolumebackups/nginx-example-backup-new-wc88w
    uid: 002c9523-910f-11ea-ac8a-028358a66085
  spec:
    backupStorageLocation: default
    node: ip-10-165-138-230.ec2.internal
    pod:
      kind: Pod
      name: nginx-deployment-6fbd5f64c4-fps5c
      namespace: nginx-example
      uid: 4640dfa5-910a-11ea-ac8a-028358a66085
    repoIdentifier: s3:s3.amazonaws.com/cva-builder-velero-backups-test/restic/nginx-example
    tags:
      backup: nginx-example-backup-new
      backup-uid: f7226a8c-910e-11ea-ac8a-028358a66085
      ns: nginx-example
      pod: nginx-deployment-6fbd5f64c4-fps5c
      pod-uid: 4640dfa5-910a-11ea-ac8a-028358a66085
      pvc-uid: d2bc87d4-9109-11ea-a626-02021a19689b
      volume: nginx-logs
    volume: nginx-logs
  status:
    completionTimestamp: "2020-05-08T09:33:42Z"
    path: /host_pods/4640dfa5-910a-11ea-ac8a-028358a66085/volumes/kubernetes.io~local-volume/nginx-logs-volume
    phase: Completed
    progress: {}
    snapshotID: ffabaa99
    startTimestamp: "2020-05-08T09:33:40Z"
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""
kubectl delete deployment -n nginx-example --all
deployment.extensions "nginx-deployment" deleted

kubectl delete pv nginx-logs-volume
persistentvolume "nginx-logs-volume" deleted

kubectl delete namespace nginx-example
namespace "nginx-example" deleted
velero restore create --from-backup nginx-example-backup-new
Restore request "nginx-example-backup-new-20200508094438" submitted successfully.
Run `velero restore describe nginx-example-backup-new-20200508094438` or `velero restore logs nginx-example-backup-new-20200508094438` for more details.
velero restore describe nginx-example-backup-new-20200508094438 --details
Name:         nginx-example-backup-new-20200508094438
Namespace:    velero
Labels:       <none>
Annotations:  <none>

Phase:  InProgress

Backup:  nginx-example-backup-new

Namespaces:
  Included:  *
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        nodes, events, events.events.k8s.io, backups.velero.io, restores.velero.io, resticrepositories.velero.io
  Cluster-scoped:  auto

Namespace mappings:  <none>

Label selector:  <none>

Restore PVs:  auto

Restic Restores:
  New:
    nginx-example/nginx-deployment-6fbd5f64c4-fps5c: nginx-logs

In this attempt - the restore hung for an hour before failing. It hung on the "waiting for all restic restores to complete"

time="2020-05-08T09:44:38Z" level=info msg="starting restore" logSource="pkg/controller/restore_controller.go:458" restore=velero/nginx-example-backup-new-20200508094438
time="2020-05-08T09:44:38Z" level=info msg="Not including resource" groupResource=nodes logSource="pkg/restore/restore.go:138" restore=velero/nginx-example-backup-new-20200508094438
time="2020-05-08T09:44:38Z" level=info msg="Not including resource" groupResource=events logSource="pkg/restore/restore.go:138" restore=velero/nginx-example-backup-new-20200508094438
time="2020-05-08T09:44:38Z" level=info msg="Not including resource" groupResource=events.events.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/nginx-example-backup-new-20200508094438
time="2020-05-08T09:44:38Z" level=info msg="Not including resource" groupResource=backups.velero.io logSource="pkg/restore/restore.go:138" restore=velero/nginx-example-backup-new-20200508094438
time="2020-05-08T09:44:38Z" level=info msg="Not including resource" groupResource=restores.velero.io logSource="pkg/restore/restore.go:138" restore=velero/nginx-example-backup-new-20200508094438
time="2020-05-08T09:44:38Z" level=info msg="Not including resource" groupResource=resticrepositories.velero.io logSource="pkg/restore/restore.go:138" restore=velero/nginx-example-backup-new-20200508094438
time="2020-05-08T09:44:38Z" level=info msg="Starting restore of backup velero/nginx-example-backup-new" logSource="pkg/restore/restore.go:394" restore=velero/nginx-example-backup-new-20200508094438
time="2020-05-08T09:44:38Z" level=info msg="Restoring cluster level resource 'persistentvolumes'" logSource="pkg/restore/restore.go:779" restore=velero/nginx-example-backup-new-20200508094438
time="2020-05-08T09:44:38Z" level=info msg="Getting client for /v1, Kind=PersistentVolume" logSource="pkg/restore/restore.go:821" restore=velero/nginx-example-backup-new-20200508094438
time="2020-05-08T09:44:38Z" level=info msg="Dynamically re-provisioning persistent volume because it has a restic backup to be restored." logSource="pkg/restore/restore.go:992" restore=velero/nginx-example-backup-new-20200508094438
time="2020-05-08T09:44:38Z" level=info msg="Restoring resource 'persistentvolumeclaims' into namespace 'nginx-example'" logSource="pkg/restore/restore.go:777" restore=velero/nginx-example-backup-new-20200508094438
time="2020-05-08T09:44:38Z" level=info msg="Getting client for /v1, Kind=PersistentVolumeClaim" logSource="pkg/restore/restore.go:821" restore=velero/nginx-example-backup-new-20200508094438
time="2020-05-08T09:44:38Z" level=info msg="Executing item action for persistentvolumeclaims" logSource="pkg/restore/restore.go:1030" restore=velero/nginx-example-backup-new-20200508094438
time="2020-05-08T09:44:38Z" level=info msg="Executing AddPVFromPVCAction" cmd=/velero logSource="pkg/restore/add_pv_from_pvc_action.go:44" pluginName=velero restore=velero/nginx-example-backup-new-20200508094438
time="2020-05-08T09:44:38Z" level=info msg="Adding PV nginx-logs-volume as an additional item to restore" cmd=/velero logSource="pkg/restore/add_pv_from_pvc_action.go:66" pluginName=velero restore=velero/nginx-example-backup-new-20200508094438
time="2020-05-08T09:44:38Z" level=info msg="Skipping persistentvolumes/nginx-logs-volume because it's already been restored." logSource="pkg/restore/restore.go:910" restore=velero/nginx-example-backup-new-20200508094438
time="2020-05-08T09:44:38Z" level=info msg="Executing item action for persistentvolumeclaims" logSource="pkg/restore/restore.go:1030" restore=velero/nginx-example-backup-new-20200508094438
time="2020-05-08T09:44:38Z" level=info msg="Executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:63" pluginName=velero restore=velero/nginx-example-backup-new-20200508094438
time="2020-05-08T09:44:38Z" level=info msg="Done executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:74" pluginName=velero restore=velero/nginx-example-backup-new-20200508094438
time="2020-05-08T09:44:38Z" level=info msg="Resetting PersistentVolumeClaim nginx-example/nginx-logs for dynamic provisioning because its PV nginx-logs-volume has a reclaim policy of Delete" logSource="pkg/restore/restore.go:1103" restore=velero/nginx-example-backup-new-20200508094438
time="2020-05-08T09:44:38Z" level=info msg="Attempting to restore PersistentVolumeClaim: nginx-logs" logSource="pkg/restore/restore.go:1136" restore=velero/nginx-example-backup-new-20200508094438
time="2020-05-08T09:44:38Z" level=info msg="Restoring resource 'secrets' into namespace 'nginx-example'" logSource="pkg/restore/restore.go:777" restore=velero/nginx-example-backup-new-20200508094438
time="2020-05-08T09:44:38Z" level=info msg="Getting client for /v1, Kind=Secret" logSource="pkg/restore/restore.go:821" restore=velero/nginx-example-backup-new-20200508094438
time="2020-05-08T09:44:38Z" level=info msg="Attempting to restore Secret: default-token-zmrr5" logSource="pkg/restore/restore.go:1136" restore=velero/nginx-example-backup-new-20200508094438
time="2020-05-08T09:44:38Z" level=info msg="Restoring resource 'serviceaccounts' into namespace 'nginx-example'" logSource="pkg/restore/restore.go:777" restore=velero/nginx-example-backup-new-20200508094438
time="2020-05-08T09:44:38Z" level=info msg="Getting client for /v1, Kind=ServiceAccount" logSource="pkg/restore/restore.go:821" restore=velero/nginx-example-backup-new-20200508094438
time="2020-05-08T09:44:38Z" level=info msg="Executing item action for serviceaccounts" logSource="pkg/restore/restore.go:1030" restore=velero/nginx-example-backup-new-20200508094438
time="2020-05-08T09:44:38Z" level=info msg="Executing ServiceAccountAction" cmd=/velero logSource="pkg/restore/service_account_action.go:47" pluginName=velero restore=velero/nginx-example-backup-new-20200508094438
time="2020-05-08T09:44:38Z" level=info msg="Done executing ServiceAccountAction" cmd=/velero logSource="pkg/restore/service_account_action.go:78" pluginName=velero restore=velero/nginx-example-backup-new-20200508094438
time="2020-05-08T09:44:38Z" level=info msg="Attempting to restore ServiceAccount: default" logSource="pkg/restore/restore.go:1136" restore=velero/nginx-example-backup-new-20200508094438
time="2020-05-08T09:44:38Z" level=info msg="Restoring resource 'pods' into namespace 'nginx-example'" logSource="pkg/restore/restore.go:777" restore=velero/nginx-example-backup-new-20200508094438
time="2020-05-08T09:44:38Z" level=info msg="Getting client for /v1, Kind=Pod" logSource="pkg/restore/restore.go:821" restore=velero/nginx-example-backup-new-20200508094438
time="2020-05-08T09:44:38Z" level=info msg="Executing item action for pods" logSource="pkg/restore/restore.go:1030" restore=velero/nginx-example-backup-new-20200508094438
time="2020-05-08T09:44:38Z" level=info msg="Executing AddPVCFromPodAction" cmd=/velero logSource="pkg/restore/add_pvc_from_pod_action.go:44" pluginName=velero restore=velero/nginx-example-backup-new-20200508094438
time="2020-05-08T09:44:38Z" level=info msg="Adding PVC nginx-example/nginx-logs as an additional item to restore" cmd=/velero logSource="pkg/restore/add_pvc_from_pod_action.go:58" pluginName=velero restore=velero/nginx-example-backup-new-20200508094438
time="2020-05-08T09:44:38Z" level=info msg="Skipping persistentvolumeclaims/nginx-example/nginx-logs because it's already been restored." logSource="pkg/restore/restore.go:910" restore=velero/nginx-example-backup-new-20200508094438
time="2020-05-08T09:44:38Z" level=info msg="Executing item action for pods" logSource="pkg/restore/restore.go:1030" restore=velero/nginx-example-backup-new-20200508094438
time="2020-05-08T09:44:38Z" level=info msg="Executing item action for pods" logSource="pkg/restore/restore.go:1030" restore=velero/nginx-example-backup-new-20200508094438
time="2020-05-08T09:44:38Z" level=info msg="Executing ResticRestoreAction" cmd=/velero logSource="pkg/restore/restic_restore_action.go:68" pluginName=velero restore=velero/nginx-example-backup-new-20200508094438
time="2020-05-08T09:44:38Z" level=info msg="Restic backups for pod found" cmd=/velero logSource="pkg/restore/restic_restore_action.go:94" pluginName=velero pod=nginx-example/nginx-deployment-6fbd5f64c4-fps5c restore=velero/nginx-example-backup-new-20200508094438
time="2020-05-08T09:44:38Z" level=info msg="Using image \"velero/velero-restic-restore-helper:v1.3.2\"" cmd=/velero logSource="pkg/restore/restic_restore_action.go:105" pluginName=velero pod=nginx-example/nginx-deployment-6fbd5f64c4-fps5c restore=velero/nginx-example-backup-new-20200508094438
time="2020-05-08T09:44:38Z" level=info msg="Done executing ResticRestoreAction" cmd=/velero logSource="pkg/restore/restic_restore_action.go:154" pluginName=velero restore=velero/nginx-example-backup-new-20200508094438
time="2020-05-08T09:44:38Z" level=info msg="Attempting to restore Pod: nginx-deployment-6fbd5f64c4-fps5c" logSource="pkg/restore/restore.go:1136" restore=velero/nginx-example-backup-new-20200508094438
time="2020-05-08T09:44:38Z" level=info msg="Restoring resource 'replicasets.apps' into namespace 'nginx-example'" logSource="pkg/restore/restore.go:777" restore=velero/nginx-example-backup-new-20200508094438
time="2020-05-08T09:44:38Z" level=info msg="Getting client for apps/v1, Kind=ReplicaSet" logSource="pkg/restore/restore.go:821" restore=velero/nginx-example-backup-new-20200508094438
time="2020-05-08T09:44:38Z" level=info msg="Attempting to restore ReplicaSet: nginx-deployment-6fbd5f64c4" logSource="pkg/restore/restore.go:1136" restore=velero/nginx-example-backup-new-20200508094438
time="2020-05-08T09:44:38Z" level=info msg="Restoring resource 'deployments.apps' into namespace 'nginx-example'" logSource="pkg/restore/restore.go:777" restore=velero/nginx-example-backup-new-20200508094438
time="2020-05-08T09:44:38Z" level=info msg="Getting client for apps/v1, Kind=Deployment" logSource="pkg/restore/restore.go:821" restore=velero/nginx-example-backup-new-20200508094438
time="2020-05-08T09:44:38Z" level=info msg="Attempting to restore Deployment: nginx-deployment" logSource="pkg/restore/restore.go:1136" restore=velero/nginx-example-backup-new-20200508094438
time="2020-05-08T09:44:38Z" level=info msg="Restoring resource 'endpoints' into namespace 'nginx-example'" logSource="pkg/restore/restore.go:777" restore=velero/nginx-example-backup-new-20200508094438
time="2020-05-08T09:44:38Z" level=info msg="Getting client for /v1, Kind=Endpoints" logSource="pkg/restore/restore.go:821" restore=velero/nginx-example-backup-new-20200508094438
time="2020-05-08T09:44:38Z" level=info msg="Attempting to restore Endpoints: my-nginx" logSource="pkg/restore/restore.go:1136" restore=velero/nginx-example-backup-new-20200508094438
time="2020-05-08T09:44:38Z" level=info msg="Restoring resource 'services' into namespace 'nginx-example'" logSource="pkg/restore/restore.go:777" restore=velero/nginx-example-backup-new-20200508094438
time="2020-05-08T09:44:38Z" level=info msg="Getting client for /v1, Kind=Service" logSource="pkg/restore/restore.go:821" restore=velero/nginx-example-backup-new-20200508094438
time="2020-05-08T09:44:38Z" level=info msg="Executing item action for services" logSource="pkg/restore/restore.go:1030" restore=velero/nginx-example-backup-new-20200508094438
time="2020-05-08T09:44:38Z" level=info msg="Attempting to restore Service: my-nginx" logSource="pkg/restore/restore.go:1136" restore=velero/nginx-example-backup-new-20200508094438
time="2020-05-08T09:44:41Z" level=info msg="Not including resource" groupResource=events logSource="pkg/restore/restore.go:138" restore=velero/nginx-example-backup-new-20200508094438
time="2020-05-08T09:44:41Z" level=info msg="Not including resource" groupResource=nodes logSource="pkg/restore/restore.go:138" restore=velero/nginx-example-backup-new-20200508094438
time="2020-05-08T09:44:41Z" level=info msg="Not including resource" groupResource=events.events.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/nginx-example-backup-new-20200508094438
time="2020-05-08T09:44:41Z" level=info msg="Not including resource" groupResource=resticrepositories.velero.io logSource="pkg/restore/restore.go:138" restore=velero/nginx-example-backup-new-20200508094438
time="2020-05-08T09:44:41Z" level=info msg="Not including resource" groupResource=restores.velero.io logSource="pkg/restore/restore.go:138" restore=velero/nginx-example-backup-new-20200508094438
time="2020-05-08T09:44:41Z" level=info msg="Not including resource" groupResource=backups.velero.io logSource="pkg/restore/restore.go:138" restore=velero/nginx-example-backup-new-20200508094438
time="2020-05-08T09:44:41Z" level=info msg="Waiting for all restic restores to complete" logSource="pkg/restore/restore.go:545" restore=velero/nginx-example-backup-new-20200508094438

Some resources get restored or attempt to be - but stay in pending state due to the absence of the PV.

kubectl get pods -n nginx-example
NAME                                READY   STATUS    RESTARTS   AGE
nginx-deployment-6fbd5f64c4-fps5c   0/1     Pending   0          5m1s

kubectl get pvc -n nginx-example
NAME         STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS    AGE
nginx-logs   Pending                                      local-storage   5m11s

kubectl get pv | grep -i nginx

As mentioned previously other storage seems to backup and restore ok.

Can anyone shed any light as to what might be happening?

Something Ive noticed - in the getPodVolumeBackups command we have this

path: /host_pods/4640dfa5-910a-11ea-ac8a-028358a66085/volumes/kubernetes.io~local-volume/nginx-logs-volume

but when I exec into the restic pods I cannot see that path or any nginx-example references.

Looking at the restic repo though there is reference to nginx-example volume...


[centos@cva-builder:4.11.4-PR-1712-29 SANDBOX2 CVA-PR-1712 tmp]$ velero restic repo get -o yaml
apiVersion: velero.io/v1
kind: ResticRepository
metadata:
  creationTimestamp: 2020-05-08T09:26:58Z
  generateName: nginx-example-default-
  generation: 3
  labels:
    velero.io/storage-location: default
    velero.io/volume-namespace: nginx-example
  name: nginx-example-default-htvtg
  namespace: velero
  resourceVersion: "308973"
  selfLink: /apis/velero.io/v1/namespaces/velero/resticrepositories/nginx-example-default-htvtg
  uid: 10bf1552-910e-11ea-ac8a-028358a66085
spec:
  backupStorageLocation: default
  maintenanceFrequency: 168h0m0s
  resticIdentifier: s3:s3.amazonaws.com/cva-builder-velero-backups-test/restic/nginx-example
  volumeNamespace: nginx-example
status:
  lastMaintenanceTime: 2020-05-08T09:33:00Z
  phase: Ready

**Environment:**

- Velero version (use `velero version`):   
velero version
Client:
        Version: v1.3.2
        Git commit: 55a9914a3e4719fb1578529c45430a8c11c28145
Server:
        Version: v1.3.2

- Velero features (use `velero client config get features`): 

velero client config get feature
feature: <NOT SET>

- Kubernetes version (use `kubectl version`):

kubectl version
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.8", GitCommit:"0c6d31a99f81476dfc9871ba3cf3f597bec29b58", GitTreeState:"clean", BuildDate:"2019-07-08T08:46:01Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.8", GitCommit:"0c6d31a99f81476dfc9871ba3cf3f597bec29b58", GitTreeState:"clean", BuildDate:"2019-07-08T08:38:54Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}

- Kubernetes installer & version:
- Cloud provider or hardware configuration: **Test performed on AWS - but need "local" storage to emulate on prem none cloud environments**
- OS (e.g. from `/etc/os-release`):

cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7
ashish-amarnath commented 4 years ago

@eddeh83 thank you for the detailed issue! We are going to have to investigate this further. Would you be able to provide info about the size of the restic backup that was being restored? This is going to help us attempt a repro that resembles your workload.

ashish-amarnath commented 4 years ago

Also, if you are able to get a consistent repro of this, can you please run the velero server with debug logs enabled which would give us more information. You will find instructions on how to enable this on our documentation site https://velero.io/docs/v1.4-pre/troubleshooting/#getting-velero-debug-logs

edmund-nelson commented 4 years ago

Hi Ashish

The backup being restored wasn't large - a few hundred MB at most.

Since raising the issue I have managed to complete a restore by using the following method at a high level.

This was with a namespace with multiple PV's and PVC's. oddly when I've just tried on one with only one PV and PVC it worked fine!

Velero log below for the multiple pv namespace

time="2020-05-12T18:40:47Z" level=info msg="starting restore" logSource="pkg/controller/restore_controller.go:458" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Not including resource" groupResource=events logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Not including resource" groupResource=nodes logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Not including resource" groupResource=events.events.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Not including resource" groupResource=restores.velero.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Not including resource" groupResource=resticrepositories.velero.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Not including resource" groupResource=backups.velero.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Starting restore of backup velero/raffia-demo3" logSource="pkg/restore/restore.go:394" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Restoring cluster level resource 'persistentvolumes'" logSource="pkg/restore/restore.go:779" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Getting client for /v1, Kind=PersistentVolume" logSource="pkg/restore/restore.go:821" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Dynamically re-provisioning persistent volume because it has a restic backup to be restored." logSource="pkg/restore/restore.go:992" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Dynamically re-provisioning persistent volume because it has a restic backup to be restored." logSource="pkg/restore/restore.go:992" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Dynamically re-provisioning persistent volume because it has a restic backup to be restored." logSource="pkg/restore/restore.go:992" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Dynamically re-provisioning persistent volume because it has a restic backup to be restored." logSource="pkg/restore/restore.go:992" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Dynamically re-provisioning persistent volume because it has a restic backup to be restored." logSource="pkg/restore/restore.go:992" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Dynamically re-provisioning persistent volume because it has a restic backup to be restored." logSource="pkg/restore/restore.go:992" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Dynamically re-provisioning persistent volume because it has a restic backup to be restored." logSource="pkg/restore/restore.go:992" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Dynamically re-provisioning persistent volume because it has a restic backup to be restored." logSource="pkg/restore/restore.go:992" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Dynamically re-provisioning persistent volume because it has a restic backup to be restored." logSource="pkg/restore/restore.go:992" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Restoring resource 'persistentvolumeclaims' into namespace 'raffia'" logSource="pkg/restore/restore.go:777" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Getting client for /v1, Kind=PersistentVolumeClaim" logSource="pkg/restore/restore.go:821" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Executing item action for persistentvolumeclaims" logSource="pkg/restore/restore.go:1030" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Executing AddPVFromPVCAction" cmd=/velero logSource="pkg/restore/add_pv_from_pvc_action.go:44" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Adding PV raffia-mongodb-data as an additional item to restore" cmd=/velero logSource="pkg/restore/add_pv_from_pvc_action.go:66" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Skipping persistentvolumes/raffia-mongodb-data because it's already been restored." logSource="pkg/restore/restore.go:910" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Executing item action for persistentvolumeclaims" logSource="pkg/restore/restore.go:1030" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:63" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=debug msg="Getting plugin config" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:66" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=debug msg="No storage class mappings found" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:73" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Done executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:74" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Resetting PersistentVolumeClaim raffia/raffia-mongodb for dynamic provisioning because its PV raffia-mongodb-data has a reclaim policy of Delete" logSource="pkg/restore/restore.go:1103" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Attempting to restore PersistentVolumeClaim: raffia-mongodb" logSource="pkg/restore/restore.go:1136" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Executing item action for persistentvolumeclaims" logSource="pkg/restore/restore.go:1030" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Executing AddPVFromPVCAction" cmd=/velero logSource="pkg/restore/add_pv_from_pvc_action.go:44" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Adding PV raffia-config as an additional item to restore" cmd=/velero logSource="pkg/restore/add_pv_from_pvc_action.go:66" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Skipping persistentvolumes/raffia-config because it's already been restored." logSource="pkg/restore/restore.go:910" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Executing item action for persistentvolumeclaims" logSource="pkg/restore/restore.go:1030" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:63" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=debug msg="Getting plugin config" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:66" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=debug msg="No storage class mappings found" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:73" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Done executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:74" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Resetting PersistentVolumeClaim raffia/raffia-raffia-config for dynamic provisioning because its PV raffia-config has a reclaim policy of Delete" logSource="pkg/restore/restore.go:1103" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Attempting to restore PersistentVolumeClaim: raffia-raffia-config" logSource="pkg/restore/restore.go:1136" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Executing item action for persistentvolumeclaims" logSource="pkg/restore/restore.go:1030" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Executing AddPVFromPVCAction" cmd=/velero logSource="pkg/restore/add_pv_from_pvc_action.go:44" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Adding PV raffia-domains as an additional item to restore" cmd=/velero logSource="pkg/restore/add_pv_from_pvc_action.go:66" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Skipping persistentvolumes/raffia-domains because it's already been restored." logSource="pkg/restore/restore.go:910" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Executing item action for persistentvolumeclaims" logSource="pkg/restore/restore.go:1030" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:63" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=debug msg="Getting plugin config" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:66" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=debug msg="No storage class mappings found" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:73" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Done executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:74" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Resetting PersistentVolumeClaim raffia/raffia-raffia-domains for dynamic provisioning because its PV raffia-domains has a reclaim policy of Delete" logSource="pkg/restore/restore.go:1103" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Attempting to restore PersistentVolumeClaim: raffia-raffia-domains" logSource="pkg/restore/restore.go:1136" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Executing item action for persistentvolumeclaims" logSource="pkg/restore/restore.go:1030" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Executing AddPVFromPVCAction" cmd=/velero logSource="pkg/restore/add_pv_from_pvc_action.go:44" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Adding PV raffia-engine-config as an additional item to restore" cmd=/velero logSource="pkg/restore/add_pv_from_pvc_action.go:66" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Skipping persistentvolumes/raffia-engine-config because it's already been restored." logSource="pkg/restore/restore.go:910" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Executing item action for persistentvolumeclaims" logSource="pkg/restore/restore.go:1030" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:63" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=debug msg="Getting plugin config" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:66" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=debug msg="No storage class mappings found" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:73" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Done executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:74" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Resetting PersistentVolumeClaim raffia/raffia-raffia-engine-config for dynamic provisioning because its PV raffia-engine-config has a reclaim policy of Delete" logSource="pkg/restore/restore.go:1103" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Attempting to restore PersistentVolumeClaim: raffia-raffia-engine-config" logSource="pkg/restore/restore.go:1136" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Executing item action for persistentvolumeclaims" logSource="pkg/restore/restore.go:1030" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Executing AddPVFromPVCAction" cmd=/velero logSource="pkg/restore/add_pv_from_pvc_action.go:44" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Adding PV raffia-engine-home as an additional item to restore" cmd=/velero logSource="pkg/restore/add_pv_from_pvc_action.go:66" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Skipping persistentvolumes/raffia-engine-home because it's already been restored." logSource="pkg/restore/restore.go:910" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Executing item action for persistentvolumeclaims" logSource="pkg/restore/restore.go:1030" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:63" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=debug msg="Getting plugin config" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:66" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=debug msg="No storage class mappings found" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:73" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Done executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:74" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Resetting PersistentVolumeClaim raffia/raffia-raffia-engine-home for dynamic provisioning because its PV raffia-engine-home has a reclaim policy of Delete" logSource="pkg/restore/restore.go:1103" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Attempting to restore PersistentVolumeClaim: raffia-raffia-engine-home" logSource="pkg/restore/restore.go:1136" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Executing item action for persistentvolumeclaims" logSource="pkg/restore/restore.go:1030" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Executing AddPVFromPVCAction" cmd=/velero logSource="pkg/restore/add_pv_from_pvc_action.go:44" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Adding PV raffia-engine-triage as an additional item to restore" cmd=/velero logSource="pkg/restore/add_pv_from_pvc_action.go:66" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Skipping persistentvolumes/raffia-engine-triage because it's already been restored." logSource="pkg/restore/restore.go:910" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Executing item action for persistentvolumeclaims" logSource="pkg/restore/restore.go:1030" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:63" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=debug msg="Getting plugin config" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:66" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=debug msg="No storage class mappings found" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:73" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Done executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:74" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Resetting PersistentVolumeClaim raffia/raffia-raffia-engine-triage for dynamic provisioning because its PV raffia-engine-triage has a reclaim policy of Delete" logSource="pkg/restore/restore.go:1103" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Attempting to restore PersistentVolumeClaim: raffia-raffia-engine-triage" logSource="pkg/restore/restore.go:1136" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Executing item action for persistentvolumeclaims" logSource="pkg/restore/restore.go:1030" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Executing AddPVFromPVCAction" cmd=/velero logSource="pkg/restore/add_pv_from_pvc_action.go:44" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Adding PV raffia-engine-var as an additional item to restore" cmd=/velero logSource="pkg/restore/add_pv_from_pvc_action.go:66" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Skipping persistentvolumes/raffia-engine-var because it's already been restored." logSource="pkg/restore/restore.go:910" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Executing item action for persistentvolumeclaims" logSource="pkg/restore/restore.go:1030" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:63" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=debug msg="Getting plugin config" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:66" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=debug msg="No storage class mappings found" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:73" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Done executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:74" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Resetting PersistentVolumeClaim raffia/raffia-raffia-engine-var for dynamic provisioning because its PV raffia-engine-var has a reclaim policy of Delete" logSource="pkg/restore/restore.go:1103" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Attempting to restore PersistentVolumeClaim: raffia-raffia-engine-var" logSource="pkg/restore/restore.go:1136" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Executing item action for persistentvolumeclaims" logSource="pkg/restore/restore.go:1030" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Executing AddPVFromPVCAction" cmd=/velero logSource="pkg/restore/add_pv_from_pvc_action.go:44" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Adding PV raffia-home as an additional item to restore" cmd=/velero logSource="pkg/restore/add_pv_from_pvc_action.go:66" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Skipping persistentvolumes/raffia-home because it's already been restored." logSource="pkg/restore/restore.go:910" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Executing item action for persistentvolumeclaims" logSource="pkg/restore/restore.go:1030" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:63" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=debug msg="Getting plugin config" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:66" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=debug msg="No storage class mappings found" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:73" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Done executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:74" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Resetting PersistentVolumeClaim raffia/raffia-raffia-home for dynamic provisioning because its PV raffia-home has a reclaim policy of Delete" logSource="pkg/restore/restore.go:1103" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Attempting to restore PersistentVolumeClaim: raffia-raffia-home" logSource="pkg/restore/restore.go:1136" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Executing item action for persistentvolumeclaims" logSource="pkg/restore/restore.go:1030" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Executing AddPVFromPVCAction" cmd=/velero logSource="pkg/restore/add_pv_from_pvc_action.go:44" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Adding PV raffia-raffia-var as an additional item to restore" cmd=/velero logSource="pkg/restore/add_pv_from_pvc_action.go:66" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Skipping persistentvolumes/raffia-raffia-var because it's already been restored." logSource="pkg/restore/restore.go:910" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Executing item action for persistentvolumeclaims" logSource="pkg/restore/restore.go:1030" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:63" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=debug msg="Getting plugin config" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:66" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=debug msg="No storage class mappings found" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:73" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Done executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:74" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Resetting PersistentVolumeClaim raffia/raffia-raffia-var for dynamic provisioning because its PV raffia-raffia-var has a reclaim policy of Delete" logSource="pkg/restore/restore.go:1103" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Attempting to restore PersistentVolumeClaim: raffia-raffia-var" logSource="pkg/restore/restore.go:1136" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Restoring resource 'secrets' into namespace 'raffia'" logSource="pkg/restore/restore.go:777" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Getting client for /v1, Kind=Secret" logSource="pkg/restore/restore.go:821" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Attempting to restore Secret: cva-id" logSource="pkg/restore/restore.go:1136" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Attempting to restore Secret: cva-info" logSource="pkg/restore/restore.go:1136" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Attempting to restore Secret: default-token-rkbd4" logSource="pkg/restore/restore.go:1136" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Attempting to restore Secret: dxcdockerslmcred" logSource="pkg/restore/restore.go:1136" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Attempting to restore Secret: raffia-cert-vault-approle" logSource="pkg/restore/restore.go:1136" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Attempting to restore Secret: raffia-deploy-vault-approle" logSource="pkg/restore/restore.go:1136" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Attempting to restore Secret: raffia-ephemeral-path" logSource="pkg/restore/restore.go:1136" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Attempting to restore Secret: raffia-mongodb" logSource="pkg/restore/restore.go:1136" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Attempting to restore Secret: raffia-sa-token-j69gm" logSource="pkg/restore/restore.go:1136" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Attempting to restore Secret: raffia-vault-approle" logSource="pkg/restore/restore.go:1136" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Attempting to restore Secret: vault-client-tls" logSource="pkg/restore/restore.go:1136" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Restoring resource 'configmaps' into namespace 'raffia'" logSource="pkg/restore/restore.go:777" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Getting client for /v1, Kind=ConfigMap" logSource="pkg/restore/restore.go:821" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Attempting to restore ConfigMap: raffia-raffia-engine-scripts" logSource="pkg/restore/restore.go:1136" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Attempting to restore ConfigMap: raffia-raffia-scripts" logSource="pkg/restore/restore.go:1136" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Restoring resource 'serviceaccounts' into namespace 'raffia'" logSource="pkg/restore/restore.go:777" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Getting client for /v1, Kind=ServiceAccount" logSource="pkg/restore/restore.go:821" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Executing item action for serviceaccounts" logSource="pkg/restore/restore.go:1030" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Executing ServiceAccountAction" cmd=/velero logSource="pkg/restore/service_account_action.go:47" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=debug msg="Checking secrets" cmd=/velero logSource="pkg/restore/service_account_action.go:57" pluginName=velero restore=velero/raffia-demo3-20200512184046 serviceaccount=raffia/default
time="2020-05-12T18:40:47Z" level=debug msg="Checking if secret default-token-rkbd4 matches default-token-" cmd=/velero logSource="pkg/restore/service_account_action.go:61" pluginName=velero restore=velero/raffia-demo3-20200512184046 serviceaccount=raffia/default
time="2020-05-12T18:40:47Z" level=debug msg="Match found - excluding this secret" cmd=/velero logSource="pkg/restore/service_account_action.go:65" pluginName=velero restore=velero/raffia-demo3-20200512184046 serviceaccount=raffia/default
time="2020-05-12T18:40:47Z" level=info msg="Done executing ServiceAccountAction" cmd=/velero logSource="pkg/restore/service_account_action.go:78" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Attempting to restore ServiceAccount: default" logSource="pkg/restore/restore.go:1136" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Executing item action for serviceaccounts" logSource="pkg/restore/restore.go:1030" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Executing ServiceAccountAction" cmd=/velero logSource="pkg/restore/service_account_action.go:47" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=debug msg="Checking secrets" cmd=/velero logSource="pkg/restore/service_account_action.go:57" pluginName=velero restore=velero/raffia-demo3-20200512184046 serviceaccount=raffia/raffia-sa
time="2020-05-12T18:40:47Z" level=debug msg="Checking if secret raffia-sa-token-j69gm matches raffia-sa-token-" cmd=/velero logSource="pkg/restore/service_account_action.go:61" pluginName=velero restore=velero/raffia-demo3-20200512184046 serviceaccount=raffia/raffia-sa
time="2020-05-12T18:40:47Z" level=debug msg="Match found - excluding this secret" cmd=/velero logSource="pkg/restore/service_account_action.go:65" pluginName=velero restore=velero/raffia-demo3-20200512184046 serviceaccount=raffia/raffia-sa
time="2020-05-12T18:40:47Z" level=info msg="Done executing ServiceAccountAction" cmd=/velero logSource="pkg/restore/service_account_action.go:78" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Attempting to restore ServiceAccount: raffia-sa" logSource="pkg/restore/restore.go:1136" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Restoring resource 'pods' into namespace 'raffia'" logSource="pkg/restore/restore.go:777" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Getting client for /v1, Kind=Pod" logSource="pkg/restore/restore.go:821" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Executing item action for pods" logSource="pkg/restore/restore.go:1030" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Executing AddPVCFromPodAction" cmd=/velero logSource="pkg/restore/add_pvc_from_pod_action.go:44" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Adding PVC raffia/raffia-mongodb as an additional item to restore" cmd=/velero logSource="pkg/restore/add_pvc_from_pod_action.go:58" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Skipping persistentvolumeclaims/raffia/raffia-mongodb because it's already been restored." logSource="pkg/restore/restore.go:910" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Executing item action for pods" logSource="pkg/restore/restore.go:1030" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Executing item action for pods" logSource="pkg/restore/restore.go:1030" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Executing ResticRestoreAction" cmd=/velero logSource="pkg/restore/restic_restore_action.go:68" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Restic backups for pod found" cmd=/velero logSource="pkg/restore/restic_restore_action.go:94" pluginName=velero pod=raffia/raffia-mongodb-54ccbdcf87-c9b6m restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=debug msg="Getting plugin config" cmd=/velero logSource="pkg/restore/restic_restore_action.go:98" pluginName=velero pod=raffia/raffia-mongodb-54ccbdcf87-c9b6m restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=debug msg="No config found for plugin" cmd=/velero logSource="pkg/restore/restic_restore_action.go:159" pluginName=velero pod=raffia/raffia-mongodb-54ccbdcf87-c9b6m restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Using image \"velero/velero-restic-restore-helper:v1.3.1\"" cmd=/velero logSource="pkg/restore/restic_restore_action.go:105" pluginName=velero pod=raffia/raffia-mongodb-54ccbdcf87-c9b6m restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=debug msg="No config found for plugin" cmd=/velero logSource="pkg/restore/restic_restore_action.go:194" pluginName=velero pod=raffia/raffia-mongodb-54ccbdcf87-c9b6m restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=debug msg="No config found for plugin" cmd=/velero logSource="pkg/restore/restic_restore_action.go:205" pluginName=velero pod=raffia/raffia-mongodb-54ccbdcf87-c9b6m restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Done executing ResticRestoreAction" cmd=/velero logSource="pkg/restore/restic_restore_action.go:154" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Attempting to restore Pod: raffia-mongodb-54ccbdcf87-c9b6m" logSource="pkg/restore/restore.go:1136" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Executing item action for pods" logSource="pkg/restore/restore.go:1030" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Executing AddPVCFromPodAction" cmd=/velero logSource="pkg/restore/add_pvc_from_pod_action.go:44" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=debug msg="Acquiring lock" backupLocation=default logSource="pkg/restic/repository_ensurer.go:122" volumeNamespace=raffia
time="2020-05-12T18:40:47Z" level=debug msg="Acquired lock" backupLocation=default logSource="pkg/restic/repository_ensurer.go:131" volumeNamespace=raffia
time="2020-05-12T18:40:47Z" level=debug msg="Ready repository found" backupLocation=default logSource="pkg/restic/repository_ensurer.go:147" volumeNamespace=raffia
time="2020-05-12T18:40:47Z" level=debug msg="Released lock" backupLocation=default logSource="pkg/restic/repository_ensurer.go:128" volumeNamespace=raffia
time="2020-05-12T18:40:47Z" level=info msg="Adding PVC raffia/raffia-raffia-config as an additional item to restore" cmd=/velero logSource="pkg/restore/add_pvc_from_pod_action.go:58" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Adding PVC raffia/raffia-raffia-domains as an additional item to restore" cmd=/velero logSource="pkg/restore/add_pvc_from_pod_action.go:58" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Adding PVC raffia/raffia-raffia-var as an additional item to restore" cmd=/velero logSource="pkg/restore/add_pvc_from_pod_action.go:58" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Adding PVC raffia/raffia-raffia-home as an additional item to restore" cmd=/velero logSource="pkg/restore/add_pvc_from_pod_action.go:58" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Skipping persistentvolumeclaims/raffia/raffia-raffia-config because it's already been restored." logSource="pkg/restore/restore.go:910" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Skipping persistentvolumeclaims/raffia/raffia-raffia-domains because it's already been restored." logSource="pkg/restore/restore.go:910" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Skipping persistentvolumeclaims/raffia/raffia-raffia-var because it's already been restored." logSource="pkg/restore/restore.go:910" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Skipping persistentvolumeclaims/raffia/raffia-raffia-home because it's already been restored." logSource="pkg/restore/restore.go:910" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Executing item action for pods" logSource="pkg/restore/restore.go:1030" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Executing item action for pods" logSource="pkg/restore/restore.go:1030" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Executing ResticRestoreAction" cmd=/velero logSource="pkg/restore/restic_restore_action.go:68" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Restic backups for pod found" cmd=/velero logSource="pkg/restore/restic_restore_action.go:94" pluginName=velero pod=raffia/raffia-raffia-5667cc7768-5cn64 restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=debug msg="Getting plugin config" cmd=/velero logSource="pkg/restore/restic_restore_action.go:98" pluginName=velero pod=raffia/raffia-raffia-5667cc7768-5cn64 restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=debug msg="No config found for plugin" cmd=/velero logSource="pkg/restore/restic_restore_action.go:159" pluginName=velero pod=raffia/raffia-raffia-5667cc7768-5cn64 restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Using image \"velero/velero-restic-restore-helper:v1.3.1\"" cmd=/velero logSource="pkg/restore/restic_restore_action.go:105" pluginName=velero pod=raffia/raffia-raffia-5667cc7768-5cn64 restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=debug msg="No config found for plugin" cmd=/velero logSource="pkg/restore/restic_restore_action.go:194" pluginName=velero pod=raffia/raffia-raffia-5667cc7768-5cn64 restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=debug msg="No config found for plugin" cmd=/velero logSource="pkg/restore/restic_restore_action.go:205" pluginName=velero pod=raffia/raffia-raffia-5667cc7768-5cn64 restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Done executing ResticRestoreAction" cmd=/velero logSource="pkg/restore/restic_restore_action.go:154" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Attempting to restore Pod: raffia-raffia-5667cc7768-5cn64" logSource="pkg/restore/restore.go:1136" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Executing item action for pods" logSource="pkg/restore/restore.go:1030" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=debug msg="Acquiring lock" backupLocation=default logSource="pkg/restic/repository_ensurer.go:122" volumeNamespace=raffia
time="2020-05-12T18:40:47Z" level=debug msg="Acquired lock" backupLocation=default logSource="pkg/restic/repository_ensurer.go:131" volumeNamespace=raffia
time="2020-05-12T18:40:47Z" level=debug msg="Ready repository found" backupLocation=default logSource="pkg/restic/repository_ensurer.go:147" volumeNamespace=raffia
time="2020-05-12T18:40:47Z" level=debug msg="Released lock" backupLocation=default logSource="pkg/restic/repository_ensurer.go:128" volumeNamespace=raffia
time="2020-05-12T18:40:47Z" level=info msg="Executing AddPVCFromPodAction" cmd=/velero logSource="pkg/restore/add_pvc_from_pod_action.go:44" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Adding PVC raffia/raffia-raffia-engine-config as an additional item to restore" cmd=/velero logSource="pkg/restore/add_pvc_from_pod_action.go:58" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Adding PVC raffia/raffia-raffia-engine-var as an additional item to restore" cmd=/velero logSource="pkg/restore/add_pvc_from_pod_action.go:58" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Adding PVC raffia/raffia-raffia-engine-home as an additional item to restore" cmd=/velero logSource="pkg/restore/add_pvc_from_pod_action.go:58" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Adding PVC raffia/raffia-raffia-engine-triage as an additional item to restore" cmd=/velero logSource="pkg/restore/add_pvc_from_pod_action.go:58" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Skipping persistentvolumeclaims/raffia/raffia-raffia-engine-config because it's already been restored." logSource="pkg/restore/restore.go:910" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Skipping persistentvolumeclaims/raffia/raffia-raffia-engine-var because it's already been restored." logSource="pkg/restore/restore.go:910" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Skipping persistentvolumeclaims/raffia/raffia-raffia-engine-home because it's already been restored." logSource="pkg/restore/restore.go:910" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Skipping persistentvolumeclaims/raffia/raffia-raffia-engine-triage because it's already been restored." logSource="pkg/restore/restore.go:910" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Executing item action for pods" logSource="pkg/restore/restore.go:1030" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Executing item action for pods" logSource="pkg/restore/restore.go:1030" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Executing ResticRestoreAction" cmd=/velero logSource="pkg/restore/restic_restore_action.go:68" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Restic backups for pod found" cmd=/velero logSource="pkg/restore/restic_restore_action.go:94" pluginName=velero pod=raffia/raffia-raffia-engine-7b665ddbb4-fq8zv restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=debug msg="Getting plugin config" cmd=/velero logSource="pkg/restore/restic_restore_action.go:98" pluginName=velero pod=raffia/raffia-raffia-engine-7b665ddbb4-fq8zv restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=debug msg="No config found for plugin" cmd=/velero logSource="pkg/restore/restic_restore_action.go:159" pluginName=velero pod=raffia/raffia-raffia-engine-7b665ddbb4-fq8zv restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Using image \"velero/velero-restic-restore-helper:v1.3.1\"" cmd=/velero logSource="pkg/restore/restic_restore_action.go:105" pluginName=velero pod=raffia/raffia-raffia-engine-7b665ddbb4-fq8zv restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=debug msg="No config found for plugin" cmd=/velero logSource="pkg/restore/restic_restore_action.go:194" pluginName=velero pod=raffia/raffia-raffia-engine-7b665ddbb4-fq8zv restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=debug msg="No config found for plugin" cmd=/velero logSource="pkg/restore/restic_restore_action.go:205" pluginName=velero pod=raffia/raffia-raffia-engine-7b665ddbb4-fq8zv restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Done executing ResticRestoreAction" cmd=/velero logSource="pkg/restore/restic_restore_action.go:154" pluginName=velero restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Attempting to restore Pod: raffia-raffia-engine-7b665ddbb4-fq8zv" logSource="pkg/restore/restore.go:1136" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Restoring resource 'replicasets.apps' into namespace 'raffia'" logSource="pkg/restore/restore.go:777" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Getting client for apps/v1, Kind=ReplicaSet" logSource="pkg/restore/restore.go:821" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Attempting to restore ReplicaSet: raffia-mongodb-54ccbdcf87" logSource="pkg/restore/restore.go:1136" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=debug msg="Acquiring lock" backupLocation=default logSource="pkg/restic/repository_ensurer.go:122" volumeNamespace=raffia
time="2020-05-12T18:40:47Z" level=debug msg="Acquired lock" backupLocation=default logSource="pkg/restic/repository_ensurer.go:131" volumeNamespace=raffia
time="2020-05-12T18:40:47Z" level=debug msg="Ready repository found" backupLocation=default logSource="pkg/restic/repository_ensurer.go:147" volumeNamespace=raffia
time="2020-05-12T18:40:47Z" level=debug msg="Released lock" backupLocation=default logSource="pkg/restic/repository_ensurer.go:128" volumeNamespace=raffia
time="2020-05-12T18:40:47Z" level=info msg="Attempting to restore ReplicaSet: raffia-raffia-5667cc7768" logSource="pkg/restore/restore.go:1136" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Attempting to restore ReplicaSet: raffia-raffia-engine-7b665ddbb4" logSource="pkg/restore/restore.go:1136" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Restoring resource 'deployments.apps' into namespace 'raffia'" logSource="pkg/restore/restore.go:777" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Getting client for apps/v1, Kind=Deployment" logSource="pkg/restore/restore.go:821" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Attempting to restore Deployment: raffia-mongodb" logSource="pkg/restore/restore.go:1136" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Attempting to restore Deployment: raffia-raffia-engine" logSource="pkg/restore/restore.go:1136" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Attempting to restore Deployment: raffia-raffia" logSource="pkg/restore/restore.go:1136" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Restoring resource 'endpoints' into namespace 'raffia'" logSource="pkg/restore/restore.go:777" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Getting client for /v1, Kind=Endpoints" logSource="pkg/restore/restore.go:821" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Attempting to restore Endpoints: raffia-mongodb" logSource="pkg/restore/restore.go:1136" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:47Z" level=info msg="Attempting to restore Endpoints: raffia-raffia" logSource="pkg/restore/restore.go:1136" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:48Z" level=info msg="Restoring resource 'ingresses.extensions' into namespace 'raffia'" logSource="pkg/restore/restore.go:777" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:48Z" level=info msg="Getting client for extensions/v1beta1, Kind=Ingress" logSource="pkg/restore/restore.go:821" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:48Z" level=info msg="Attempting to restore Ingress: raffia-raffia" logSource="pkg/restore/restore.go:1136" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:48Z" level=info msg="Restoring resource 'rolebindings.rbac.authorization.k8s.io' into namespace 'raffia'" logSource="pkg/restore/restore.go:777" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:48Z" level=info msg="Getting client for rbac.authorization.k8s.io/v1, Kind=RoleBinding" logSource="pkg/restore/restore.go:821" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:48Z" level=info msg="Executing item action for rolebindings.rbac.authorization.k8s.io" logSource="pkg/restore/restore.go:1030" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:48Z" level=info msg="Attempting to restore RoleBinding: raffia-read-only-role-binding" logSource="pkg/restore/restore.go:1136" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:48Z" level=info msg="Restoring resource 'roles.rbac.authorization.k8s.io' into namespace 'raffia'" logSource="pkg/restore/restore.go:777" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:48Z" level=info msg="Getting client for rbac.authorization.k8s.io/v1, Kind=Role" logSource="pkg/restore/restore.go:821" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:48Z" level=info msg="Attempting to restore Role: raffia-role-read-only" logSource="pkg/restore/restore.go:1136" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:48Z" level=info msg="Restoring resource 'services' into namespace 'raffia'" logSource="pkg/restore/restore.go:777" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:48Z" level=info msg="Getting client for /v1, Kind=Service" logSource="pkg/restore/restore.go:821" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:48Z" level=info msg="Executing item action for services" logSource="pkg/restore/restore.go:1030" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:48Z" level=info msg="Attempting to restore Service: raffia-mongodb" logSource="pkg/restore/restore.go:1136" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:48Z" level=info msg="Executing item action for services" logSource="pkg/restore/restore.go:1030" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:48Z" level=info msg="Attempting to restore Service: raffia-raffia" logSource="pkg/restore/restore.go:1136" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:51Z" level=info msg="Not including resource" groupResource=events logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:51Z" level=info msg="Not including resource" groupResource=nodes logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:51Z" level=info msg="Not including resource" groupResource=events.events.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:51Z" level=info msg="Not including resource" groupResource=backups.velero.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:51Z" level=info msg="Not including resource" groupResource=resticrepositories.velero.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:51Z" level=info msg="Not including resource" groupResource=restores.velero.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:51Z" level=info msg="Waiting for all restic restores to complete" logSource="pkg/restore/restore.go:545" restore=velero/raffia-demo3-20200512184046
time="2020-05-12T18:40:59Z" level=debug msg="Checking for existing backup storage locations to sync into cluster" controller=backup-sync logSource="pkg/controller/backup_sync_controller.go:124"
time="2020-05-12T18:40:59Z" level=debug msg="Checking if backups need to be synced at this time for this location" backupLocation=default controller=backup-sync logSource="pkg/controller/backup_sync_controller.go:156"
time="2020-05-12T18:40:59Z" level=debug msg="Checking backup location for backups to sync into cluster" backupLocation=default controller=backup-sync logSource="pkg/controller/backup_sync_controller.go:163"
time="2020-05-12T18:40:59Z" level=debug msg="looking for plugin in registry" controller=backup-sync kind=ObjectStore logSource="pkg/plugin/clientmgmt/manager.go:99" name=velero.io/aws
time="2020-05-12T18:40:59Z" level=debug msg="creating new restartable plugin process" command=/plugins/velero-plugin-for-aws controller=backup-sync kind=ObjectStore logSource="pkg/plugin/clientmgmt/manager.go:114" name=velero.io/aws
time="2020-05-12T18:40:59Z" level=debug msg="starting plugin" args="[/plugins/velero-plugin-for-aws --log-level debug]" cmd=/plugins/velero-plugin-for-aws controller=backup-sync logSource="pkg/plugin/clientmgmt/logrus_adapter.go:74" path=/plugins/velero-plugin-for-aws
time="2020-05-12T18:40:59Z" level=debug msg="plugin started" cmd=/plugins/velero-plugin-for-aws controller=backup-sync logSource="pkg/plugin/clientmgmt/logrus_adapter.go:74" path=/plugins/velero-plugin-for-aws pid=1863
time="2020-05-12T18:40:59Z" level=debug msg="waiting for RPC address" cmd=/plugins/velero-plugin-for-aws controller=backup-sync logSource="pkg/plugin/clientmgmt/logrus_adapter.go:74" path=/plugins/velero-plugin-for-aws
time="2020-05-12T18:40:59Z" level=debug msg="using plugin" cmd=/plugins/velero-plugin-for-aws controller=backup-sync logSource="pkg/plugin/clientmgmt/logrus_adapter.go:74" version=2
time="2020-05-12T18:40:59Z" level=debug msg="plugin address" address=/tmp/plugin788120112 cmd=/plugins/velero-plugin-for-aws controller=backup-sync logSource="pkg/plugin/clientmgmt/logrus_adapter.go:74" network=unix pluginName=velero-plugin-for-aws
time="2020-05-12T18:40:59Z" level=debug msg="Got backups from backup store" backupCount=16 backupLocation=default controller=backup-sync logSource="pkg/controller/backup_sync_controller.go:178"
time="2020-05-12T18:40:59Z" level=debug msg="Got backups from cluster" backupCount=16 backupLocation=default controller=backup-sync logSource="pkg/controller/backup_sync_controller.go:185"
time="2020-05-12T18:40:59Z" level=debug msg="No backups found in the backup location that need to be synced into the cluster" backupLocation=default controller=backup-sync logSource="pkg/controller/backup_sync_controller.go:198"
time="2020-05-12T18:40:59Z" level=debug msg="plugin process exited" cmd=/plugins/velero-plugin-for-aws controller=backup-sync logSource="pkg/plugin/clientmgmt/logrus_adapter.go:74" path=/plugins/velero-plugin-for-aws pid=1863
time="2020-05-12T18:40:59Z" level=debug msg="plugin exited" cmd=/plugins/velero-plugin-for-aws controller=backup-sync logSource="pkg/plugin/clientmgmt/logrus_adapter.go:74"
time="2020-05-12T18:41:29Z" level=debug msg="Checking for existing backup storage locations to sync into cluster" controller=backup-sync logSource="pkg/controller/backup_sync_controller.go:124"
time="2020-05-12T18:41:29Z" level=debug msg="Checking if backups need to be synced at this time for this location" backupLocation=default controller=backup-sync logSource="pkg/controller/backup_sync_controller.go:156"
time="2020-05-12T18:41:59Z" level=debug msg="Checking for existing backup storage locations to sync into cluster" controller=backup-sync logSource="pkg/controller/backup_sync_controller.go:124"
time="2020-05-12T18:41:59Z" level=debug msg="Checking if backups need to be synced at this time for this location" backupLocation=default controller=backup-sync logSource="pkg/controller/backup_sync_controller.go:156"
time="2020-05-12T18:41:59Z" level=debug msg="Checking backup location for backups to sync into cluster" backupLocation=default controller=backup-sync logSource="pkg/controller/backup_sync_controller.go:163"
time="2020-05-12T18:41:59Z" level=debug msg="looking for plugin in registry" controller=backup-sync kind=ObjectStore logSource="pkg/plugin/clientmgmt/manager.go:99" name=velero.io/aws
time="2020-05-12T18:41:59Z" level=debug msg="creating new restartable plugin process" command=/plugins/velero-plugin-for-aws controller=backup-sync kind=ObjectStore logSource="pkg/plugin/clientmgmt/manager.go:114" name=velero.io/aws
time="2020-05-12T18:41:59Z" level=debug msg="starting plugin" args="[/plugins/velero-plugin-for-aws --log-level debug]" cmd=/plugins/velero-plugin-for-aws controller=backup-sync logSource="pkg/plugin/clientmgmt/logrus_adapter.go:74" path=/plugins/velero-plugin-for-aws
time="2020-05-12T18:41:59Z" level=debug msg="plugin started" cmd=/plugins/velero-plugin-for-aws controller=backup-sync logSource="pkg/plugin/clientmgmt/logrus_adapter.go:74" path=/plugins/velero-plugin-for-aws pid=1880
time="2020-05-12T18:41:59Z" level=debug msg="waiting for RPC address" cmd=/plugins/velero-plugin-for-aws controller=backup-sync logSource="pkg/plugin/clientmgmt/logrus_adapter.go:74" path=/plugins/velero-plugin-for-aws
time="2020-05-12T18:41:59Z" level=debug msg="plugin address" address=/tmp/plugin575175281 cmd=/plugins/velero-plugin-for-aws controller=backup-sync logSource="pkg/plugin/clientmgmt/logrus_adapter.go:74" network=unix pluginName=velero-plugin-for-aws
time="2020-05-12T18:41:59Z" level=debug msg="using plugin" cmd=/plugins/velero-plugin-for-aws controller=backup-sync logSource="pkg/plugin/clientmgmt/logrus_adapter.go:74" version=2
time="2020-05-12T18:41:59Z" level=debug msg="Got backups from backup store" backupCount=16 backupLocation=default controller=backup-sync logSource="pkg/controller/backup_sync_controller.go:178"
time="2020-05-12T18:41:59Z" level=debug msg="Got backups from cluster" backupCount=16 backupLocation=default controller=backup-sync logSource="pkg/controller/backup_sync_controller.go:185"
time="2020-05-12T18:41:59Z" level=debug msg="No backups found in the backup location that need to be synced into the cluster" backupLocation=default controller=backup-sync logSource="pkg/controller/backup_sync_controller.go:198"
time="2020-05-12T18:41:59Z" level=debug msg="plugin process exited" cmd=/plugins/velero-plugin-for-aws controller=backup-sync logSource="pkg/plugin/clientmgmt/logrus_adapter.go:74" path=/plugins/velero-plugin-for-aws pid=1880
time="2020-05-12T18:41:59Z" level=debug msg="plugin exited" cmd=/plugins/velero-plugin-for-aws controller=backup-sync logSource="pkg/plugin/clientmgmt/logrus_adapter.go:74"
time="2020-05-12T18:42:19Z" level=debug msg=resticRepositoryController.enqueueAllRepositories controller=restic-repository logSource="pkg/controller/restic_repository_controller.go:96"
time="2020-05-12T18:42:19Z" level=debug msg="Running processQueueItem" controller=restic-repository key=velero/vault-default-4nhjh logSource="pkg/controller/restic_repository_controller.go:111"
time="2020-05-12T18:42:19Z" level=debug msg="Checking repository for stale locks" controller=restic-repository logSource="pkg/controller/restic_repository_controller.go:140" name=vault-default-4nhjh namespace=velero
time="2020-05-12T18:42:29Z" level=debug msg="Checking for existing backup storage locations to sync into cluster" controller=backup-sync logSource="pkg/controller/backup_sync_controller.go:124"
time="2020-05-12T18:42:29Z" level=debug msg="Checking if backups need to be synced at this time for this location" backupLocation=default controller=backup-sync logSource="pkg/controller/backup_sync_controller.go:156"
time="2020-05-12T18:42:46Z" level=debug msg="Running processDownloadRequest" controller=downloadrequest key=velero/nginx-example-restic-test3-20200512184246 logSource="pkg/controller/download_request_controller.go:114"
time="2020-05-12T18:42:46Z" level=debug msg="looking for plugin in registry" controller=downloadrequest key=velero/nginx-example-restic-test3-20200512184246 kind=ObjectStore logSource="pkg/plugin/clientmgmt/manager.go:99" name=velero.io/aws
time="2020-05-12T18:42:46Z" level=debug msg="creating new restartable plugin process" command=/plugins/velero-plugin-for-aws controller=downloadrequest key=velero/nginx-example-restic-test3-20200512184246 kind=ObjectStore logSource="pkg/plugin/clientmgmt/manager.go:114" name=velero.io/aws
time="2020-05-12T18:42:46Z" level=debug msg="starting plugin" args="[/plugins/velero-plugin-for-aws --log-level debug]" cmd=/plugins/velero-plugin-for-aws controller=downloadrequest key=velero/nginx-example-restic-test3-20200512184246 logSource="pkg/plugin/clientmgmt/logrus_adapter.go:74" path=/plugins/velero-plugin-for-aws
time="2020-05-12T18:42:46Z" level=debug msg="plugin started" cmd=/plugins/velero-plugin-for-aws controller=downloadrequest key=velero/nginx-example-restic-test3-20200512184246 logSource="pkg/plugin/clientmgmt/logrus_adapter.go:74" path=/plugins/velero-plugin-for-aws pid=1911
time="2020-05-12T18:42:46Z" level=debug msg="waiting for RPC address" cmd=/plugins/velero-plugin-for-aws controller=downloadrequest key=velero/nginx-example-restic-test3-20200512184246 logSource="pkg/plugin/clientmgmt/logrus_adapter.go:74" path=/plugins/velero-plugin-for-aws
time="2020-05-12T18:42:46Z" level=debug msg="using plugin" cmd=/plugins/velero-plugin-for-aws controller=downloadrequest key=velero/nginx-example-restic-test3-20200512184246 logSource="pkg/plugin/clientmgmt/logrus_adapter.go:74" version=2
time="2020-05-12T18:42:46Z" level=debug msg="plugin address" address=/tmp/plugin732159865 cmd=/plugins/velero-plugin-for-aws controller=downloadrequest key=velero/nginx-example-restic-test3-20200512184246 logSource="pkg/plugin/clientmgmt/logrus_adapter.go:74" network=unix pluginName=velero-plugin-for-aws
time="2020-05-12T18:42:46Z" level=debug msg="plugin process exited" cmd=/plugins/velero-plugin-for-aws controller=downloadrequest key=velero/nginx-example-restic-test3-20200512184246 logSource="pkg/plugin/clientmgmt/logrus_adapter.go:74" path=/plugins/velero-plugin-for-aws pid=1911
time="2020-05-12T18:42:46Z" level=debug msg="plugin exited" cmd=/plugins/velero-plugin-for-aws controller=downloadrequest key=velero/nginx-example-restic-test3-20200512184246 logSource="pkg/plugin/clientmgmt/logrus_adapter.go:74"
time="2020-05-12T18:42:50Z" level=debug msg="Ran restic command" command="restic unlock --repo=s3:s3.amazonaws.com/cva-builder-velero-backups-test/restic/vault --password-file=/tmp/velero-restic-credentials-vault499987547 --cache-dir=/scratch/.cache/restic" logSource="pkg/restic/repository_manager.go:275" repository=vault stderr= stdout="successfully removed locks\n"
time="2020-05-12T18:42:50Z" level=debug msg=resticRepositoryController.runMaintenanceIfDue controller=restic-repository logSource="pkg/controller/restic_repository_controller.go:217" name=vault-default-4nhjh namespace=velero
time="2020-05-12T18:42:50Z" level=debug msg="not due for maintenance" controller=restic-repository logSource="pkg/controller/restic_repository_controller.go:222" name=vault-default-4nhjh namespace=velero
time="2020-05-12T18:42:50Z" level=debug msg="Running processQueueItem" controller=restic-repository key=velero/nginx-example-default-44t8d logSource="pkg/controller/restic_repository_controller.go:111"
time="2020-05-12T18:42:50Z" level=debug msg="Checking repository for stale locks" controller=restic-repository logSource="pkg/controller/restic_repository_controller.go:140" name=nginx-example-default-44t8d namespace=velero
time="2020-05-12T18:42:59Z" level=debug msg="Checking for existing backup storage locations to sync into cluster" controller=backup-sync logSource="pkg/controller/backup_sync_controller.go:124"
time="2020-05-12T18:42:59Z" level=debug msg="Checking if backups need to be synced at this time for this location" backupLocation=default controller=backup-sync logSource="pkg/controller/backup_sync_controller.go:156"
time="2020-05-12T18:42:59Z" level=debug msg="Checking backup location for backups to sync into cluster" backupLocation=default controller=backup-sync logSource="pkg/controller/backup_sync_controller.go:163"
time="2020-05-12T18:42:59Z" level=debug msg="looking for plugin in registry" controller=backup-sync kind=ObjectStore logSource="pkg/plugin/clientmgmt/manager.go:99" name=velero.io/aws
time="2020-05-12T18:42:59Z" level=debug msg="creating new restartable plugin process" command=/plugins/velero-plugin-for-aws controller=backup-sync kind=ObjectStore logSource="pkg/plugin/clientmgmt/manager.go:114" name=velero.io/aws
time="2020-05-12T18:42:59Z" level=debug msg="starting plugin" args="[/plugins/velero-plugin-for-aws --log-level debug]" cmd=/plugins/velero-plugin-for-aws controller=backup-sync logSource="pkg/plugin/clientmgmt/logrus_adapter.go:74" path=/plugins/velero-plugin-for-aws
time="2020-05-12T18:42:59Z" level=debug msg="plugin started" cmd=/plugins/velero-plugin-for-aws controller=backup-sync logSource="pkg/plugin/clientmgmt/logrus_adapter.go:74" path=/plugins/velero-plugin-for-aws pid=1937
time="2020-05-12T18:42:59Z" level=debug msg="waiting for RPC address" cmd=/plugins/velero-plugin-for-aws controller=backup-sync logSource="pkg/plugin/clientmgmt/logrus_adapter.go:74" path=/plugins/velero-plugin-for-aws
time="2020-05-12T18:42:59Z" level=debug msg="plugin address" address=/tmp/plugin194370527 cmd=/plugins/velero-plugin-for-aws controller=backup-sync logSource="pkg/plugin/clientmgmt/logrus_adapter.go:74" network=unix pluginName=velero-plugin-for-aws
time="2020-05-12T18:42:59Z" level=debug msg="using plugin" cmd=/plugins/velero-plugin-for-aws controller=backup-sync logSource="pkg/plugin/clientmgmt/logrus_adapter.go:74" version=2
time="2020-05-12T18:42:59Z" level=debug msg="Got backups from backup store" backupCount=16 backupLocation=default controller=backup-sync logSource="pkg/controller/backup_sync_controller.go:178"
time="2020-05-12T18:42:59Z" level=debug msg="Got backups from cluster" backupCount=16 backupLocation=default controller=backup-sync logSource="pkg/controller/backup_sync_controller.go:185"
time="2020-05-12T18:42:59Z" level=debug msg="No backups found in the backup location that need to be synced into the cluster" backupLocation=default controller=backup-sync logSource="pkg/controller/backup_sync_controller.go:198"
time="2020-05-12T18:42:59Z" level=debug msg="plugin process exited" cmd=/plugins/velero-plugin-for-aws controller=backup-sync logSource="pkg/plugin/clientmgmt/logrus_adapter.go:74" path=/plugins/velero-plugin-for-aws pid=1937
time="2020-05-12T18:42:59Z" level=debug msg="plugin exited" cmd=/plugins/velero-plugin-for-aws controller=backup-sync logSource="pkg/plugin/clientmgmt/logrus_adapter.go:74"
time="2020-05-12T18:43:21Z" level=debug msg="Ran restic command" command="restic unlock --repo=s3:s3.amazonaws.com/cva-builder-velero-backups-test/restic/nginx-example --password-file=/tmp/velero-restic-credentials-nginx-example886195710 --cache-dir=/scratch/.cache/restic" logSource="pkg/restic/repository_manager.go:275" repository=nginx-example stderr= stdout="successfully removed locks\n"
time="2020-05-12T18:43:21Z" level=debug msg=resticRepositoryController.runMaintenanceIfDue controller=restic-repository logSource="pkg/controller/restic_repository_controller.go:217" name=nginx-example-default-44t8d namespace=velero
time="2020-05-12T18:43:21Z" level=debug msg="not due for maintenance" controller=restic-repository logSource="pkg/controller/restic_repository_controller.go:222" name=nginx-example-default-44t8d namespace=velero
time="2020-05-12T18:43:21Z" level=debug msg="Running processQueueItem" controller=restic-repository key=velero/raffia-default-47hcn logSource="pkg/controller/restic_repository_controller.go:111"
time="2020-05-12T18:43:21Z" level=debug msg="Checking repository for stale locks" controller=restic-repository logSource="pkg/controller/restic_repository_controller.go:140" name=raffia-default-47hcn namespace=velero
time="2020-05-12T18:43:22Z" level=debug msg="Ran restic command" command="restic unlock --repo=s3:s3.amazonaws.com/cva-builder-velero-backups-test/restic/raffia --password-file=/tmp/velero-restic-credentials-raffia503717189 --cache-dir=/scratch/.cache/restic" logSource="pkg/restic/repository_manager.go:275" repository=raffia stderr= stdout="successfully removed locks\n"
time="2020-05-12T18:43:22Z" level=debug msg=resticRepositoryController.runMaintenanceIfDue controller=restic-repository logSource="pkg/controller/restic_repository_controller.go:217" name=raffia-default-47hcn namespace=velero
time="2020-05-12T18:43:22Z" level=debug msg="not due for maintenance" controller=restic-repository logSource="pkg/controller/restic_repository_controller.go:222" name=raffia-default-47hcn namespace=velero
time="2020-05-12T18:43:29Z" level=debug msg="Checking for existing backup storage locations to sync into cluster" controller=backup-sync logSource="pkg/controller/backup_sync_controller.go:124"
time="2020-05-12T18:43:29Z" level=debug msg="Checking if backups need to be synced at this time for this location" backupLocation=default controller=backup-sync logSource="pkg/controller/backup_sync_controller.go:156"
time="2020-05-12T18:43:59Z" level=debug msg="Checking for existing backup storage locations to sync into cluster" controller=backup-sync logSource="pkg/controller/backup_sync_controller.go:124"
time="2020-05-12T18:43:59Z" level=debug msg="Checking if backups need to be synced at this time for this location" backupLocation=default controller=backup-sync logSource="pkg/controller/backup_sync_controller.go:156"
time="2020-05-12T18:43:59Z" level=debug msg="Checking backup location for backups to sync into cluster" backupLocation=default controller=backup-sync logSource="pkg/controller/backup_sync_controller.go:163"
time="2020-05-12T18:43:59Z" level=debug msg="looking for plugin in registry" controller=backup-sync kind=ObjectStore logSource="pkg/plugin/clientmgmt/manager.go:99" name=velero.io/aws
time="2020-05-12T18:43:59Z" level=debug msg="creating new restartable plugin process" command=/plugins/velero-plugin-for-aws controller=backup-sync kind=ObjectStore logSource="pkg/plugin/clientmgmt/manager.go:114" name=velero.io/aws
time="2020-05-12T18:43:59Z" level=debug msg="starting plugin" args="[/plugins/velero-plugin-for-aws --log-level debug]" cmd=/plugins/velero-plugin-for-aws controller=backup-sync logSource="pkg/plugin/clientmgmt/logrus_adapter.go:74" path=/plugins/velero-plugin-for-aws
time="2020-05-12T18:43:59Z" level=debug msg="plugin started" cmd=/plugins/velero-plugin-for-aws controller=backup-sync logSource="pkg/plugin/clientmgmt/logrus_adapter.go:74" path=/plugins/velero-plugin-for-aws pid=1971
time="2020-05-12T18:43:59Z" level=debug msg="waiting for RPC address" cmd=/plugins/velero-plugin-for-aws controller=backup-sync logSource="pkg/plugin/clientmgmt/logrus_adapter.go:74" path=/plugins/velero-plugin-for-aws
time="2020-05-12T18:43:59Z" level=debug msg="plugin address" address=/tmp/plugin926215913 cmd=/plugins/velero-plugin-for-aws controller=backup-sync logSource="pkg/plugin/clientmgmt/logrus_adapter.go:74" network=unix pluginName=velero-plugin-for-aws
time="2020-05-12T18:43:59Z" level=debug msg="using plugin" cmd=/plugins/velero-plugin-for-aws controller=backup-sync logSource="pkg/plugin/clientmgmt/logrus_adapter.go:74" version=2
time="2020-05-12T18:43:59Z" level=debug msg="Got backups from backup store" backupCount=16 backupLocation=default controller=backup-sync logSource="pkg/controller/backup_sync_controller.go:178"
time="2020-05-12T18:43:59Z" level=debug msg="Got backups from cluster" backupCount=16 backupLocation=default controller=backup-sync logSource="pkg/controller/backup_sync_controller.go:185"
time="2020-05-12T18:43:59Z" level=debug msg="No backups found in the backup location that need to be synced into the cluster" backupLocation=default controller=backup-sync logSource="pkg/controller/backup_sync_controller.go:198"
time="2020-05-12T18:43:59Z" level=debug msg="plugin process exited" cmd=/plugins/velero-plugin-for-aws controller=backup-sync logSource="pkg/plugin/clientmgmt/logrus_adapter.go:74" path=/plugins/velero-plugin-for-aws pid=1971
time="2020-05-12T18:43:59Z" level=debug msg="plugin exited" cmd=/plugins/velero-plugin-for-aws controller=backup-sync logSource="pkg/plugin/clientmgmt/logrus_adapter.go:74"

The restore hangs on in progress, pvcs get created but no pv's, so they stay in pending until timing out after an hour.

edmund-nelson commented 4 years ago

Just trying another approach - I deleted the namespace and all pvs (of which there are 9)

I then tried a restore of just the PV's (without creating them via yaml first)

velero restore create --from-backup raffia-demo3 --include-resources persistentvolumes
Restore request "raffia-demo3-20200513100836" submitted successfully.
Run `velero restore describe raffia-demo3-20200513100836` or `velero restore logs raffia-demo3-20200513100836` for more details.

velero restore logs raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=debug msg="looking for plugin in registry" kind=ObjectStore logSource="pkg/plugin/clientmgmt/manager.go:99" name=velero.io/add-pv-from-pvc restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=debug msg="creating new restartable plugin process" command=/velero kind=ObjectStore logSource="pkg/plugin/clientmgmt/manager.go:114" name=velero.io/add-pv-from-pvc restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=debug msg="starting plugin" args="[/velero run-plugins --log-level debug]" cmd=/velero logSource="pkg/plugin/clientmgmt/logrus_adapter.go:74" path=/velero restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=debug msg="plugin started" cmd=/velero logSource="pkg/plugin/clientmgmt/logrus_adapter.go:74" path=/velero pid=130 restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=debug msg="waiting for RPC address" cmd=/velero logSource="pkg/plugin/clientmgmt/logrus_adapter.go:74" path=/velero restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=debug msg="Setting log level to DEBUG" cmd=/velero logSource="pkg/plugin/framework/server.go:172" pluginName=velero restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=debug msg="plugin address" address=/tmp/plugin512942643 cmd=/velero logSource="pkg/plugin/clientmgmt/logrus_adapter.go:74" network=unix pluginName=velero restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=debug msg="using plugin" cmd=/velero logSource="pkg/plugin/clientmgmt/logrus_adapter.go:74" restore=velero/raffia-demo3-20200513100836 version=2
time="2020-05-13T10:08:36Z" level=debug msg="looking for plugin in registry" kind=ObjectStore logSource="pkg/plugin/clientmgmt/manager.go:99" name=velero.io/add-pvc-from-pod restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=debug msg="found preexisting restartable plugin process" command=/velero kind=ObjectStore logSource="pkg/plugin/clientmgmt/manager.go:110" name=velero.io/add-pvc-from-pod restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=debug msg="looking for plugin in registry" kind=ObjectStore logSource="pkg/plugin/clientmgmt/manager.go:99" name=velero.io/change-storage-class restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=debug msg="found preexisting restartable plugin process" command=/velero kind=ObjectStore logSource="pkg/plugin/clientmgmt/manager.go:110" name=velero.io/change-storage-class restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=debug msg="looking for plugin in registry" kind=ObjectStore logSource="pkg/plugin/clientmgmt/manager.go:99" name=velero.io/cluster-role-bindings restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=debug msg="found preexisting restartable plugin process" command=/velero kind=ObjectStore logSource="pkg/plugin/clientmgmt/manager.go:110" name=velero.io/cluster-role-bindings restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=debug msg="looking for plugin in registry" kind=ObjectStore logSource="pkg/plugin/clientmgmt/manager.go:99" name=velero.io/crd-preserve-fields restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=debug msg="found preexisting restartable plugin process" command=/velero kind=ObjectStore logSource="pkg/plugin/clientmgmt/manager.go:110" name=velero.io/crd-preserve-fields restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=debug msg="looking for plugin in registry" kind=ObjectStore logSource="pkg/plugin/clientmgmt/manager.go:99" name=velero.io/job restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=debug msg="found preexisting restartable plugin process" command=/velero kind=ObjectStore logSource="pkg/plugin/clientmgmt/manager.go:110" name=velero.io/job restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=debug msg="looking for plugin in registry" kind=ObjectStore logSource="pkg/plugin/clientmgmt/manager.go:99" name=velero.io/pod restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=debug msg="found preexisting restartable plugin process" command=/velero kind=ObjectStore logSource="pkg/plugin/clientmgmt/manager.go:110" name=velero.io/pod restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=debug msg="looking for plugin in registry" kind=ObjectStore logSource="pkg/plugin/clientmgmt/manager.go:99" name=velero.io/restic restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=debug msg="found preexisting restartable plugin process" command=/velero kind=ObjectStore logSource="pkg/plugin/clientmgmt/manager.go:110" name=velero.io/restic restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=debug msg="looking for plugin in registry" kind=ObjectStore logSource="pkg/plugin/clientmgmt/manager.go:99" name=velero.io/role-bindings restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=debug msg="found preexisting restartable plugin process" command=/velero kind=ObjectStore logSource="pkg/plugin/clientmgmt/manager.go:110" name=velero.io/role-bindings restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=debug msg="looking for plugin in registry" kind=ObjectStore logSource="pkg/plugin/clientmgmt/manager.go:99" name=velero.io/service restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=debug msg="found preexisting restartable plugin process" command=/velero kind=ObjectStore logSource="pkg/plugin/clientmgmt/manager.go:110" name=velero.io/service restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=debug msg="looking for plugin in registry" kind=ObjectStore logSource="pkg/plugin/clientmgmt/manager.go:99" name=velero.io/service-account restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=debug msg="found preexisting restartable plugin process" command=/velero kind=ObjectStore logSource="pkg/plugin/clientmgmt/manager.go:110" name=velero.io/service-account restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=debug msg="Copied Backup to file" backup=raffia-demo3 bytes=24618 fileName=/tmp/raffia-demo3808738384 logSource="pkg/controller/restore_controller.go:550" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="starting restore" logSource="pkg/controller/restore_controller.go:458" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=customresourcedefinitions.apiextensions.k8s.io logSource="pkg/restore/restore.go:118" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=namespaces logSource="pkg/restore/restore.go:118" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=storageclasses.storage.k8s.io logSource="pkg/restore/restore.go:118" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=persistentvolumeclaims logSource="pkg/restore/restore.go:118" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=secrets logSource="pkg/restore/restore.go:118" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=configmaps logSource="pkg/restore/restore.go:118" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=serviceaccounts logSource="pkg/restore/restore.go:118" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=limitranges logSource="pkg/restore/restore.go:118" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=pods logSource="pkg/restore/restore.go:118" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=replicasets.apps logSource="pkg/restore/restore.go:118" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=namespaces logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=events logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=limitranges logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=services logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=configmaps logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=secrets logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=nodes logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=pods logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=endpoints logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=podtemplates logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=resourcequotas logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=persistentvolumeclaims logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=serviceaccounts logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=replicationcontrollers logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=apiservices.apiregistration.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=deployments.apps logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=controllerrevisions.apps logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=daemonsets.apps logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=statefulsets.apps logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=replicasets.apps logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=events.events.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=horizontalpodautoscalers.autoscaling logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=jobs.batch logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=cronjobs.batch logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=certificatesigningrequests.certificates.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=networkpolicies.networking.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=poddisruptionbudgets.policy logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=podsecuritypolicies.policy logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=roles.rbac.authorization.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=clusterrolebindings.rbac.authorization.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=rolebindings.rbac.authorization.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=clusterroles.rbac.authorization.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=storageclasses.storage.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=volumeattachments.storage.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=validatingwebhookconfigurations.admissionregistration.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=mutatingwebhookconfigurations.admissionregistration.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=initializerconfigurations.admissionregistration.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=customresourcedefinitions.apiextensions.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=priorityclasses.scheduling.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=leases.coordination.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=cephnfses.ceph.rook.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=cephclusters.ceph.rook.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=cephfilesystems.ceph.rook.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=cephobjectstoreusers.ceph.rook.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=cephobjectstores.ceph.rook.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=cephblockpools.ceph.rook.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=servicemonitors.monitoring.coreos.com logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=prometheusrules.monitoring.coreos.com logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=prometheuses.monitoring.coreos.com logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=alertmanagers.monitoring.coreos.com logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=podmonitors.monitoring.coreos.com logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=restores.velero.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=serverstatusrequests.velero.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=backupstoragelocations.velero.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=podvolumerestores.velero.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=volumesnapshotlocations.velero.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=deletebackuprequests.velero.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=schedules.velero.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=downloadrequests.velero.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=backups.velero.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=resticrepositories.velero.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=podvolumebackups.velero.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=issuers.certmanager.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=clusterissuers.certmanager.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=certificates.certmanager.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=nfsservers.nfs.rook.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=objectbucketclaims.objectbucket.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=objectbuckets.objectbucket.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=volumesnapshots.snapshot.storage.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=volumesnapshotcontents.snapshot.storage.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=volumesnapshotclasses.snapshot.storage.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=volumes.rook.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=httptriggers.kubeless.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=cronjobtriggers.kubeless.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=functions.kubeless.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=deployments.extensions logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=daemonsets.extensions logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=podsecuritypolicies.extensions logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=networkpolicies.extensions logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=replicasets.extensions logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Not including resource" groupResource=ingresses.extensions logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Starting restore of backup velero/raffia-demo3" logSource="pkg/restore/restore.go:394" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Restoring cluster level resource 'persistentvolumes'" logSource="pkg/restore/restore.go:779" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Getting client for /v1, Kind=PersistentVolume" logSource="pkg/restore/restore.go:821" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Dynamically re-provisioning persistent volume because it has a restic backup to be restored." logSource="pkg/restore/restore.go:992" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Dynamically re-provisioning persistent volume because it has a restic backup to be restored." logSource="pkg/restore/restore.go:992" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Dynamically re-provisioning persistent volume because it has a restic backup to be restored." logSource="pkg/restore/restore.go:992" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Dynamically re-provisioning persistent volume because it has a restic backup to be restored." logSource="pkg/restore/restore.go:992" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Restoring persistent volume as-is because it doesn't have a snapshot and its reclaim policy is not Delete." logSource="pkg/restore/restore.go:1006" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="No snapshot found for persistent volume" logSource="pkg/restore/pv_restorer.go:81" persistentVolume=raffia-engine-triage restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Executing item action for persistentvolumes" logSource="pkg/restore/restore.go:1030" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:63" pluginName=velero restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=debug msg="Getting plugin config" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:66" pluginName=velero restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=debug msg="No storage class mappings found" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:73" pluginName=velero restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Done executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:74" pluginName=velero restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Attempting to restore PersistentVolume: raffia-engine-triage" logSource="pkg/restore/restore.go:1136" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Dynamically re-provisioning persistent volume because it has a restic backup to be restored." logSource="pkg/restore/restore.go:992" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Restoring persistent volume as-is because it doesn't have a snapshot and its reclaim policy is not Delete." logSource="pkg/restore/restore.go:1006" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="No snapshot found for persistent volume" logSource="pkg/restore/pv_restorer.go:81" persistentVolume=raffia-home restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Executing item action for persistentvolumes" logSource="pkg/restore/restore.go:1030" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:63" pluginName=velero restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=debug msg="Getting plugin config" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:66" pluginName=velero restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=debug msg="No storage class mappings found" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:73" pluginName=velero restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Done executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:74" pluginName=velero restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Attempting to restore PersistentVolume: raffia-home" logSource="pkg/restore/restore.go:1136" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Dynamically re-provisioning persistent volume because it has a restic backup to be restored." logSource="pkg/restore/restore.go:992" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:36Z" level=info msg="Dynamically re-provisioning persistent volume because it has a restic backup to be restored." logSource="pkg/restore/restore.go:992" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=customresourcedefinitions.apiextensions.k8s.io logSource="pkg/restore/restore.go:118" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=namespaces logSource="pkg/restore/restore.go:118" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=storageclasses.storage.k8s.io logSource="pkg/restore/restore.go:118" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=persistentvolumeclaims logSource="pkg/restore/restore.go:118" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=secrets logSource="pkg/restore/restore.go:118" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=configmaps logSource="pkg/restore/restore.go:118" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=serviceaccounts logSource="pkg/restore/restore.go:118" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=limitranges logSource="pkg/restore/restore.go:118" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=pods logSource="pkg/restore/restore.go:118" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=replicasets.apps logSource="pkg/restore/restore.go:118" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=endpoints logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=podtemplates logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=replicationcontrollers logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=services logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=namespaces logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=pods logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=configmaps logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=nodes logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=persistentvolumeclaims logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=resourcequotas logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=secrets logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=events logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=limitranges logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=serviceaccounts logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=apiservices.apiregistration.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=controllerrevisions.apps logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=statefulsets.apps logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=daemonsets.apps logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=replicasets.apps logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=deployments.apps logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=events.events.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=horizontalpodautoscalers.autoscaling logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=jobs.batch logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=cronjobs.batch logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=certificatesigningrequests.certificates.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=networkpolicies.networking.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=poddisruptionbudgets.policy logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=podsecuritypolicies.policy logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=rolebindings.rbac.authorization.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=clusterroles.rbac.authorization.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=clusterrolebindings.rbac.authorization.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=roles.rbac.authorization.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=volumeattachments.storage.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=storageclasses.storage.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=mutatingwebhookconfigurations.admissionregistration.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=validatingwebhookconfigurations.admissionregistration.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=initializerconfigurations.admissionregistration.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=customresourcedefinitions.apiextensions.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=priorityclasses.scheduling.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=leases.coordination.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=cephnfses.ceph.rook.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=cephclusters.ceph.rook.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=cephblockpools.ceph.rook.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=cephfilesystems.ceph.rook.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=cephobjectstoreusers.ceph.rook.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=cephobjectstores.ceph.rook.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=prometheuses.monitoring.coreos.com logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=podmonitors.monitoring.coreos.com logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=prometheusrules.monitoring.coreos.com logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=servicemonitors.monitoring.coreos.com logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=alertmanagers.monitoring.coreos.com logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=volumesnapshotlocations.velero.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=backupstoragelocations.velero.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=resticrepositories.velero.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=podvolumebackups.velero.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=schedules.velero.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=podvolumerestores.velero.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=backups.velero.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=serverstatusrequests.velero.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=deletebackuprequests.velero.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=restores.velero.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=downloadrequests.velero.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=issuers.certmanager.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=certificates.certmanager.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=clusterissuers.certmanager.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=nfsservers.nfs.rook.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=objectbuckets.objectbucket.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=objectbucketclaims.objectbucket.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=volumesnapshotcontents.snapshot.storage.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=volumesnapshots.snapshot.storage.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=volumesnapshotclasses.snapshot.storage.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=volumes.rook.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=cronjobtriggers.kubeless.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=functions.kubeless.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=httptriggers.kubeless.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=daemonsets.extensions logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=networkpolicies.extensions logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=deployments.extensions logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=ingresses.extensions logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=podsecuritypolicies.extensions logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Not including resource" groupResource=replicasets.extensions logSource="pkg/restore/restore.go:138" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Waiting for all restic restores to complete" logSource="pkg/restore/restore.go:545" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="Done waiting for all restic restores to complete" logSource="pkg/restore/restore.go:561" restore=velero/raffia-demo3-20200513100836
time="2020-05-13T10:08:39Z" level=info msg="restore completed" logSource="pkg/controller/restore_controller.go:473" restore=velero/raffia-demo3-20200513100836

only two PV's get restored - and the others were not excluded from the backup

NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS        CLAIM                                STORAGECLASS         REASON   AGE
raffia-engine-triage                       2Gi        RWO            Retain           Available                                          local-storage                 3m28s
raffia-home                                1M         RWO            Retain           Available                                          local-storage                 3m28s

The errors oddly say "No snapshot found for persistent volume" logSource="pkg/restore/pv_restorer.go:81" persistentVolume=raffia-home " and "No snapshot found for persistent volume" logSource="pkg/restore/pv_restorer.go:81" persistentVolume=raffia-engine-triage"

which are the ones which actually have restored.

When describing the backup I can see some PV's - however im going to re-run a pr and backups to double check my annotations are correct as some look to be missing


Persistent Volumes: <none included>

Restic Backups:
  Completed:
    raffia/raffia-mongodb-54ccbdcf87-c9b6m: data
    raffia/raffia-raffia-5667cc7768-5cn64: raffia-config, raffia-domains, raffia-scripts, raffia-var, vaultca
    raffia/raffia-raffia-engine-7b665ddbb4-fq8zv: raffia-engine-scripts, raffiaengine-config, raffiaengine-home, raffiaengine-var, vaultca
``
edmund-nelson commented 4 years ago

I've tried another test - creating a new backup on a new cluster of a namespace. All the pods are correctly annotated with the volumenames. I deleted the raffia namespace and the volumes before the restore

velero backup create raffia-pr-1724 --include-namespaces raffia --snapshot-volumes
kubectl delete namespace raffia
kubectl delete pv raffia-config raffia-domains raffia-engine-config raffia-engine-home raffia-engine-triage raffia-engine-var raffia-home raffia-mongodb-data raffia-raffia-var
persistentvolume "raffia-config" deleted
persistentvolume "raffia-domains" deleted
persistentvolume "raffia-engine-config" deleted
persistentvolume "raffia-engine-home" deleted
persistentvolume "raffia-engine-triage" deleted
persistentvolume "raffia-engine-var" deleted
persistentvolume "raffia-home" deleted
persistentvolume "raffia-mongodb-data" deleted
persistentvolume "raffia-raffia-var" deleted
velero describe backup raffia-pr-1724 --details
Name:         raffia-pr-1724
Namespace:    velero
Labels:       velero.io/storage-location=default
Annotations:  <none>

Phase:  Completed

Namespaces:
  Included:  raffia
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  auto

Label selector:  <none>

Storage Location:  default

Snapshot PVs:  true

TTL:  720h0m0s

Hooks:  <none>

Backup Format Version:  1

Started:    2020-05-13 11:19:33 +0000 UTC
Completed:  2020-05-13 11:20:43 +0000 UTC

Expiration:  2020-06-12 11:19:33 +0000 UTC

Resource List:
  apps/v1/Deployment:
    - raffia/raffia-mongodb
    - raffia/raffia-raffia
    - raffia/raffia-raffia-engine
  apps/v1/ReplicaSet:
    - raffia/raffia-mongodb-54ccbdcf87
    - raffia/raffia-raffia-5667cc7768
    - raffia/raffia-raffia-engine-7b665ddbb4
  extensions/v1beta1/Ingress:
    - raffia/raffia-raffia
  rbac.authorization.k8s.io/v1/Role:
    - raffia/raffia-role-read-only
  rbac.authorization.k8s.io/v1/RoleBinding:
    - raffia/raffia-read-only-role-binding
  v1/ConfigMap:
    - raffia/raffia-raffia-engine-scripts
    - raffia/raffia-raffia-scripts
  v1/Endpoints:
    - raffia/raffia-mongodb
    - raffia/raffia-raffia
  v1/Event:
    - raffia/raffia-mongodb-54ccbdcf87-c9b6m.160e910a2726cbba
    - raffia/raffia-mongodb-54ccbdcf87-c9b6m.160e910a55654e67
    - raffia/raffia-mongodb-54ccbdcf87-c9b6m.160e910a9d9720e2
    - raffia/raffia-mongodb-54ccbdcf87-c9b6m.160e910aa08d5c25
    - raffia/raffia-mongodb-54ccbdcf87-c9b6m.160e910aa9df38fb
    - raffia/raffia-mongodb-54ccbdcf87-c9b6m.160e910cd57b8a46
    - raffia/raffia-mongodb-54ccbdcf87-c9b6m.160e910cf47878fe
    - raffia/raffia-mongodb-54ccbdcf87-c9b6m.160e910cfdcd5c4a
    - raffia/raffia-mongodb.160e9109fd1dc5d3
    - raffia/raffia-raffia-5667cc7768-5cn64.160e910a7016456e
    - raffia/raffia-raffia-5667cc7768-5cn64.160e910b24a68184
    - raffia/raffia-raffia-5667cc7768-5cn64.160e910b26ff5740
    - raffia/raffia-raffia-5667cc7768-5cn64.160e910b30440692
    - raffia/raffia-raffia-5667cc7768-5cn64.160e9111fb46dfb5
    - raffia/raffia-raffia-5667cc7768-5cn64.160e9111fec427d6
    - raffia/raffia-raffia-5667cc7768-5cn64.160e91120b814e0e
    - raffia/raffia-raffia-5667cc7768-5cn64.160e9115b3761e5d
    - raffia/raffia-raffia-5667cc7768-5cn64.160e9115b66cdfff
    - raffia/raffia-raffia-5667cc7768-5cn64.160e9115c02cc9c5
    - raffia/raffia-raffia-5667cc7768-5cn64.160e91197c2c70ee
    - raffia/raffia-raffia-5667cc7768-5cn64.160e911a0d8278f5
    - raffia/raffia-raffia-5667cc7768-5cn64.160e911a164d9fca
    - raffia/raffia-raffia-config.160e9109fe5ec659
    - raffia/raffia-raffia-domains.160e9109ffe31e90
    - raffia/raffia-raffia-engine-7b665ddbb4-fq8zv.160e910a89aedbed
    - raffia/raffia-raffia-engine-7b665ddbb4-fq8zv.160e910c1f5d7b2b
    - raffia/raffia-raffia-engine-7b665ddbb4-fq8zv.160e910c21f5537d
    - raffia/raffia-raffia-engine-7b665ddbb4-fq8zv.160e910c2adf937a
    - raffia/raffia-raffia-engine-7b665ddbb4-fq8zv.160e91156c04430d
    - raffia/raffia-raffia-engine-7b665ddbb4-fq8zv.160e91156ea01d51
    - raffia/raffia-raffia-engine-7b665ddbb4-fq8zv.160e911578322a40
    - raffia/raffia-raffia-engine-7b665ddbb4-fq8zv.160e91191cc0f012
    - raffia/raffia-raffia-engine-7b665ddbb4-fq8zv.160e91191f6653c5
    - raffia/raffia-raffia-engine-7b665ddbb4-fq8zv.160e911928e937e0
    - raffia/raffia-raffia-engine-7b665ddbb4-fq8zv.160e91252862bf6e
    - raffia/raffia-raffia-engine-7b665ddbb4-fq8zv.160e91252ade8fe1
    - raffia/raffia-raffia-engine-7b665ddbb4-fq8zv.160e912533cc0730
    - raffia/raffia-raffia-engine-7b665ddbb4-fq8zv.160e9128db373306
    - raffia/raffia-raffia-engine-7b665ddbb4-fq8zv.160e9128df4f1af1
    - raffia/raffia-raffia-engine-7b665ddbb4-fq8zv.160e9128e9223df2
    - raffia/raffia-raffia-engine-config.160e910a0167de6a
    - raffia/raffia-raffia-engine-home.160e910a022c64ef
    - raffia/raffia-raffia-engine-var.160e910a0545076a
    - raffia/raffia-raffia-var.160e910a0937d349
    - raffia/raffia-raffia.160e910a2a405cd6
    - raffia/raffia-raffia.160e910a2bed4257
    - raffia/raffia-raffia.160e911569c7b95d
    - raffia/raffia-raffia.160e911569c96430
  v1/Namespace:
    - raffia
  v1/PersistentVolume:
    - raffia-config
    - raffia-domains
    - raffia-engine-config
    - raffia-engine-home
    - raffia-engine-triage
    - raffia-engine-var
    - raffia-home
    - raffia-mongodb-data
    - raffia-raffia-var
  v1/PersistentVolumeClaim:
    - raffia/raffia-mongodb
    - raffia/raffia-raffia-config
    - raffia/raffia-raffia-domains
    - raffia/raffia-raffia-engine-config
    - raffia/raffia-raffia-engine-home
    - raffia/raffia-raffia-engine-triage
    - raffia/raffia-raffia-engine-var
    - raffia/raffia-raffia-home
    - raffia/raffia-raffia-var
  v1/Pod:
    - raffia/raffia-mongodb-54ccbdcf87-c9b6m
    - raffia/raffia-raffia-5667cc7768-5cn64
    - raffia/raffia-raffia-engine-7b665ddbb4-fq8zv
  v1/Secret:
    - raffia/cva-id
    - raffia/cva-info
    - raffia/default-token-287bb
    - raffia/dxcdockerslmcred
    - raffia/raffia-cert-vault-approle
    - raffia/raffia-deploy-vault-approle
    - raffia/raffia-ephemeral-path
    - raffia/raffia-mongodb
    - raffia/raffia-sa-token-xhls7
    - raffia/raffia-vault-approle
    - raffia/vault-client-tls
  v1/Service:
    - raffia/raffia-mongodb
    - raffia/raffia-raffia
  v1/ServiceAccount:
    - raffia/default
    - raffia/raffia-sa

Persistent Volumes: <none included>

Restic Backups:
  Completed:
    raffia/raffia-mongodb-54ccbdcf87-c9b6m: data, default-token-287bb
    raffia/raffia-raffia-5667cc7768-5cn64: default-token-287bb, raffia-config, raffia-domains, raffia-home, raffia-scripts, raffia-var, vaultca
    raffia/raffia-raffia-engine-7b665ddbb4-fq8zv: default-token-287bb, raffia-engine-scripts, raffiaengine-config, raffiaengine-home, raffiaengine-triage, raffiaengine-var, vaultca

Tried a restore of JUST persistentvolumes

velero restore create --from-backup raffia-pr-1724 --include-resources persistentvolumes
Restore request "raffia-pr-1724-20200513112330" submitted successfully.
velero restore logs raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=debug msg="looking for plugin in registry" kind=ObjectStore logSource="pkg/plugin/clientmgmt/manager.go:99" name=velero.io/add-pv-from-pvc restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=debug msg="creating new restartable plugin process" command=/velero kind=ObjectStore logSource="pkg/plugin/clientmgmt/manager.go:114" name=velero.io/add-pv-from-pvc restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=debug msg="starting plugin" args="[/velero run-plugins --log-level debug]" cmd=/velero logSource="pkg/plugin/clientmgmt/logrus_adapter.go:74" path=/velero restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=debug msg="plugin started" cmd=/velero logSource="pkg/plugin/clientmgmt/logrus_adapter.go:74" path=/velero pid=1776 restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=debug msg="waiting for RPC address" cmd=/velero logSource="pkg/plugin/clientmgmt/logrus_adapter.go:74" path=/velero restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=debug msg="Setting log level to DEBUG" cmd=/velero logSource="pkg/plugin/framework/server.go:172" pluginName=velero restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=debug msg="plugin address" address=/tmp/plugin013891697 cmd=/velero logSource="pkg/plugin/clientmgmt/logrus_adapter.go:74" network=unix pluginName=velero restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=debug msg="using plugin" cmd=/velero logSource="pkg/plugin/clientmgmt/logrus_adapter.go:74" restore=velero/raffia-pr-1724-20200513112330 version=2
time="2020-05-13T11:23:30Z" level=debug msg="looking for plugin in registry" kind=ObjectStore logSource="pkg/plugin/clientmgmt/manager.go:99" name=velero.io/add-pvc-from-pod restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=debug msg="found preexisting restartable plugin process" command=/velero kind=ObjectStore logSource="pkg/plugin/clientmgmt/manager.go:110" name=velero.io/add-pvc-from-pod restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=debug msg="looking for plugin in registry" kind=ObjectStore logSource="pkg/plugin/clientmgmt/manager.go:99" name=velero.io/change-storage-class restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=debug msg="found preexisting restartable plugin process" command=/velero kind=ObjectStore logSource="pkg/plugin/clientmgmt/manager.go:110" name=velero.io/change-storage-class restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=debug msg="looking for plugin in registry" kind=ObjectStore logSource="pkg/plugin/clientmgmt/manager.go:99" name=velero.io/cluster-role-bindings restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=debug msg="found preexisting restartable plugin process" command=/velero kind=ObjectStore logSource="pkg/plugin/clientmgmt/manager.go:110" name=velero.io/cluster-role-bindings restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=debug msg="looking for plugin in registry" kind=ObjectStore logSource="pkg/plugin/clientmgmt/manager.go:99" name=velero.io/crd-preserve-fields restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=debug msg="found preexisting restartable plugin process" command=/velero kind=ObjectStore logSource="pkg/plugin/clientmgmt/manager.go:110" name=velero.io/crd-preserve-fields restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=debug msg="looking for plugin in registry" kind=ObjectStore logSource="pkg/plugin/clientmgmt/manager.go:99" name=velero.io/job restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=debug msg="found preexisting restartable plugin process" command=/velero kind=ObjectStore logSource="pkg/plugin/clientmgmt/manager.go:110" name=velero.io/job restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=debug msg="looking for plugin in registry" kind=ObjectStore logSource="pkg/plugin/clientmgmt/manager.go:99" name=velero.io/pod restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=debug msg="found preexisting restartable plugin process" command=/velero kind=ObjectStore logSource="pkg/plugin/clientmgmt/manager.go:110" name=velero.io/pod restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=debug msg="looking for plugin in registry" kind=ObjectStore logSource="pkg/plugin/clientmgmt/manager.go:99" name=velero.io/restic restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=debug msg="found preexisting restartable plugin process" command=/velero kind=ObjectStore logSource="pkg/plugin/clientmgmt/manager.go:110" name=velero.io/restic restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=debug msg="looking for plugin in registry" kind=ObjectStore logSource="pkg/plugin/clientmgmt/manager.go:99" name=velero.io/role-bindings restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=debug msg="found preexisting restartable plugin process" command=/velero kind=ObjectStore logSource="pkg/plugin/clientmgmt/manager.go:110" name=velero.io/role-bindings restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=debug msg="looking for plugin in registry" kind=ObjectStore logSource="pkg/plugin/clientmgmt/manager.go:99" name=velero.io/service restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=debug msg="found preexisting restartable plugin process" command=/velero kind=ObjectStore logSource="pkg/plugin/clientmgmt/manager.go:110" name=velero.io/service restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=debug msg="looking for plugin in registry" kind=ObjectStore logSource="pkg/plugin/clientmgmt/manager.go:99" name=velero.io/service-account restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=debug msg="found preexisting restartable plugin process" command=/velero kind=ObjectStore logSource="pkg/plugin/clientmgmt/manager.go:110" name=velero.io/service-account restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=debug msg="Copied Backup to file" backup=raffia-pr-1724 bytes=28473 fileName=/tmp/raffia-pr-1724837252429 logSource="pkg/controller/restore_controller.go:550" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="starting restore" logSource="pkg/controller/restore_controller.go:458" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=customresourcedefinitions.apiextensions.k8s.io logSource="pkg/restore/restore.go:118" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=namespaces logSource="pkg/restore/restore.go:118" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=storageclasses.storage.k8s.io logSource="pkg/restore/restore.go:118" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=persistentvolumeclaims logSource="pkg/restore/restore.go:118" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=secrets logSource="pkg/restore/restore.go:118" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=configmaps logSource="pkg/restore/restore.go:118" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=serviceaccounts logSource="pkg/restore/restore.go:118" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=limitranges logSource="pkg/restore/restore.go:118" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=pods logSource="pkg/restore/restore.go:118" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=replicasets.apps logSource="pkg/restore/restore.go:118" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=resourcequotas logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=secrets logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=configmaps logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=events logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=nodes logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=podtemplates logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=limitranges logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=replicationcontrollers logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=serviceaccounts logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=services logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=endpoints logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=pods logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=namespaces logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=persistentvolumeclaims logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=apiservices.apiregistration.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=statefulsets.apps logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=replicasets.apps logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=controllerrevisions.apps logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=deployments.apps logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=daemonsets.apps logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=events.events.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=horizontalpodautoscalers.autoscaling logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=jobs.batch logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=cronjobs.batch logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=certificatesigningrequests.certificates.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=networkpolicies.networking.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=poddisruptionbudgets.policy logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=podsecuritypolicies.policy logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=roles.rbac.authorization.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=clusterroles.rbac.authorization.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=clusterrolebindings.rbac.authorization.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=rolebindings.rbac.authorization.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=storageclasses.storage.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=volumeattachments.storage.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=mutatingwebhookconfigurations.admissionregistration.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=validatingwebhookconfigurations.admissionregistration.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=initializerconfigurations.admissionregistration.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=customresourcedefinitions.apiextensions.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=priorityclasses.scheduling.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=leases.coordination.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=cephblockpools.ceph.rook.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=cephnfses.ceph.rook.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=cephobjectstoreusers.ceph.rook.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=cephobjectstores.ceph.rook.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=cephfilesystems.ceph.rook.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=cephclusters.ceph.rook.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=servicemonitors.monitoring.coreos.com logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=prometheusrules.monitoring.coreos.com logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=alertmanagers.monitoring.coreos.com logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=prometheuses.monitoring.coreos.com logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=podmonitors.monitoring.coreos.com logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=podvolumebackups.velero.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=backupstoragelocations.velero.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=restores.velero.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=serverstatusrequests.velero.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=resticrepositories.velero.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=downloadrequests.velero.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=schedules.velero.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=deletebackuprequests.velero.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=volumesnapshotlocations.velero.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=podvolumerestores.velero.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=backups.velero.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=issuers.certmanager.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=certificates.certmanager.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=clusterissuers.certmanager.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=nfsservers.nfs.rook.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=objectbucketclaims.objectbucket.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=objectbuckets.objectbucket.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=volumesnapshots.snapshot.storage.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=volumesnapshotclasses.snapshot.storage.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=volumesnapshotcontents.snapshot.storage.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=volumes.rook.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=functions.kubeless.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=httptriggers.kubeless.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=cronjobtriggers.kubeless.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=daemonsets.extensions logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=replicasets.extensions logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=ingresses.extensions logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=networkpolicies.extensions logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=podsecuritypolicies.extensions logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Not including resource" groupResource=deployments.extensions logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Starting restore of backup velero/raffia-pr-1724" logSource="pkg/restore/restore.go:394" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Restoring cluster level resource 'persistentvolumes'" logSource="pkg/restore/restore.go:779" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Getting client for /v1, Kind=PersistentVolume" logSource="pkg/restore/restore.go:821" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Dynamically re-provisioning persistent volume because it has a restic backup to be restored." logSource="pkg/restore/restore.go:992" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Dynamically re-provisioning persistent volume because it has a restic backup to be restored." logSource="pkg/restore/restore.go:992" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Dynamically re-provisioning persistent volume because it has a restic backup to be restored." logSource="pkg/restore/restore.go:992" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Dynamically re-provisioning persistent volume because it has a restic backup to be restored." logSource="pkg/restore/restore.go:992" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Dynamically re-provisioning persistent volume because it has a restic backup to be restored." logSource="pkg/restore/restore.go:992" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Dynamically re-provisioning persistent volume because it has a restic backup to be restored." logSource="pkg/restore/restore.go:992" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Dynamically re-provisioning persistent volume because it has a restic backup to be restored." logSource="pkg/restore/restore.go:992" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Dynamically re-provisioning persistent volume because it has a restic backup to be restored." logSource="pkg/restore/restore.go:992" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:30Z" level=info msg="Dynamically re-provisioning persistent volume because it has a restic backup to be restored." logSource="pkg/restore/restore.go:992" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=customresourcedefinitions.apiextensions.k8s.io logSource="pkg/restore/restore.go:118" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=namespaces logSource="pkg/restore/restore.go:118" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=storageclasses.storage.k8s.io logSource="pkg/restore/restore.go:118" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=persistentvolumeclaims logSource="pkg/restore/restore.go:118" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=secrets logSource="pkg/restore/restore.go:118" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=configmaps logSource="pkg/restore/restore.go:118" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=serviceaccounts logSource="pkg/restore/restore.go:118" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=limitranges logSource="pkg/restore/restore.go:118" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=pods logSource="pkg/restore/restore.go:118" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=replicasets.apps logSource="pkg/restore/restore.go:118" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=podtemplates logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=pods logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=secrets logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=replicationcontrollers logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=endpoints logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=limitranges logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=nodes logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=resourcequotas logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=events logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=serviceaccounts logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=services logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=namespaces logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=configmaps logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=persistentvolumeclaims logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=apiservices.apiregistration.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=replicasets.apps logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=statefulsets.apps logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=daemonsets.apps logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=deployments.apps logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=controllerrevisions.apps logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=events.events.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=horizontalpodautoscalers.autoscaling logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=jobs.batch logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=cronjobs.batch logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=certificatesigningrequests.certificates.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=networkpolicies.networking.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=podsecuritypolicies.policy logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=poddisruptionbudgets.policy logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=roles.rbac.authorization.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=clusterrolebindings.rbac.authorization.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=clusterroles.rbac.authorization.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=rolebindings.rbac.authorization.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=storageclasses.storage.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=volumeattachments.storage.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=mutatingwebhookconfigurations.admissionregistration.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=validatingwebhookconfigurations.admissionregistration.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=initializerconfigurations.admissionregistration.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=customresourcedefinitions.apiextensions.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=priorityclasses.scheduling.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=leases.coordination.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=cephobjectstores.ceph.rook.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=cephfilesystems.ceph.rook.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=cephblockpools.ceph.rook.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=cephnfses.ceph.rook.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=cephobjectstoreusers.ceph.rook.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=cephclusters.ceph.rook.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=alertmanagers.monitoring.coreos.com logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=podmonitors.monitoring.coreos.com logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=prometheusrules.monitoring.coreos.com logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=prometheuses.monitoring.coreos.com logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=servicemonitors.monitoring.coreos.com logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=volumesnapshotlocations.velero.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=podvolumerestores.velero.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=backups.velero.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=downloadrequests.velero.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=restores.velero.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=serverstatusrequests.velero.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=backupstoragelocations.velero.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=schedules.velero.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=resticrepositories.velero.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=deletebackuprequests.velero.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=podvolumebackups.velero.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=issuers.certmanager.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=clusterissuers.certmanager.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=certificates.certmanager.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=nfsservers.nfs.rook.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=objectbucketclaims.objectbucket.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=objectbuckets.objectbucket.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=volumesnapshotclasses.snapshot.storage.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=volumesnapshotcontents.snapshot.storage.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=volumesnapshots.snapshot.storage.k8s.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=volumes.rook.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=cronjobtriggers.kubeless.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=httptriggers.kubeless.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=functions.kubeless.io logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=networkpolicies.extensions logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=podsecuritypolicies.extensions logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=ingresses.extensions logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=daemonsets.extensions logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=deployments.extensions logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Not including resource" groupResource=replicasets.extensions logSource="pkg/restore/restore.go:138" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Waiting for all restic restores to complete" logSource="pkg/restore/restore.go:545" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="Done waiting for all restic restores to complete" logSource="pkg/restore/restore.go:561" restore=velero/raffia-pr-1724-20200513112330
time="2020-05-13T11:23:33Z" level=info msg="restore completed" logSource="pkg/controller/restore_controller.go:473" restore=velero/raffia-pr-1724-20200513112330

raffia namespace is created - but no PV's

kubectl get namespace | grep raffia
raffia               Active   12m

kubectl get pv | grep -i raffia
tmp]$

Looked at the restic log and see the following...

kubectl logs restic-x5z2g -n velero
time="2020-05-13T09:56:24Z" level=info msg="Setting log-level to INFO"
time="2020-05-13T09:56:24Z" level=info msg="Starting Velero restic server v1.3.1 (0665b05321eefeb7b7fdd6984750745b7429774f)" logSource="pkg/cmd/cli/restic/server.go:62"
time="2020-05-13T09:56:24Z" level=info msg="Starting controllers" logSource="pkg/cmd/cli/restic/server.go:156"
time="2020-05-13T09:56:24Z" level=info msg="Controllers started successfully" logSource="pkg/cmd/cli/restic/server.go:199"
time="2020-05-13T09:56:24Z" level=info msg="Starting controller" controller=pod-volume-backup logSource="pkg/controller/generic_controller.go:76"
time="2020-05-13T09:56:24Z" level=info msg="Waiting for caches to sync" controller=pod-volume-backup logSource="pkg/controller/generic_controller.go:79"
time="2020-05-13T09:56:24Z" level=info msg="Starting controller" controller=pod-volume-restore logSource="pkg/controller/generic_controller.go:76"
time="2020-05-13T09:56:24Z" level=info msg="Waiting for caches to sync" controller=pod-volume-restore logSource="pkg/controller/generic_controller.go:79"
time="2020-05-13T09:56:24Z" level=info msg="Caches are synced" controller=pod-volume-backup logSource="pkg/controller/generic_controller.go:83"
time="2020-05-13T09:56:24Z" level=info msg="Caches are synced" controller=pod-volume-restore logSource="pkg/controller/generic_controller.go:83"
time="2020-05-13T10:53:15Z" level=info msg="Restore starting" controller=pod-volume-restore logSource="pkg/controller/pod_volume_restore_controller.go:262" name=raffia-demo3-20200513105239-7fccp namespace=velero restore=velero/raffia-demo3-20200513105239
time="2020-05-13T10:53:22Z" level=info msg="Restore completed" controller=pod-volume-restore logSource="pkg/controller/pod_volume_restore_controller.go:311" name=raffia-demo3-20200513105239-7fccp namespace=velero restore=velero/raffia-demo3-20200513105239
time="2020-05-13T10:53:22Z" level=info msg="Restore starting" controller=pod-volume-restore logSource="pkg/controller/pod_volume_restore_controller.go:262" name=raffia-demo3-20200513105239-nh4r6 namespace=velero restore=velero/raffia-demo3-20200513105239
time="2020-05-13T10:53:25Z" level=info msg="Restore completed" controller=pod-volume-restore logSource="pkg/controller/pod_volume_restore_controller.go:311" name=raffia-demo3-20200513105239-nh4r6 namespace=velero restore=velero/raffia-demo3-20200513105239
time="2020-05-13T10:53:25Z" level=info msg="Restore starting" controller=pod-volume-restore logSource="pkg/controller/pod_volume_restore_controller.go:262" name=raffia-demo3-20200513105239-25ts4 namespace=velero restore=velero/raffia-demo3-20200513105239
time="2020-05-13T10:53:28Z" level=info msg="Restore completed" controller=pod-volume-restore logSource="pkg/controller/pod_volume_restore_controller.go:311" name=raffia-demo3-20200513105239-25ts4 namespace=velero restore=velero/raffia-demo3-20200513105239
time="2020-05-13T10:53:28Z" level=info msg="Restore starting" controller=pod-volume-restore logSource="pkg/controller/pod_volume_restore_controller.go:262" name=raffia-demo3-20200513105239-5h2xq namespace=velero restore=velero/raffia-demo3-20200513105239
time="2020-05-13T10:53:37Z" level=info msg="Restore completed" controller=pod-volume-restore logSource="pkg/controller/pod_volume_restore_controller.go:311" name=raffia-demo3-20200513105239-5h2xq namespace=velero restore=velero/raffia-demo3-20200513105239
time="2020-05-13T10:53:37Z" level=info msg="Restore starting" controller=pod-volume-restore logSource="pkg/controller/pod_volume_restore_controller.go:262" name=raffia-demo3-20200513105239-vjmjq namespace=velero restore=velero/raffia-demo3-20200513105239
time="2020-05-13T10:53:40Z" level=info msg="Restore completed" controller=pod-volume-restore logSource="pkg/controller/pod_volume_restore_controller.go:311" name=raffia-demo3-20200513105239-vjmjq namespace=velero restore=velero/raffia-demo3-20200513105239
time="2020-05-13T10:53:40Z" level=info msg="Restore starting" controller=pod-volume-restore logSource="pkg/controller/pod_volume_restore_controller.go:262" name=raffia-demo3-20200513105239-mkx9g namespace=velero restore=velero/raffia-demo3-20200513105239
time="2020-05-13T10:53:43Z" level=info msg="Restore completed" controller=pod-volume-restore logSource="pkg/controller/pod_volume_restore_controller.go:311" name=raffia-demo3-20200513105239-mkx9g namespace=velero restore=velero/raffia-demo3-20200513105239
time="2020-05-13T10:53:43Z" level=info msg="Restore starting" controller=pod-volume-restore logSource="pkg/controller/pod_volume_restore_controller.go:262" name=raffia-demo3-20200513105239-llxvn namespace=velero restore=velero/raffia-demo3-20200513105239
time="2020-05-13T10:53:46Z" level=info msg="Restore completed" controller=pod-volume-restore logSource="pkg/controller/pod_volume_restore_controller.go:311" name=raffia-demo3-20200513105239-llxvn namespace=velero restore=velero/raffia-demo3-20200513105239
time="2020-05-13T10:53:46Z" level=info msg="Restore starting" controller=pod-volume-restore logSource="pkg/controller/pod_volume_restore_controller.go:262" name=raffia-demo3-20200513105239-lk4mh namespace=velero restore=velero/raffia-demo3-20200513105239
time="2020-05-13T10:53:49Z" level=info msg="Restore completed" controller=pod-volume-restore logSource="pkg/controller/pod_volume_restore_controller.go:311" name=raffia-demo3-20200513105239-lk4mh namespace=velero restore=velero/raffia-demo3-20200513105239
time="2020-05-13T10:53:49Z" level=info msg="Restore starting" controller=pod-volume-restore logSource="pkg/controller/pod_volume_restore_controller.go:262" name=raffia-demo3-20200513105239-9wrrj namespace=velero restore=velero/raffia-demo3-20200513105239
time="2020-05-13T10:53:52Z" level=info msg="Restore completed" controller=pod-volume-restore logSource="pkg/controller/pod_volume_restore_controller.go:311" name=raffia-demo3-20200513105239-9wrrj namespace=velero restore=velero/raffia-demo3-20200513105239
time="2020-05-13T10:53:52Z" level=info msg="Restore starting" controller=pod-volume-restore logSource="pkg/controller/pod_volume_restore_controller.go:262" name=raffia-demo3-20200513105239-hqn8t namespace=velero restore=velero/raffia-demo3-20200513105239
time="2020-05-13T10:53:55Z" level=info msg="Restore completed" controller=pod-volume-restore logSource="pkg/controller/pod_volume_restore_controller.go:311" name=raffia-demo3-20200513105239-hqn8t namespace=velero restore=velero/raffia-demo3-20200513105239
time="2020-05-13T10:53:55Z" level=info msg="Restore starting" controller=pod-volume-restore logSource="pkg/controller/pod_volume_restore_controller.go:262" name=raffia-demo3-20200513105239-s8tjg namespace=velero restore=velero/raffia-demo3-20200513105239
time="2020-05-13T10:53:58Z" level=info msg="Restore completed" controller=pod-volume-restore logSource="pkg/controller/pod_volume_restore_controller.go:311" name=raffia-demo3-20200513105239-s8tjg namespace=velero restore=velero/raffia-demo3-20200513105239
time="2020-05-13T11:19:48Z" level=info msg="Backup starting" backup=velero/raffia-pr-1724 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:178" name=raffia-pr-1724-4pgdf namespace=velero
time="2020-05-13T11:19:48Z" level=info msg="Looking for most recent completed pod volume backup for this PVC" backup=velero/raffia-pr-1724 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:307" name=raffia-pr-1724-4pgdf namespace=velero pvcUID=efa100df-9507-11ea-8f88-02bc09c18c05
time="2020-05-13T11:19:48Z" level=info msg="No completed pod volume backup found for PVC" backup=velero/raffia-pr-1724 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:337" name=raffia-pr-1724-4pgdf namespace=velero pvcUID=efa100df-9507-11ea-8f88-02bc09c18c05
time="2020-05-13T11:19:48Z" level=info msg="No parent snapshot found for PVC, not using --parent flag for this backup" backup=velero/raffia-pr-1724 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:253" name=raffia-pr-1724-4pgdf namespace=velero
time="2020-05-13T11:19:58Z" level=info msg="Backup completed" backup=velero/raffia-pr-1724 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:297" name=raffia-pr-1724-4pgdf namespace=velero
time="2020-05-13T11:19:58Z" level=info msg="Backup starting" backup=velero/raffia-pr-1724 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:178" name=raffia-pr-1724-2nzgd namespace=velero
time="2020-05-13T11:20:01Z" level=info msg="Backup completed" backup=velero/raffia-pr-1724 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:297" name=raffia-pr-1724-2nzgd namespace=velero
time="2020-05-13T11:20:01Z" level=info msg="Backup starting" backup=velero/raffia-pr-1724 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:178" name=raffia-pr-1724-sd4xp namespace=velero
time="2020-05-13T11:20:04Z" level=info msg="Backup completed" backup=velero/raffia-pr-1724 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:297" name=raffia-pr-1724-sd4xp namespace=velero
time="2020-05-13T11:20:04Z" level=info msg="Backup starting" backup=velero/raffia-pr-1724 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:178" name=raffia-pr-1724-zwqqt namespace=velero
time="2020-05-13T11:20:04Z" level=info msg="Looking for most recent completed pod volume backup for this PVC" backup=velero/raffia-pr-1724 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:307" name=raffia-pr-1724-zwqqt namespace=velero pvcUID=efa3f3dc-9507-11ea-8f88-02bc09c18c05
time="2020-05-13T11:20:04Z" level=info msg="No completed pod volume backup found for PVC" backup=velero/raffia-pr-1724 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:337" name=raffia-pr-1724-zwqqt namespace=velero pvcUID=efa3f3dc-9507-11ea-8f88-02bc09c18c05
time="2020-05-13T11:20:04Z" level=info msg="No parent snapshot found for PVC, not using --parent flag for this backup" backup=velero/raffia-pr-1724 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:253" name=raffia-pr-1724-zwqqt namespace=velero
time="2020-05-13T11:20:06Z" level=info msg="Backup completed" backup=velero/raffia-pr-1724 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:297" name=raffia-pr-1724-zwqqt namespace=velero
time="2020-05-13T11:20:06Z" level=info msg="Backup starting" backup=velero/raffia-pr-1724 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:178" name=raffia-pr-1724-xvnlq namespace=velero
time="2020-05-13T11:20:06Z" level=info msg="Looking for most recent completed pod volume backup for this PVC" backup=velero/raffia-pr-1724 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:307" name=raffia-pr-1724-xvnlq namespace=velero pvcUID=efa89a58-9507-11ea-8f88-02bc09c18c05
time="2020-05-13T11:20:06Z" level=info msg="No completed pod volume backup found for PVC" backup=velero/raffia-pr-1724 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:337" name=raffia-pr-1724-xvnlq namespace=velero pvcUID=efa89a58-9507-11ea-8f88-02bc09c18c05
time="2020-05-13T11:20:06Z" level=info msg="No parent snapshot found for PVC, not using --parent flag for this backup" backup=velero/raffia-pr-1724 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:253" name=raffia-pr-1724-xvnlq namespace=velero
time="2020-05-13T11:20:12Z" level=info msg="Backup completed" backup=velero/raffia-pr-1724 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:297" name=raffia-pr-1724-xvnlq namespace=velero
time="2020-05-13T11:20:12Z" level=info msg="Backup starting" backup=velero/raffia-pr-1724 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:178" name=raffia-pr-1724-r6x2t namespace=velero
time="2020-05-13T11:20:12Z" level=info msg="Looking for most recent completed pod volume backup for this PVC" backup=velero/raffia-pr-1724 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:307" name=raffia-pr-1724-r6x2t namespace=velero pvcUID=efbcaa9c-9507-11ea-8f88-02bc09c18c05
time="2020-05-13T11:20:12Z" level=info msg="No completed pod volume backup found for PVC" backup=velero/raffia-pr-1724 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:337" name=raffia-pr-1724-r6x2t namespace=velero pvcUID=efbcaa9c-9507-11ea-8f88-02bc09c18c05
time="2020-05-13T11:20:12Z" level=info msg="No parent snapshot found for PVC, not using --parent flag for this backup" backup=velero/raffia-pr-1724 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:253" name=raffia-pr-1724-r6x2t namespace=velero
time="2020-05-13T11:20:15Z" level=info msg="Backup completed" backup=velero/raffia-pr-1724 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:297" name=raffia-pr-1724-r6x2t namespace=velero
time="2020-05-13T11:20:15Z" level=info msg="Backup starting" backup=velero/raffia-pr-1724 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:178" name=raffia-pr-1724-pqfw2 namespace=velero
time="2020-05-13T11:20:15Z" level=info msg="Looking for most recent completed pod volume backup for this PVC" backup=velero/raffia-pr-1724 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:307" name=raffia-pr-1724-pqfw2 namespace=velero pvcUID=efb8de9c-9507-11ea-8f88-02bc09c18c05
time="2020-05-13T11:20:15Z" level=info msg="No completed pod volume backup found for PVC" backup=velero/raffia-pr-1724 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:337" name=raffia-pr-1724-pqfw2 namespace=velero pvcUID=efb8de9c-9507-11ea-8f88-02bc09c18c05
time="2020-05-13T11:20:15Z" level=info msg="No parent snapshot found for PVC, not using --parent flag for this backup" backup=velero/raffia-pr-1724 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:253" name=raffia-pr-1724-pqfw2 namespace=velero
time="2020-05-13T11:20:16Z" level=info msg="Backup completed" backup=velero/raffia-pr-1724 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:297" name=raffia-pr-1724-pqfw2 namespace=velero
time="2020-05-13T11:20:16Z" level=info msg="Backup starting" backup=velero/raffia-pr-1724 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:178" name=raffia-pr-1724-hwgqv namespace=velero
time="2020-05-13T11:20:19Z" level=info msg="Backup completed" backup=velero/raffia-pr-1724 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:297" name=raffia-pr-1724-hwgqv namespace=velero
time="2020-05-13T11:20:19Z" level=info msg="Backup starting" backup=velero/raffia-pr-1724 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:178" name=raffia-pr-1724-k2qqb namespace=velero
time="2020-05-13T11:20:22Z" level=info msg="Backup completed" backup=velero/raffia-pr-1724 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:297" name=raffia-pr-1724-k2qqb namespace=velero
time="2020-05-13T11:20:22Z" level=info msg="Backup starting" backup=velero/raffia-pr-1724 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:178" name=raffia-pr-1724-pr7nr namespace=velero
time="2020-05-13T11:20:24Z" level=info msg="Backup completed" backup=velero/raffia-pr-1724 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:297" name=raffia-pr-1724-pr7nr namespace=velero
time="2020-05-13T11:20:24Z" level=info msg="Backup starting" backup=velero/raffia-pr-1724 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:178" name=raffia-pr-1724-2sf8k namespace=velero
time="2020-05-13T11:20:25Z" level=info msg="Looking for most recent completed pod volume backup for this PVC" backup=velero/raffia-pr-1724 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:307" name=raffia-pr-1724-2sf8k namespace=velero pvcUID=efab9589-9507-11ea-8f88-02bc09c18c05
time="2020-05-13T11:20:25Z" level=info msg="No completed pod volume backup found for PVC" backup=velero/raffia-pr-1724 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:337" name=raffia-pr-1724-2sf8k namespace=velero pvcUID=efab9589-9507-11ea-8f88-02bc09c18c05
time="2020-05-13T11:20:25Z" level=info msg="No parent snapshot found for PVC, not using --parent flag for this backup" backup=velero/raffia-pr-1724 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:253" name=raffia-pr-1724-2sf8k namespace=velero
time="2020-05-13T11:20:28Z" level=info msg="Backup completed" backup=velero/raffia-pr-1724 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:297" name=raffia-pr-1724-2sf8k namespace=velero
time="2020-05-13T11:20:28Z" level=info msg="Backup starting" backup=velero/raffia-pr-1724 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:178" name=raffia-pr-1724-p4zt2 namespace=velero
time="2020-05-13T11:20:28Z" level=info msg="Looking for most recent completed pod volume backup for this PVC" backup=velero/raffia-pr-1724 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:307" name=raffia-pr-1724-p4zt2 namespace=velero pvcUID=efb5ff09-9507-11ea-8f88-02bc09c18c05
time="2020-05-13T11:20:28Z" level=info msg="No completed pod volume backup found for PVC" backup=velero/raffia-pr-1724 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:337" name=raffia-pr-1724-p4zt2 namespace=velero pvcUID=efb5ff09-9507-11ea-8f88-02bc09c18c05
time="2020-05-13T11:20:28Z" level=info msg="No parent snapshot found for PVC, not using --parent flag for this backup" backup=velero/raffia-pr-1724 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:253" name=raffia-pr-1724-p4zt2 namespace=velero
time="2020-05-13T11:20:30Z" level=info msg="Backup completed" backup=velero/raffia-pr-1724 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:297" name=raffia-pr-1724-p4zt2 namespace=velero
time="2020-05-13T11:20:30Z" level=info msg="Backup starting" backup=velero/raffia-pr-1724 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:178" name=raffia-pr-1724-xrnjb namespace=velero
time="2020-05-13T11:20:30Z" level=info msg="Looking for most recent completed pod volume backup for this PVC" backup=velero/raffia-pr-1724 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:307" name=raffia-pr-1724-xrnjb namespace=velero pvcUID=efae685e-9507-11ea-8f88-02bc09c18c05
time="2020-05-13T11:20:30Z" level=info msg="No completed pod volume backup found for PVC" backup=velero/raffia-pr-1724 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:337" name=raffia-pr-1724-xrnjb namespace=velero pvcUID=efae685e-9507-11ea-8f88-02bc09c18c05
time="2020-05-13T11:20:30Z" level=info msg="No parent snapshot found for PVC, not using --parent flag for this backup" backup=velero/raffia-pr-1724 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:253" name=raffia-pr-1724-xrnjb namespace=velero
time="2020-05-13T11:20:34Z" level=info msg="Backup completed" backup=velero/raffia-pr-1724 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:297" name=raffia-pr-1724-xrnjb namespace=velero
time="2020-05-13T11:20:34Z" level=info msg="Backup starting" backup=velero/raffia-pr-1724 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:178" name=raffia-pr-1724-tpwcw namespace=velero
time="2020-05-13T11:20:34Z" level=info msg="Looking for most recent completed pod volume backup for this PVC" backup=velero/raffia-pr-1724 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:307" name=raffia-pr-1724-tpwcw namespace=velero pvcUID=efb1151c-9507-11ea-8f88-02bc09c18c05
time="2020-05-13T11:20:34Z" level=info msg="No completed pod volume backup found for PVC" backup=velero/raffia-pr-1724 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:337" name=raffia-pr-1724-tpwcw namespace=velero pvcUID=efb1151c-9507-11ea-8f88-02bc09c18c05
time="2020-05-13T11:20:34Z" level=info msg="No parent snapshot found for PVC, not using --parent flag for this backup" backup=velero/raffia-pr-1724 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:253" name=raffia-pr-1724-tpwcw namespace=velero
time="2020-05-13T11:20:36Z" level=info msg="Backup completed" backup=velero/raffia-pr-1724 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:297" name=raffia-pr-1724-tpwcw namespace=velero
time="2020-05-13T11:20:36Z" level=info msg="Backup starting" backup=velero/raffia-pr-1724 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:178" name=raffia-pr-1724-hfrgx namespace=velero
time="2020-05-13T11:20:39Z" level=info msg="Backup completed" backup=velero/raffia-pr-1724 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:297" name=raffia-pr-1724-hfrgx namespace=velero
time="2020-05-13T11:20:39Z" level=info msg="Backup starting" backup=velero/raffia-pr-1724 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:178" name=raffia-pr-1724-gv99j namespace=velero
time="2020-05-13T11:20:41Z" level=info msg="Backup completed" backup=velero/raffia-pr-1724 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:297" name=raffia-pr-1724-gv99j namespace=velero

So far I have only been able to restore the PV's after creating them first via kubectl apply -f before running the restore.

I did notice when I tried on another more simple app with one PV I didnt have to create the PV first, the restore took care of that - but i don't know if that was a one-off - i can try that again though.

Does restic struggle with multiple PV's to one app?

edmund-nelson commented 4 years ago

Hi - is anyone still looking at this please?

skriss commented 4 years ago

@eddeh83 here's what I think is going on: when velero goes to restore a PV using restic, the first thing it does is dynamically provision a new, empty PV. It does this by creating a PVC, and then waiting for a storage provisioner to create a PV and bind it to the PVC.

If I'm not mistaken, local PVs are not usually dynamically provisioned. So, there's no provisioner that is processing the PVC that's waiting for a PV. As a result, velero never has a volume to restore into.

You said that if you manually (statically) provisioned a PV with the right name before the restore, then it worked - which would make sense here, since the velero-restored PVC would be able to bind to that PV that you created, and then restore data into it.

Does that match your observations? We can get into possible solutions/fixes next, but just want to make sure I'm understanding the problem first.

edmund-nelson commented 4 years ago

Hi @skriss - thanks for getting back to me

That does sound plausible - and a test I've done since raising the issue does suggest it's some kind of conflict with restic that causes the issue.

To test I did what I previously did - backup the whole cluster, delete the raffia namespace and pv's then attempt a restore from the full cluster backup. The behavior I'd seen before was the same (as an addendum, this is now using 1.4.0 rather than 1.3.2 previously)

I also created a pv only backup which I'll get to later, but this didnt include any restic resources.

velero backup create full-cluster

velero backup create pv-only --include-resources pv

for i in `kubectl get pv | grep raffia | awk '{print $1}'`
> do
> kubectl delete pv $i
> done
persistentvolume "raffia-config" deleted
persistentvolume "raffia-domains" deleted
persistentvolume "raffia-engine-config" deleted
persistentvolume "raffia-engine-home" deleted
persistentvolume "raffia-engine-triage" deleted
persistentvolume "raffia-engine-var" deleted
persistentvolume "raffia-home" deleted
persistentvolume "raffia-mongodb-data" deleted
persistentvolume "raffia-raffia-var" deleted

kubectl delete namespace raffia
namespace "raffia" deleted

velero restore create --from-backup full-cluster

velero restore describe full-cluster-20200611120024 --details
Name:         full-cluster-20200611120024
Namespace:    velero
Labels:       <none>
Annotations:  <none>

Phase:  InProgress

Backup:  full-cluster

Namespaces:
  Included:  all namespaces found in the backup
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        nodes, events, events.events.k8s.io, backups.velero.io, restores.velero.io, resticrepositories.velero.io
  Cluster-scoped:  auto

Namespace mappings:  <none>

Label selector:  <none>

Restore PVs:  auto

Restic Restores:
  New:
    raffia/raffia-mongodb-5d47b57cbd-6rfnm: data
    raffia/raffia-raffia-54f7475668-hwgnz: raffia-config, raffia-domains, raffia-scripts, raffia-var, vaultca
    raffia/raffia-raffia-engine-5c5f6669b6-rr2t9: raffia-engine-scripts, raffiaengine-config, raffiaengine-home, raffiaengine-var, vaultca

The restore pauses with the above and times out after an hour

checking resources on the cluster...

kubectl get all -n raffia
NAME                                        READY   STATUS    RESTARTS   AGE
pod/raffia-mongodb-5d47b57cbd-6rfnm         0/1     Pending   0          23m
pod/raffia-raffia-54f7475668-hwgnz          0/1     Pending   0          23m
pod/raffia-raffia-engine-5c5f6669b6-rr2t9   0/1     Pending   0          23m

NAME                     TYPE        CLUSTER-IP        EXTERNAL-IP   PORT(S)     AGE
service/raffia-mongodb   ClusterIP   130.174.241.246   <none>        27017/TCP   22m
service/raffia-raffia    ClusterIP   130.174.240.172   <none>        3030/TCP    22m

NAME                                   READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/raffia-mongodb         0/1     1            0           22m
deployment.apps/raffia-raffia          0/1     1            0           22m
deployment.apps/raffia-raffia-engine   0/1     1            0           22m

NAME                                              DESIRED   CURRENT   READY   AGE
replicaset.apps/raffia-mongodb-54ccbdcf87         0         0         0       23m
replicaset.apps/raffia-mongodb-5d47b57cbd         1         1         0       23m
replicaset.apps/raffia-raffia-54f7475668          1         1         0       23m
replicaset.apps/raffia-raffia-585dfd7d96          0         0         0       23m
replicaset.apps/raffia-raffia-engine-5c5f6669b6   1         1         0       23m
replicaset.apps/raffia-raffia-engine-858bc9dc44   0         0         0       23m

kubectl get pvc -n raffia
NAME                          STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS    AGE
raffia-mongodb                Pending                                      local-storage   24m
raffia-raffia-config          Pending                                      local-storage   24m
raffia-raffia-domains         Pending                                      local-storage   24m
raffia-raffia-engine-config   Pending                                      local-storage   24m
raffia-raffia-engine-home     Pending                                      local-storage   24m
raffia-raffia-engine-triage   Pending                                      local-storage   24m
raffia-raffia-engine-var      Pending                                      local-storage   24m
raffia-raffia-home            Pending                                      local-storage   24m
raffia-raffia-var             Pending                                      local-storage   24m

kubectl get pv | grep raffia
*no results*

I deleted everything to do with raffia and the namespace again, then carried out a restore from the pv-only backup, then the cluster backup

velero restore create --from-backup pv-only

kubectl get pv | grep raffia
raffia-config                              1M         RWO            Retain           Available                                        local-storage                 60s
raffia-domains                             2Gi        RWO            Retain           Available                                        local-storage                 60s
raffia-engine-config                       1M         RWO            Retain           Available                                        local-storage                 60s
raffia-engine-home                         10M        RWO            Retain           Available                                        local-storage                 60s
raffia-engine-triage                       2Gi        RWO            Retain           Available                                        local-storage                 60s
raffia-engine-var                          1M         RWO            Retain           Available                                        local-storage                 59s
raffia-home                                1M         RWO            Retain           Available                                        local-storage                 59s
raffia-mongodb-data                        8Gi        RWO            Retain           Available                                        local-storage                 59s
raffia-raffia-var                          2Gi        RWO            Retain           Available                                        local-storage                 59s

velero restore create --from-backup full-cluster

kubectl get all -n raffia
NAME                                        READY   STATUS    RESTARTS   AGE
pod/raffia-mongodb-5d47b57cbd-6rfnm         1/1     Running   0          3m8s
pod/raffia-raffia-54f7475668-hwgnz          1/1     Running   0          3m7s
pod/raffia-raffia-engine-5c5f6669b6-rr2t9   1/1     Running   0          3m7s

NAME                     TYPE        CLUSTER-IP        EXTERNAL-IP   PORT(S)     AGE
service/raffia-mongodb   ClusterIP   130.174.240.53    <none>        27017/TCP   2m1s
service/raffia-raffia    ClusterIP   130.174.240.150   <none>        3030/TCP    2m1s

NAME                                   READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/raffia-mongodb         1/1     1            1           2m25s
deployment.apps/raffia-raffia          1/1     1            1           2m25s
deployment.apps/raffia-raffia-engine   1/1     1            1           2m25s

NAME                                              DESIRED   CURRENT   READY   AGE
replicaset.apps/raffia-mongodb-54ccbdcf87         0         0         0       2m56s
replicaset.apps/raffia-mongodb-5d47b57cbd         1         1         1       2m56s
replicaset.apps/raffia-raffia-54f7475668          1         1         1       2m56s
replicaset.apps/raffia-raffia-585dfd7d96          0         0         0       2m56s
replicaset.apps/raffia-raffia-engine-5c5f6669b6   1         1         1       2m56s
replicaset.apps/raffia-raffia-engine-858bc9dc44   0         0         0       2m56s

kubectl get pv | grep raffia
raffia-config                              1M         RWO            Retain           Bound    raffia/raffia-raffia-config          local-storage                 6m4s
raffia-domains                             2Gi        RWO            Retain           Bound    raffia/raffia-raffia-domains         local-storage                 6m4s
raffia-engine-config                       1M         RWO            Retain           Bound    raffia/raffia-raffia-engine-config   local-storage                 6m4s
raffia-engine-home                         10M        RWO            Retain           Bound    raffia/raffia-raffia-engine-home     local-storage                 6m4s
raffia-engine-triage                       2Gi        RWO            Retain           Bound    raffia/raffia-raffia-engine-triage   local-storage                 6m4s
raffia-engine-var                          1M         RWO            Retain           Bound    raffia/raffia-raffia-engine-var      local-storage                 6m3s
raffia-home                                1M         RWO            Retain           Bound    raffia/raffia-raffia-home            local-storage                 6m3s
raffia-mongodb-data                        8Gi        RWO            Retain           Bound    raffia/raffia-mongodb                local-storage                 6m3s
raffia-raffia-var                          2Gi        RWO            Retain           Bound    raffia/raffia-raffia-var             local-storage                 6m3s

Everything is back

the log for the pv restore:-

velero restore logs pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="starting restore" logSource="pkg/controller/restore_controller.go:453" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="Starting restore of backup velero/pv-only" logSource="pkg/restore/restore.go:345" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="Restoring cluster level resource 'persistentvolumes'" logSource="pkg/restore/restore.go:704" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="Getting client for /v1, Kind=PersistentVolume" logSource="pkg/restore/restore.go:746" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="Restoring persistent volume as-is because it doesn't have a snapshot and its reclaim policy is not Delete." logSource="pkg/restore/restore.go:940" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="No snapshot found for persistent volume" logSource="pkg/restore/pv_restorer.go:81" persistentVolume=datadir-vault-consul-0 restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="Executing item action for persistentvolumes" logSource="pkg/restore/restore.go:964" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="Executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:63" pluginName=velero restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="Done executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:74" pluginName=velero restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="Attempting to restore PersistentVolume: datadir-vault-consul-0" logSource="pkg/restore/restore.go:1070" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="Restoring persistent volume as-is because it doesn't have a snapshot and its reclaim policy is not Delete." logSource="pkg/restore/restore.go:940" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="No snapshot found for persistent volume" logSource="pkg/restore/pv_restorer.go:81" persistentVolume=datadir-vault-consul-1 restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="Executing item action for persistentvolumes" logSource="pkg/restore/restore.go:964" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="Executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:63" pluginName=velero restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="Done executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:74" pluginName=velero restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="Attempting to restore PersistentVolume: datadir-vault-consul-1" logSource="pkg/restore/restore.go:1070" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="Restoring persistent volume as-is because it doesn't have a snapshot and its reclaim policy is not Delete." logSource="pkg/restore/restore.go:940" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="No snapshot found for persistent volume" logSource="pkg/restore/pv_restorer.go:81" persistentVolume=datadir-vault-consul-2 restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="Executing item action for persistentvolumes" logSource="pkg/restore/restore.go:964" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="Executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:63" pluginName=velero restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="Done executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:74" pluginName=velero restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="Attempting to restore PersistentVolume: datadir-vault-consul-2" logSource="pkg/restore/restore.go:1070" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="Restoring persistent volume as-is because it doesn't have a snapshot and its reclaim policy is not Delete." logSource="pkg/restore/restore.go:940" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="No snapshot found for persistent volume" logSource="pkg/restore/pv_restorer.go:81" persistentVolume=datadir-vault-vault-audit-logs-0 restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="Executing item action for persistentvolumes" logSource="pkg/restore/restore.go:964" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="Executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:63" pluginName=velero restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="Done executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:74" pluginName=velero restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="Attempting to restore PersistentVolume: datadir-vault-vault-audit-logs-0" logSource="pkg/restore/restore.go:1070" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="Restoring persistent volume as-is because it doesn't have a snapshot and its reclaim policy is not Delete." logSource="pkg/restore/restore.go:940" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="No snapshot found for persistent volume" logSource="pkg/restore/pv_restorer.go:81" persistentVolume=datadir-vault-vault-audit-logs-1 restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="Executing item action for persistentvolumes" logSource="pkg/restore/restore.go:964" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="Executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:63" pluginName=velero restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="Done executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:74" pluginName=velero restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="Attempting to restore PersistentVolume: datadir-vault-vault-audit-logs-1" logSource="pkg/restore/restore.go:1070" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="Restoring persistent volume as-is because it doesn't have a snapshot and its reclaim policy is not Delete." logSource="pkg/restore/restore.go:940" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="No snapshot found for persistent volume" logSource="pkg/restore/pv_restorer.go:81" persistentVolume=datadir-vault-vault-audit-logs-2 restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="Executing item action for persistentvolumes" logSource="pkg/restore/restore.go:964" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="Executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:63" pluginName=velero restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="Done executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:74" pluginName=velero restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="Attempting to restore PersistentVolume: datadir-vault-vault-audit-logs-2" logSource="pkg/restore/restore.go:1070" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="Restoring persistent volume as-is because it doesn't have a snapshot and its reclaim policy is not Delete." logSource="pkg/restore/restore.go:940" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="No snapshot found for persistent volume" logSource="pkg/restore/pv_restorer.go:81" persistentVolume=logstash-config restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="Executing item action for persistentvolumes" logSource="pkg/restore/restore.go:964" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="Executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:63" pluginName=velero restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="Done executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:74" pluginName=velero restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="Attempting to restore PersistentVolume: logstash-config" logSource="pkg/restore/restore.go:1070" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="Restoring persistent volume as-is because it doesn't have a snapshot and its reclaim policy is not Delete." logSource="pkg/restore/restore.go:940" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="No snapshot found for persistent volume" logSource="pkg/restore/pv_restorer.go:81" persistentVolume=logstash-data restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="Executing item action for persistentvolumes" logSource="pkg/restore/restore.go:964" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="Executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:63" pluginName=velero restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="Done executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:74" pluginName=velero restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="Attempting to restore PersistentVolume: logstash-data" logSource="pkg/restore/restore.go:1070" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="Restoring persistent volume as-is because it doesn't have a snapshot and its reclaim policy is not Delete." logSource="pkg/restore/restore.go:940" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="No snapshot found for persistent volume" logSource="pkg/restore/pv_restorer.go:81" persistentVolume=nagios-configs restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="Executing item action for persistentvolumes" logSource="pkg/restore/restore.go:964" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="Executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:63" pluginName=velero restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="Done executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:74" pluginName=velero restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="Attempting to restore PersistentVolume: nagios-configs" logSource="pkg/restore/restore.go:1070" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="Restoring persistent volume as-is because it doesn't have a snapshot and its reclaim policy is not Delete." logSource="pkg/restore/restore.go:940" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="No snapshot found for persistent volume" logSource="pkg/restore/pv_restorer.go:81" persistentVolume=nagios-last-servers restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="Executing item action for persistentvolumes" logSource="pkg/restore/restore.go:964" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="Executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:63" pluginName=velero restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="Done executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:74" pluginName=velero restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="Attempting to restore PersistentVolume: nagios-last-servers" logSource="pkg/restore/restore.go:1070" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="Restoring persistent volume as-is because it doesn't have a snapshot and its reclaim policy is not Delete." logSource="pkg/restore/restore.go:940" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="No snapshot found for persistent volume" logSource="pkg/restore/pv_restorer.go:81" persistentVolume=nagios-servers restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="Executing item action for persistentvolumes" logSource="pkg/restore/restore.go:964" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="Executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:63" pluginName=velero restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="Done executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:74" pluginName=velero restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="Attempting to restore PersistentVolume: nagios-servers" logSource="pkg/restore/restore.go:1070" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="Dynamically re-provisioning persistent volume because it doesn't have a snapshot and its reclaim policy is Delete." logSource="pkg/restore/restore.go:933" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="Dynamically re-provisioning persistent volume because it doesn't have a snapshot and its reclaim policy is Delete." logSource="pkg/restore/restore.go:933" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="Restoring persistent volume as-is because it doesn't have a snapshot and its reclaim policy is not Delete." logSource="pkg/restore/restore.go:940" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="No snapshot found for persistent volume" logSource="pkg/restore/pv_restorer.go:81" persistentVolume=raffia-config restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="Executing item action for persistentvolumes" logSource="pkg/restore/restore.go:964" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:22Z" level=info msg="Executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:63" pluginName=velero restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:23Z" level=info msg="Done executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:74" pluginName=velero restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:23Z" level=info msg="Attempting to restore PersistentVolume: raffia-config" logSource="pkg/restore/restore.go:1070" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:23Z" level=info msg="Restoring persistent volume as-is because it doesn't have a snapshot and its reclaim policy is not Delete." logSource="pkg/restore/restore.go:940" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:23Z" level=info msg="No snapshot found for persistent volume" logSource="pkg/restore/pv_restorer.go:81" persistentVolume=raffia-domains restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:23Z" level=info msg="Executing item action for persistentvolumes" logSource="pkg/restore/restore.go:964" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:23Z" level=info msg="Executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:63" pluginName=velero restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:23Z" level=info msg="Done executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:74" pluginName=velero restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:23Z" level=info msg="Attempting to restore PersistentVolume: raffia-domains" logSource="pkg/restore/restore.go:1070" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:23Z" level=info msg="Restoring persistent volume as-is because it doesn't have a snapshot and its reclaim policy is not Delete." logSource="pkg/restore/restore.go:940" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:23Z" level=info msg="No snapshot found for persistent volume" logSource="pkg/restore/pv_restorer.go:81" persistentVolume=raffia-engine-config restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:23Z" level=info msg="Executing item action for persistentvolumes" logSource="pkg/restore/restore.go:964" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:23Z" level=info msg="Executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:63" pluginName=velero restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:23Z" level=info msg="Done executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:74" pluginName=velero restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:23Z" level=info msg="Attempting to restore PersistentVolume: raffia-engine-config" logSource="pkg/restore/restore.go:1070" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:23Z" level=info msg="Restoring persistent volume as-is because it doesn't have a snapshot and its reclaim policy is not Delete." logSource="pkg/restore/restore.go:940" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:23Z" level=info msg="No snapshot found for persistent volume" logSource="pkg/restore/pv_restorer.go:81" persistentVolume=raffia-engine-home restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:23Z" level=info msg="Executing item action for persistentvolumes" logSource="pkg/restore/restore.go:964" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:23Z" level=info msg="Executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:63" pluginName=velero restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:23Z" level=info msg="Done executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:74" pluginName=velero restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:23Z" level=info msg="Attempting to restore PersistentVolume: raffia-engine-home" logSource="pkg/restore/restore.go:1070" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:23Z" level=info msg="Restoring persistent volume as-is because it doesn't have a snapshot and its reclaim policy is not Delete." logSource="pkg/restore/restore.go:940" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:23Z" level=info msg="No snapshot found for persistent volume" logSource="pkg/restore/pv_restorer.go:81" persistentVolume=raffia-engine-triage restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:23Z" level=info msg="Executing item action for persistentvolumes" logSource="pkg/restore/restore.go:964" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:23Z" level=info msg="Executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:63" pluginName=velero restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:23Z" level=info msg="Done executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:74" pluginName=velero restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:23Z" level=info msg="Attempting to restore PersistentVolume: raffia-engine-triage" logSource="pkg/restore/restore.go:1070" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:23Z" level=info msg="Restoring persistent volume as-is because it doesn't have a snapshot and its reclaim policy is not Delete." logSource="pkg/restore/restore.go:940" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:23Z" level=info msg="No snapshot found for persistent volume" logSource="pkg/restore/pv_restorer.go:81" persistentVolume=raffia-engine-var restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:23Z" level=info msg="Executing item action for persistentvolumes" logSource="pkg/restore/restore.go:964" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:23Z" level=info msg="Executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:63" pluginName=velero restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:24Z" level=info msg="Done executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:74" pluginName=velero restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:24Z" level=info msg="Attempting to restore PersistentVolume: raffia-engine-var" logSource="pkg/restore/restore.go:1070" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:24Z" level=info msg="Restoring persistent volume as-is because it doesn't have a snapshot and its reclaim policy is not Delete." logSource="pkg/restore/restore.go:940" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:24Z" level=info msg="No snapshot found for persistent volume" logSource="pkg/restore/pv_restorer.go:81" persistentVolume=raffia-home restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:24Z" level=info msg="Executing item action for persistentvolumes" logSource="pkg/restore/restore.go:964" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:24Z" level=info msg="Executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:63" pluginName=velero restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:24Z" level=info msg="Done executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:74" pluginName=velero restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:24Z" level=info msg="Attempting to restore PersistentVolume: raffia-home" logSource="pkg/restore/restore.go:1070" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:24Z" level=info msg="Restoring persistent volume as-is because it doesn't have a snapshot and its reclaim policy is not Delete." logSource="pkg/restore/restore.go:940" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:24Z" level=info msg="No snapshot found for persistent volume" logSource="pkg/restore/pv_restorer.go:81" persistentVolume=raffia-mongodb-data restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:24Z" level=info msg="Executing item action for persistentvolumes" logSource="pkg/restore/restore.go:964" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:24Z" level=info msg="Executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:63" pluginName=velero restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:24Z" level=info msg="Done executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:74" pluginName=velero restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:24Z" level=info msg="Attempting to restore PersistentVolume: raffia-mongodb-data" logSource="pkg/restore/restore.go:1070" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:24Z" level=info msg="Restoring persistent volume as-is because it doesn't have a snapshot and its reclaim policy is not Delete." logSource="pkg/restore/restore.go:940" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:24Z" level=info msg="No snapshot found for persistent volume" logSource="pkg/restore/pv_restorer.go:81" persistentVolume=raffia-raffia-var restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:24Z" level=info msg="Executing item action for persistentvolumes" logSource="pkg/restore/restore.go:964" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:24Z" level=info msg="Executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:63" pluginName=velero restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:24Z" level=info msg="Done executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:74" pluginName=velero restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:24Z" level=info msg="Attempting to restore PersistentVolume: raffia-raffia-var" logSource="pkg/restore/restore.go:1070" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:24Z" level=info msg="Waiting for all restic restores to complete" logSource="pkg/restore/restore.go:470" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:24Z" level=info msg="Done waiting for all restic restores to complete" logSource="pkg/restore/restore.go:486" restore=velero/pv-only-20200611122854
time="2020-06-11T12:30:24Z" level=info msg="restore completed" logSource="pkg/controller/restore_controller.go:468" restore=velero/pv-only-20200611122854

So perhaps there needs to be logic when carrying out a restore from a full cluster backup that the pv's be done first?

Also to bear in mind the pv's arent namespaced

skriss commented 4 years ago

How do you usually create the PVs that your app uses, before Velero even comes into the picture? Do you "statically provision" them by directly applying PV YAML to your cluster, then creating the PVCs to claim them?

edmund-nelson commented 4 years ago

@skriss Yeah they are created directly on the cluster via their own yaml

skriss commented 4 years ago

OK, that makes sense. So I think I at least understand this issue.

The restic restore code starts by attempting to dynamically provision a new, empty PV that it can restore into. It does this by creating a PVC, and then waiting for an external provisioner to create a PV + underlying storage disk and bind it to the PVC. After the bind is complete and the pod using the PVC can start, the restic restore can actually begin writing data to the new, empty disk.

In your case (and likely for all local PV users), the PV is statically provisioned and there is no dynamic provisioner. So, when the velero code creates a PVC and waits for an external provisioner to create a PV, nothing happens, and velero eventually times out and does not perform a restic restore.

Will need to think some more about if/how we can address this. The two-step restore process that you described (first restore the PVs themselves, then restore everything else) sounds workable. The one pitfall I can think of is that after step 1, if the restored PVs are pointing to an existing location on the host, you might have pre-existing data in there that you don't want. Ideally, after step 1, the PVs would be empty so velero/restic could do a clean restore into them.

edmund-nelson commented 4 years ago

Thanks @skriss

It would be good if as part of the restore, that before the pv's come down existing data on the pv's were purged - or at least velero/restic was able to look and decipher if there are any differences between the data on disk and in the backup - and decide whether to restore or not - an idea?

In the meantime it isnt too much of a problem for us to document the backup and restore process at our end to include two seperate backups and two restores - its workable for now

aaronbernardino commented 4 years ago

I believe I encountered the same issue but I wasn't using "local" PVs. I had multiple NFS backed PVs and only 1 was getting restored. The workaround to have a pv-only backup and restore worked for me as well.
However, I did notice a related issue if I try to backup and restore with PVs in 1 go:

So restoring my PVC gets stuck in pending with the following error:

Type       Reason                   Age                      From                                     Message
 ----        ------                      ----                       ----                                       -------
Warning  ProvisioningFailed  4s (x3 over 31s)  persistentvolume-controller  storageclass.storage.k8s.io "cfg" not found

The thing is "cfg" is not a real dynamic storage class. We have static PV and PVCs with "cfg" as storageClassName used to logically bind our PV and PVCs of a certain "class". Our PVs are still statically provisioned. By the way this error goes away if I remove using storageClassNames with the static PV and PVCs. It will then still stay in Pending because the PV does not get restored due to the original problem.

maikelthedev commented 4 years ago

I've got this exact same problem. The only solution that works for me is to create the PV (local volume) first then restore with Velero.

cloudcafetech commented 3 years ago

Same issue my PVs are NFS backed PVs. Restic not restoring snapshot. Instead its creating new one.

asoltesz commented 3 years ago

I have the same issue but with OpenEBS local volumes (openebs-hostpath).

The backup itself happens properly. (I have checked the S3 bucket and the content seems to be there based on file sizes).

Restore simply creates new, empty PVs (not even from the same storage class as it was before backed up) and doesn't run Restic on them. It should:

asoltesz commented 3 years ago

@skriss This seems to be a serious bug. Can we expect a fix?

ashish-amarnath commented 3 years ago

@asoltesz Steve doesn't work on the project. We will investigate this and prioritize it accordingly.

travnewmatic commented 3 years ago

i'm also experiencing this issue, though i'm trying to change storageclasses in the process.

source pv is using 'local-storage' storageclass destination is using 'longhorn' storageclass

the backup appears to be created properly

the restore will create a longhorn pv/pvc.. but its empty. and neither restic nor velero pods seem to be busy doing anything (like moving data around)

i've created a configmap to change storageclasses per the documentation https://velero.io/docs/v1.5/restore-reference/#changing-pvpvc-storage-classes

asoltesz commented 3 years ago

@ashish-amarnath : Thanks. This is currently a blocker for me so any workaround is appreciated.

maikelthedev commented 3 years ago

What work for me (local storage is what I use) is to create first the local storage (different yaml file with the pv and pvc) then restore a velero backup. This is the only way I've managed to get the backed up files back in there from a blank folder.

It's quite annoying though. It shouldn't be a hack.

Get Outlook for Androidhttps://aka.ms/ghei36

From: asoltesz notifications@github.com Sent: Saturday, November 21, 2020 12:51:08 PM To: vmware-tanzu/velero velero@noreply.github.com Cc: Miguel Frias Mosquea maikel@maikel.uk; Comment comment@noreply.github.com Subject: Re: [vmware-tanzu/velero] restic: unable to restore "Local" Persistent Volumes since they're statically provisioned (#2520)

@ashish-amarnathhttps://github.com/ashish-amarnath : Thanks. This is currently a blocker for me so any workaround is appreciated.

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/vmware-tanzu/velero/issues/2520#issuecomment-731568472, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AHAYX5N5CTOPPYTN5XEYVALSQ6SSZANCNFSM4M4CDHIQ.

eleanor-millman commented 3 years ago

Closing because workaround is to use the CSI driver to dynamically provision the volumes.

eleanor-millman commented 3 years ago

Duplicate of #1927

AndrewSav commented 3 years ago

@eleanor-millman

Closing because workaround is to use the CSI driver to dynamically provision the volumes.

Could you please explain what you mean by that?

eleanor-millman commented 3 years ago

@dsu-igeek Are you able to answer Andrew's question? I don't know enough of the details.

AndrewSav commented 3 years ago

@eleanor-millman @dsu-igeek from what I can tell, this may be relevant to the linked #1927 which uses nfs, but I cannot see how this can be relevant to this issue which is about local. There is no CSI driver for local, that I'm aware of and local does NOT allow to dynamically provision volumes. Could you please re-open until this could be clarified?

dsu-igeek commented 3 years ago

The issue is that we have no way to create the volumes via the Velero restore process. This isn't really a supported use case for us.

AndrewSav commented 3 years ago

@dsu-igeek I do not understand why not? Why kubeclt apply -f can, but velero, from the same backed up manifest cannot? What's the fundamental issue?

pupseba commented 2 years ago

Good question @AndrewSav I would also asume that the PV manifest should be restored even for manually provisioned PVs. This is a minimum imho and no different as just a "kubectl apply -f pv.yaml" definition.

AndrewSav commented 2 years ago

I think this should be re-opened because this is an issue that was not fixed and it is not clear at all what the problem is with getting the fix @dsu-igeek @eleanor-millman

eleanor-millman commented 2 years ago

The reason we want to leave it closed is that Velero does not support statically provisioned PV in restore process involving restic (https://github.com/vmware-tanzu/velero/issues/2520#issuecomment-642941806). This is how Velero currently works. Frankly, I'm not even sure if it should be marked a bug. Since we don't currently see a quick code solution and since a workaround exists (https://github.com/vmware-tanzu/velero/issues/2520#issuecomment-696254782), this is low priority compared to a number of items on our roadmap. So we would not be able to work on it for the next year or two.

If we do hear from a number of users that this affects them, then we can make it a higher priority. Also, if you have any interest in working on this, then please come to a community meeting and we can discuss whether there is a design that would make sense.

eleanor-millman commented 2 years ago

The team talked a bit more and we will reopen this ticket and put it in the icebox. This is to acknowledge that some users are still feeling pain about this and would like a solution, but also that there are higher priority items that the team needs to tackle first.

pupseba commented 2 years ago

During testing, I can confirm the following using restic for manually provisioned disks:

  1. Restoring a static provisioned PV will not happen automatically when a restore operation starts
  2. The PV manifest is saved as a .json in the backup files
  3. Manually applying that .json into the cluster will allow the restore operation to move forward
  4. All existing files in the PV backend will be overriten by the files in the backup
  5. Files that exist in the PV backend but are not present in the backup, will not be deleted
  6. During restore, the "path" of the PV can be changed to something else, allowing a restore of the whole PV contents in a different place (backend). This place will need to be manually provisioned before (like creating an nfs share or such)

Based on these, if this feature ever takes place, it would be nice to have the possibility to define a different path (and server?) for the PV, like we can do with mapping namespaces.

Hope this help others if they see this issue. Thank you @eleanor-millman for reopening this one, it will certainly be a nice to have feature (if possible) in the future.