search

vmware-tanzu / velero

Backup and migrate Kubernetes applications and their persistent volumes
https://velero.io
Apache License 2.0
8.61k stars 1.39k forks source link

Unable to perform restore via CSI driver: Failed to get volumesnapshots.snapshot.storage.k8s.io not found #7444

Closed jkroepke closed 7 months ago

jkroepke commented 7 months ago

What steps did you take and what happened:

I have 2 Storage Class in my cluster. Both Storage Classes has separate providers. And both VolumeSnapshotClasses are annotated:

% kubectl get volumesnapshotclass --show-labels
NAME            DRIVER               DELETIONPOLICY   AGE    LABELS
opsstack        disk.csi.azure.com   Delete           116d   app.kubernetes.io/managed-by=Helm,velero.io/csi-volumesnapshot-class=true
opsstack-file   file.csi.azure.com   Delete           143m   app.kubernetes.io/managed-by=Helm,velero.io/csi-volumesnapshot-class=true

If I execute

velero -n opsstack restore create --from-backup opsstack-velero-hourly-opsstack-20240219173046 -l app.kubernetes.io/name=opsstack-loki --include-resources pvc,pv

I'm getting this error:

{"level":"error","logSource":"pkg/controller/restore_controller.go:580","msg":"Namespace opsstack, resource restore error: error preparing persistentvolumeclaims/opsstack/storage-opsstack-loki-0: rpc error: code = Unknown desc = Failed to get Volumesnapshot opsstack/velero-storage-opsstack-loki-0-r67lz to restore PVC opsstack/storage-opsstack-loki-0: volumesnapshots.snapshot.storage.k8s.io \"velero-storage-opsstack-loki-0-r67lz\" not found","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"}
{"level":"error","logSource":"pkg/controller/restore_controller.go:580","msg":"Namespace opsstack, resource restore error: error preparing persistentvolumeclaims/opsstack/storage-opsstack-loki-1: rpc error: code = Unknown desc = Failed to get Volumesnapshot opsstack/velero-storage-opsstack-loki-1-r5rhj to restore PVC opsstack/storage-opsstack-loki-1: volumesnapshots.snapshot.storage.k8s.io \"velero-storage-opsstack-loki-1-r5rhj\" not found","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"}

What did you expect to happen: PVC restored successfully

The following information will help us better understand what's going on:

If you are using velero v1.7.0+:
Please use velero debug --backup <backupname> --restore <restorename> to generate the support bundle, and attach to this issue, more options please refer to velero debug --help

If you are using earlier versions:
Please provide the output of the following commands (Pasting long output into a GitHub gist or other pastebin is fine.)

Name:         opsstack-velero-hourly-opsstack-20240219180007
Namespace:    opsstack
Labels:       app.kubernetes.io/instance=opsstack
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=velero
              helm.sh/chart=velero-5.3.0
              velero.io/schedule-name=opsstack-velero-hourly-opsstack
              velero.io/storage-location=opsstack
Annotations:  meta.helm.sh/release-name=opsstack
              meta.helm.sh/release-namespace=opsstack
              velero.io/resource-timeout=10m0s
              velero.io/source-cluster-k8s-gitversion=v1.27.7
              velero.io/source-cluster-k8s-major-version=1
              velero.io/source-cluster-k8s-minor-version=27

Phase:  Completed

Warnings:
  Velero:     <none>
  Cluster:    <none>
  Namespaces:
    opsstack:   resource: /pods name: /opsstack-grafana message: /Additional item was not found in Kubernetes API, can't back it up

Namespaces:
  Included:  opsstack
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  included

Label selector:  <none>

Or label selector:  <none>

Storage Location:  opsstack

Velero-Native Snapshot PVs:  true
Snapshot Move Data:          false
Data Mover:                  velero

TTL:  6h0m0s

CSISnapshotTimeout:    10m0s
ItemOperationTimeout:  4h0m0s

Hooks:  <none>

Backup Format Version:  1.1.0

Started:    2024-02-19 19:00:07 +0100 CET
Completed:  2024-02-19 19:00:45 +0100 CET

Expiration:  2024-02-20 01:00:07 +0100 CET

Total items to be backed up:  1275
Items backed up:              1275

Backup Item Operations:  8 of 8 completed successfully, 0 failed (specify --details for more information)
Backup Volumes:
  <error getting backup volume info: DownloadRequest.velero.io "opsstack-velero-hourly-opsstack-20240219180007-5e30d0cf-26af-4141-8d78-cd66fd4a3534" is invalid: spec.target.kind: Unsupported value: "BackupVolumeInfos": supported values: "BackupLog", "BackupContents", "BackupVolumeSnapshots", "BackupItemOperations", "BackupResourceList", "BackupResults", "RestoreLog", "RestoreResults", "RestoreResourceList", "RestoreItemOperations", "CSIBackupVolumeSnapshots", "CSIBackupVolumeSnapshotContents">

Details ``` Name: opsstack-velero-hourly-opsstack-20240219173046-20240219184517 Namespace: opsstack Labels: Annotations: Phase: PartiallyFailed (run 'velero restore logs opsstack-velero-hourly-opsstack-20240219173046-20240219184517' for more information) Total items to be restored: 4 Items restored: 4 Started: 2024-02-19 18:45:17 +0100 CET Completed: 2024-02-19 18:45:20 +0100 CET Errors: Velero: Cluster: Namespaces: opsstack: error preparing persistentvolumeclaims/opsstack/storage-opsstack-loki-0: rpc error: code = Unknown desc = Failed to get Volumesnapshot opsstack/velero-storage-opsstack-loki-0-r67lz to restore PVC opsstack/storage-opsstack-loki-0: volumesnapshots.snapshot.storage.k8s.io "velero-storage-opsstack-loki-0-r67lz" not found error preparing persistentvolumeclaims/opsstack/storage-opsstack-loki-1: rpc error: code = Unknown desc = Failed to get Volumesnapshot opsstack/velero-storage-opsstack-loki-1-r5rhj to restore PVC opsstack/storage-opsstack-loki-1: volumesnapshots.snapshot.storage.k8s.io "velero-storage-opsstack-loki-1-r5rhj" not found Backup: opsstack-velero-hourly-opsstack-20240219173046 Namespaces: Included: all namespaces found in the backup Excluded: Resources: Included: pvc, pv Excluded: nodes, events, events.events.k8s.io, backups.velero.io, restores.velero.io, resticrepositories.velero.io, csinodes.storage.k8s.io, volumeattachments.storage.k8s.io, backuprepositories.velero.io Cluster-scoped: auto Namespace mappings: Label selector: app.kubernetes.io/name=opsstack-loki Or label selector: Restore PVs: auto Existing Resource Policy: ItemOperationTimeout: 4h0m0s Preserve Service NodePorts: auto ```
Details ``` {"cmd":"/plugins/velero-plugin-for-microsoft-azure","level":"info","logSource":"/go/pkg/mod/github.com/vmware-tanzu/velero@v1.13.0/pkg/util/azure/storage.go:213","msg":"use the storage account URI retrieved from the storage account properties \"https://opsstackdataf76mpsm3s.blob.core.windows.net/\"","pluginName":"velero-plugin-for-microsoft-azure","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:18Z"} {"cmd":"/plugins/velero-plugin-for-microsoft-azure","level":"info","logSource":"/go/pkg/mod/github.com/vmware-tanzu/velero@v1.13.0/pkg/util/azure/storage.go:114","msg":"auth with Azure AD","pluginName":"velero-plugin-for-microsoft-azure","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:18Z"} {"level":"info","logSource":"pkg/controller/restore_controller.go:535","msg":"starting restore","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:18Z"} {"level":"info","logSource":"pkg/restore/restore.go:430","msg":"Starting restore of backup opsstack/opsstack-velero-hourly-opsstack-20240219173046","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:18Z"} {"level":"info","logSource":"pkg/restore/restore.go:2202","msg":"Skipping restore of resource because the restore spec excludes it","resource":"customresourcedefinitions.apiextensions.k8s.io","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2202","msg":"Skipping restore of resource because the restore spec excludes it","resource":"customresourcedefinitions.apiextensions.k8s.io","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2202","msg":"Skipping restore of resource because the restore spec excludes it","resource":"namespaces","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2202","msg":"Skipping restore of resource because the restore spec excludes it","resource":"storageclasses.storage.k8s.io","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2202","msg":"Skipping restore of resource because the restore spec excludes it","resource":"volumesnapshotclasses.snapshot.storage.k8s.io","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2202","msg":"Skipping restore of resource because the restore spec excludes it","resource":"volumesnapshotcontents.snapshot.storage.k8s.io","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2202","msg":"Skipping restore of resource because the restore spec excludes it","resource":"volumesnapshots.snapshot.storage.k8s.io","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2202","msg":"Skipping restore of resource because the restore spec excludes it","resource":"datauploads.velero.io","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2266","msg":"Resource 'persistentvolumes' will be restored at cluster scope","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2264","msg":"Resource 'persistentvolumeclaims' will be restored into namespace 'opsstack'","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2202","msg":"Skipping restore of resource because the restore spec excludes it","resource":"serviceaccounts","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2202","msg":"Skipping restore of resource because the restore spec excludes it","resource":"secrets","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2202","msg":"Skipping restore of resource because the restore spec excludes it","resource":"configmaps","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2202","msg":"Skipping restore of resource because the restore spec excludes it","resource":"limitranges","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2202","msg":"Skipping restore of resource because the restore spec excludes it","resource":"pods","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2202","msg":"Skipping restore of resource because the restore spec excludes it","resource":"replicasets.apps","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2185","msg":"Skipping restore of resource because it cannot be resolved via discovery","resource":"clusterclasses.cluster.x-k8s.io","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2202","msg":"Skipping restore of resource because the restore spec excludes it","resource":"endpoints","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2202","msg":"Skipping restore of resource because the restore spec excludes it","resource":"services","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2202","msg":"Skipping restore of resource because the restore spec excludes it","resource":"alertmanagers.monitoring.coreos.com","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2202","msg":"Skipping restore of resource because the restore spec excludes it","resource":"apiservices.apiregistration.k8s.io","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2202","msg":"Skipping restore of resource because the restore spec excludes it","resource":"backups.velero.io","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2202","msg":"Skipping restore of resource because the restore spec excludes it","resource":"backupstoragelocations.velero.io","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2202","msg":"Skipping restore of resource because the restore spec excludes it","resource":"certificaterequests.cert-manager.io","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2202","msg":"Skipping restore of resource because the restore spec excludes it","resource":"certificates.cert-manager.io","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2202","msg":"Skipping restore of resource because the restore spec excludes it","resource":"challenges.acme.cert-manager.io","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2202","msg":"Skipping restore of resource because the restore spec excludes it","resource":"clusterissuers.cert-manager.io","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2202","msg":"Skipping restore of resource because the restore spec excludes it","resource":"clusterrolebindings.rbac.authorization.k8s.io","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2202","msg":"Skipping restore of resource because the restore spec excludes it","resource":"clusterroles.rbac.authorization.k8s.io","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2202","msg":"Skipping restore of resource because the restore spec excludes it","resource":"controllerrevisions.apps","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2202","msg":"Skipping restore of resource because the restore spec excludes it","resource":"csidrivers.storage.k8s.io","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2202","msg":"Skipping restore of resource because the restore spec excludes it","resource":"csinodes.storage.k8s.io","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2202","msg":"Skipping restore of resource because the restore spec excludes it","resource":"customresourcedefinitions.apiextensions.k8s.io","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2202","msg":"Skipping restore of resource because the restore spec excludes it","resource":"daemonsets.apps","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2202","msg":"Skipping restore of resource because the restore spec excludes it","resource":"deployments.apps","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2202","msg":"Skipping restore of resource because the restore spec excludes it","resource":"downloadrequests.velero.io","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2202","msg":"Skipping restore of resource because the restore spec excludes it","resource":"endpointslices.discovery.k8s.io","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2202","msg":"Skipping restore of resource because the restore spec excludes it","resource":"events","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2202","msg":"Skipping restore of resource because the restore spec excludes it","resource":"flowschemas.flowcontrol.apiserver.k8s.io","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2202","msg":"Skipping restore of resource because the restore spec excludes it","resource":"ingressclasses.networking.k8s.io","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2202","msg":"Skipping restore of resource because the restore spec excludes it","resource":"ingresses.networking.k8s.io","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2202","msg":"Skipping restore of resource because the restore spec excludes it","resource":"issuers.cert-manager.io","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2202","msg":"Skipping restore of resource because the restore spec excludes it","resource":"leases.coordination.k8s.io","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2202","msg":"Skipping restore of resource because the restore spec excludes it","resource":"mutatingwebhookconfigurations.admissionregistration.k8s.io","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2202","msg":"Skipping restore of resource because the restore spec excludes it","resource":"nodes","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2202","msg":"Skipping restore of resource because the restore spec excludes it","resource":"orders.acme.cert-manager.io","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2202","msg":"Skipping restore of resource because the restore spec excludes it","resource":"poddisruptionbudgets.policy","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2202","msg":"Skipping restore of resource because the restore spec excludes it","resource":"priorityclasses.scheduling.k8s.io","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2202","msg":"Skipping restore of resource because the restore spec excludes it","resource":"prioritylevelconfigurations.flowcontrol.apiserver.k8s.io","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2202","msg":"Skipping restore of resource because the restore spec excludes it","resource":"probes.monitoring.coreos.com","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2202","msg":"Skipping restore of resource because the restore spec excludes it","resource":"prometheuses.monitoring.coreos.com","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2202","msg":"Skipping restore of resource because the restore spec excludes it","resource":"prometheusrules.monitoring.coreos.com","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2202","msg":"Skipping restore of resource because the restore spec excludes it","resource":"restores.velero.io","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2202","msg":"Skipping restore of resource because the restore spec excludes it","resource":"rolebindings.rbac.authorization.k8s.io","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2202","msg":"Skipping restore of resource because the restore spec excludes it","resource":"roles.rbac.authorization.k8s.io","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2202","msg":"Skipping restore of resource because the restore spec excludes it","resource":"schedules.velero.io","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2202","msg":"Skipping restore of resource because the restore spec excludes it","resource":"servicemonitors.monitoring.coreos.com","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2202","msg":"Skipping restore of resource because the restore spec excludes it","resource":"statefulsets.apps","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2202","msg":"Skipping restore of resource because the restore spec excludes it","resource":"storageclasses.storage.k8s.io","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2202","msg":"Skipping restore of resource because the restore spec excludes it","resource":"validatingwebhookconfigurations.admissionregistration.k8s.io","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2202","msg":"Skipping restore of resource because the restore spec excludes it","resource":"volumeattachments.storage.k8s.io","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2202","msg":"Skipping restore of resource because the restore spec excludes it","resource":"volumesnapshotclasses.snapshot.storage.k8s.io","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2202","msg":"Skipping restore of resource because the restore spec excludes it","resource":"volumesnapshotlocations.velero.io","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2185","msg":"Skipping restore of resource because it cannot be resolved via discovery","resource":"clusterbootstraps.run.tanzu.vmware.com","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2185","msg":"Skipping restore of resource because it cannot be resolved via discovery","resource":"clusters.cluster.x-k8s.io","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:2185","msg":"Skipping restore of resource because it cannot be resolved via discovery","resource":"clusterresourcesets.addons.cluster.x-k8s.io","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:604","msg":"waiting informer cache sync ...","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:1050","msg":"Getting client for /v1, Kind=PersistentVolumeClaim","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:1342","msg":"restore status includes excludes: \u003cnil\u003e","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:1356","msg":"Executing item action for persistentvolumeclaims","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"cmd":"/velero","level":"info","logSource":"pkg/restore/add_pv_from_pvc_action.go:44","msg":"Executing AddPVFromPVCAction","pluginName":"velero","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"cmd":"/velero","level":"info","logSource":"pkg/restore/add_pv_from_pvc_action.go:66","msg":"Adding PV pvc-0a35f9f4-539e-40e9-89a6-1c87230e92d2 as an additional item to restore","pluginName":"velero","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:1050","msg":"Getting client for /v1, Kind=PersistentVolume","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:1228","msg":"Find VolumeInfo for PV pvc-0a35f9f4-539e-40e9-89a6-1c87230e92d2.","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"groupResource":"persistentvolumes","level":"info","logSource":"pkg/restore/restore.go:1249","msg":"Dynamically re-provisioning persistent volume because it has a CSI VolumeSnapshot or a related snapshot DataUpload.","name":"pvc-0a35f9f4-539e-40e9-89a6-1c87230e92d2","namespace":"","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:1356","msg":"Executing item action for persistentvolumeclaims","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"cmd":"/velero","level":"info","logSource":"pkg/restore/change_pvc_node_selector.go:66","msg":"Executing ChangePVCNodeSelectorAction","pluginName":"velero","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"cmd":"/velero","kind":"PersistentVolumeClaim","level":"info","logSource":"pkg/restore/change_pvc_node_selector.go:129","msg":"Clearing selected-node because node named aks-opsstack-25878551-vmss000000 does not exist","name":"storage-opsstack-loki-0","namespace":"opsstack","pluginName":"velero","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"cmd":"/velero","level":"info","logSource":"pkg/restore/change_pvc_node_selector.go:138","msg":"Done executing ChangePVCNodeSelectorAction","pluginName":"velero","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:1356","msg":"Executing item action for persistentvolumeclaims","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"cmd":"/velero","level":"info","logSource":"pkg/restore/change_storageclass_action.go:68","msg":"Executing ChangeStorageClassAction","pluginName":"velero","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"cmd":"/velero","level":"info","logSource":"pkg/restore/change_storageclass_action.go:79","msg":"Done executing ChangeStorageClassAction","pluginName":"velero","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:1356","msg":"Executing item action for persistentvolumeclaims","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"Action":"PVCRestoreItemAction","PVC":"opsstack/storage-opsstack-loki-0","Restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","cmd":"/plugins/velero-plugin-for-csi","level":"info","logSource":"/go/src/velero-plugin-for-csi/internal/restore/pvc_action.go:132","msg":"Starting PVCRestoreItemAction for PVC","pluginName":"velero-plugin-for-csi","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"Action":"PVCRestoreItemAction","PVC":"opsstack/storage-opsstack-loki-0","Restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","cmd":"/plugins/velero-plugin-for-csi","level":"error","logSource":"/go/src/velero-plugin-for-csi/internal/restore/pvc_action.go:204","msg":"Failed to restore PVC from VolumeSnapshot.","pluginName":"velero-plugin-for-csi","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:807","msg":"Restored 2 items out of an estimated total of 3 (estimate will change throughout the restore)","name":"storage-opsstack-loki-0","namespace":"opsstack","progress":"","resource":"persistentvolumeclaims","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:1342","msg":"restore status includes excludes: \u003cnil\u003e","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:1356","msg":"Executing item action for persistentvolumeclaims","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"cmd":"/velero","level":"info","logSource":"pkg/restore/add_pv_from_pvc_action.go:44","msg":"Executing AddPVFromPVCAction","pluginName":"velero","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"cmd":"/velero","level":"info","logSource":"pkg/restore/add_pv_from_pvc_action.go:66","msg":"Adding PV pvc-52590e4a-e995-4617-8eb8-e2f627a2a1fa as an additional item to restore","pluginName":"velero","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:1228","msg":"Find VolumeInfo for PV pvc-52590e4a-e995-4617-8eb8-e2f627a2a1fa.","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"groupResource":"persistentvolumes","level":"info","logSource":"pkg/restore/restore.go:1249","msg":"Dynamically re-provisioning persistent volume because it has a CSI VolumeSnapshot or a related snapshot DataUpload.","name":"pvc-52590e4a-e995-4617-8eb8-e2f627a2a1fa","namespace":"","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:1356","msg":"Executing item action for persistentvolumeclaims","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"cmd":"/velero","level":"info","logSource":"pkg/restore/change_pvc_node_selector.go:66","msg":"Executing ChangePVCNodeSelectorAction","pluginName":"velero","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"cmd":"/velero","kind":"PersistentVolumeClaim","level":"info","logSource":"pkg/restore/change_pvc_node_selector.go:129","msg":"Clearing selected-node because node named aks-opsstack-25878551-vmss000001 does not exist","name":"storage-opsstack-loki-1","namespace":"opsstack","pluginName":"velero","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"cmd":"/velero","level":"info","logSource":"pkg/restore/change_pvc_node_selector.go:138","msg":"Done executing ChangePVCNodeSelectorAction","pluginName":"velero","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:1356","msg":"Executing item action for persistentvolumeclaims","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"cmd":"/velero","level":"info","logSource":"pkg/restore/change_storageclass_action.go:68","msg":"Executing ChangeStorageClassAction","pluginName":"velero","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"cmd":"/velero","level":"info","logSource":"pkg/restore/change_storageclass_action.go:79","msg":"Done executing ChangeStorageClassAction","pluginName":"velero","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:1356","msg":"Executing item action for persistentvolumeclaims","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"Action":"PVCRestoreItemAction","PVC":"opsstack/storage-opsstack-loki-1","Restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","cmd":"/plugins/velero-plugin-for-csi","level":"info","logSource":"/go/src/velero-plugin-for-csi/internal/restore/pvc_action.go:132","msg":"Starting PVCRestoreItemAction for PVC","pluginName":"velero-plugin-for-csi","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"Action":"PVCRestoreItemAction","PVC":"opsstack/storage-opsstack-loki-1","Restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","cmd":"/plugins/velero-plugin-for-csi","level":"error","logSource":"/go/src/velero-plugin-for-csi/internal/restore/pvc_action.go:204","msg":"Failed to restore PVC from VolumeSnapshot.","pluginName":"velero-plugin-for-csi","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:807","msg":"Restored 4 items out of an estimated total of 4 (estimate will change throughout the restore)","name":"storage-opsstack-loki-1","namespace":"opsstack","progress":"","resource":"persistentvolumeclaims","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:660","msg":"Waiting for all pod volume restores to complete","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:676","msg":"Done waiting for all pod volume restores to complete","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:680","msg":"Waiting for all post-restore-exec hooks to complete","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:688","msg":"Done waiting for all post-restore exec hooks to complete","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/restore/restore.go:695","msg":"hookTracker: map[], hookAttempted: 0, hookFailed: 0","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"error","logSource":"pkg/controller/restore_controller.go:580","msg":"Namespace opsstack, resource restore error: error preparing persistentvolumeclaims/opsstack/storage-opsstack-loki-0: rpc error: code = Unknown desc = Failed to get Volumesnapshot opsstack/velero-storage-opsstack-loki-0-r67lz to restore PVC opsstack/storage-opsstack-loki-0: volumesnapshots.snapshot.storage.k8s.io \"velero-storage-opsstack-loki-0-r67lz\" not found","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"error","logSource":"pkg/controller/restore_controller.go:580","msg":"Namespace opsstack, resource restore error: error preparing persistentvolumeclaims/opsstack/storage-opsstack-loki-1: rpc error: code = Unknown desc = Failed to get Volumesnapshot opsstack/velero-storage-opsstack-loki-1-r5rhj to restore PVC opsstack/storage-opsstack-loki-1: volumesnapshots.snapshot.storage.k8s.io \"velero-storage-opsstack-loki-1-r5rhj\" not found","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} {"level":"info","logSource":"pkg/controller/restore_controller.go:594","msg":"restore completed","restore":"opsstack/opsstack-velero-hourly-opsstack-20240219173046-20240219184517","time":"2024-02-19T17:45:19Z"} ```

Anything else you would like to add:

Environment:

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

jkroepke commented 7 months ago

FYI: I tried to use velero debug --backup opsstack-velero-hourly-opsstack-20240219173046 --restore opsstack-velero-hourly-opsstack-20240219173046-20240219184517 first and looked into the bundle. However it included all pod logs of the namespaces, including non-velero related which is too much to share.

blackpiglet commented 7 months ago

@jkroepke The velero debug command should not collect the pod information other than the Velero-related ones. Could you check the Velero CLI version you are using?

First, from the backup description information, I think Velero's CRDs are not updated to date.

<error getting backup volume info: DownloadRequest.velero.io "opsstack-velero-hourly-opsstack-20240219180007-5e30d0cf-26af-4141-8d78-cd66fd4a3534" is invalid: spec.target.kind: Unsupported value: "BackupVolumeInfos": supported values: "BackupLog", "BackupContents", "BackupVolumeSnapshots", "BackupItemOperations", "BackupResourceList", "BackupResults", "RestoreLog", "RestoreResults", "RestoreResourceList", "RestoreItemOperations", "CSIBackupVolumeSnapshots", "CSIBackupVolumeSnapshotContents">

Please follow the Velero update procedure document: https://velero.io/docs/v1.13/upgrade-to-1.13/.

Second, please do not limit the resources in the restore process. Just PVC and PV are not enough for the Velero to work properly, because, in your scenario, the VolumeSnapshot and VolumeSnapshotContent are generated as the intermediate resources. They are also needed to make the CSI volume restore to work.

Resources:
  Included:        pvc, pv
  Excluded:        nodes, events, events.events.k8s.io, backups.velero.io, restores.velero.io, resticrepositories.velero.io, csinodes.storage.k8s.io, volumeattachments.storage.k8s.io, backuprepositories.velero.io
  Cluster-scoped:  auto
jkroepke commented 7 months ago

I think Velero's CRDs are not updated to date

Thanks, I fixed that.

Looking at the crashd, I guess crashd will collect logs from all pods in the namespace. To due some restriction, I have to bundle velero with other pods on the same namespace.

https://github.com/vmware-tanzu/velero/blob/d45b313f0737030c62f8fd98d1f0fc01a1319094/pkg/cmd/cli/debug/cshd-scripts/velero.cshd#L26C1-L26C13

Second, please do not limit the resources in the restore process.

If there an recommend list, if I want only to restore a PVC, e.g:

--include-resources pvc,pv,VolumeSnapshotContent,VolumeSnapshot

I also found other issues like:

Which I expect that velero should automatically added the VolumeSnapshot and VolumeSnapshotContent, or not? Or do I have to set backup.velero.io/must-include-additional-items?

blackpiglet commented 7 months ago

Looking at the crashd, I guess crashd will collect logs from all pods in the namespace. To due some restriction, I have to bundle velero with other pods on the same namespace.

Got it. Thanks for the information.

Which I expect that velero should automatically added the VolumeSnapshot and VolumeSnapshotContent, or not? Or do I have to set backup.velero.io/must-include-additional-items?

No need to add the annotation manually.

blackpiglet commented 7 months ago

@jkroepke Does this explanation sound reasonable to you?

jkroepke commented 7 months ago

It's fine for me. It understand it now. Thanks.