Velero backup PartiallyFailed due to `failed to get volumesnapshot`

[nihaldivyam]

What steps did you take and what happened:

• I attempted to perform a Velero backup.
• The backup process failed with the following error related to volumesnapshots not being found.

Name:         daily-backup-20231221144308
Namespace:    velero
Labels:       argocd.argoproj.io/instance=velero
              velero.io/schedule-name=daily-backup
              velero.io/storage-location=default
Annotations:  kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"velero.io/v1","kind":"Schedule","metadata":{"annotations":{},"labels":{"argocd.argoproj.io/instance":"velero"},"name":"daily-backup","namespace":"velero"},"spec":{"schedule":"0 5 * * 1-6","template":{"csiSnapshotTimeout":"40m","hooks":{},"includedNamespaces":["*"],"itemOperationTimeout":"300m","metadata":{"labels":{"argocd.argoproj.io/instance":"velero"}},"ttl":"720h0m0s"}}}

  velero.io/resource-timeout=10m0s
  velero.io/source-cluster-k8s-gitversion=v1.26.6
  velero.io/source-cluster-k8s-major-version=1
  velero.io/source-cluster-k8s-minor-version=26

Phase:  PartiallyFailed (run `velero backup logs daily-backup-20231221144308` for more information)

Errors:
  Velero:    name: /web-customer-group-benns-dk-57fc845978-lt988 error: /error executing custom action (groupResource=volumesnapshots.snapshot.storage.k8s.io, namespace=web-customer-group, name=velero-web-customer-group-frontend-data-pv-claim-9c46c): rpc error: code = Unknown desc = failed to get volumesnapshot web-customer-group/velero-web-customer-group-frontend-data-pv-claim-9c46c: volumesnapshots.snapshot.storage.k8s.io "velero-web-customer-group-frontend-data-pv-claim-9c46c" not found
             name: /web-customer-group-cms-0 error: /error executing custom action (groupResource=volumesnapshots.snapshot.storage.k8s.io, namespace=web-customer-group, name=velero-web-customer-group-data-storage-pv-claim-f6c4p): rpc error: code = Unknown desc = failed to get volumesnapshot web-customer-group/velero-web-customer-group-data-storage-pv-claim-f6c4p: volumesnapshots.snapshot.storage.k8s.io "velero-web-customer-group-data-storage-pv-claim-f6c4p" not found
             name: /web-customer-group-media-659d7c8499-cdm6q error: /error executing custom action (groupResource=volumesnapshots.snapshot.storage.k8s.io, namespace=web-customer-group, name=velero-web-customer-group-tours-media-pv-claim-7x97t): rpc error: code = Unknown desc = failed to get volumesnapshot web-customer-group/velero-web-customer-group-tours-media-pv-claim-7x97t: volumesnapshots.snapshot.storage.k8s.io "velero-web-customer-group-tours-media-pv-claim-7x97t" not found
             name: /web-customer-group-mongo1-0 error: /error executing custom action (groupResource=volumesnapshots.snapshot.storage.k8s.io, namespace=web-customer-group, name=velero-web-customer-group-mongo1-pv-claim-ctz8h): rpc error: code = Unknown desc = failed to get volumesnapshot web-customer-group/velero-web-customer-group-mongo1-pv-claim-ctz8h: volumesnapshots.snapshot.storage.k8s.io "velero-web-customer-group-mongo1-pv-claim-ctz8h" not found
             name: /web-customer-group-mongo2-0 error: /error executing custom action (groupResource=volumesnapshots.snapshot.storage.k8s.io, namespace=web-customer-group, name=velero-web-customer-group-mongo2-pv-claim-dk5gm): rpc error: code = Unknown desc = failed to get volumesnapshot web-customer-group/velero-web-customer-group-mongo2-pv-claim-dk5gm: volumesnapshots.snapshot.storage.k8s.io "velero-web-customer-group-mongo2-pv-claim-dk5gm" not found
             name: /web-customer-group-pdfs-transfer-cronjob-28386000-s5jdj error: /error executing custom action (groupResource=volumesnapshots.snapshot.storage.k8s.io, namespace=web-customer-group, name=velero-web-customer-group-lasernet-pv-claim-p9hfh): rpc error: code = Unknown desc = failed to get volumesnapshot web-customer-group/velero-web-customer-group-lasernet-pv-claim-p9hfh: volumesnapshots.snapshot.storage.k8s.io "velero-web-customer-group-lasernet-pv-claim-p9hfh" not found
             name: /web-customer-group-mongo-key-pv-claim error: /error executing custom action (groupResource=volumesnapshots.snapshot.storage.k8s.io, namespace=web-customer-group, name=velero-web-customer-group-mongo-key-pv-claim-s9zz8): rpc error: code = Unknown desc = failed to get volumesnapshot web-customer-group/velero-web-customer-group-mongo-key-pv-claim-s9zz8: volumesnapshots.snapshot.storage.k8s.io "velero-web-customer-group-mongo-key-pv-claim-s9zz8" not found
  Cluster:    <none>
  Namespaces: <none>

Namespaces:
  Included:  *
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  auto

Label selector:  <none>

Or label selector:  <none>

Storage Location:  default

Velero-Native Snapshot PVs:  auto
Snapshot Move Data:          false
Data Mover:                  velero

TTL:  720h0m0s

CSISnapshotTimeout:    40m0s
ItemOperationTimeout:  5h0m0s

Hooks:  <none>

Backup Format Version:  1.1.0

Started:    2023-12-21 20:13:08 +0530 IST
Completed:  2023-12-22 00:55:24 +0530 IST

Expiration:  2024-01-20 20:13:08 +0530 IST

Total items to be backed up:  19513
Items backed up:              19513

Backup Item Operations:  70 of 70 completed successfully, 0 failed (specify --details for more information)
Velero-Native Snapshots: <none included>

What did you expect to happen:

• The Velero backup process should complete successfully without errors.

The output of the following commands will help us better understand what's going on:

• https://gist.github.com/nihaldivyam/b3a2f44e592a2ba87457fb8ecf06852a

Anything else you would like to add:

• I have tried increasing the casi timeout but that didnt help
• The issue sometimes goes away and the backup get completed successfully.

$ velero get backup
NAME                          STATUS            ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
daily-backup-20231222050004   PartiallyFailed   7        0          2023-12-22 10:30:04 +0530 IST   29d       default            <none>
daily-backup-20231222000008   PartiallyFailed   13       0          2023-12-22 05:30:08 +0530 IST   29d       default            <none>
daily-backup-20231221144308   PartiallyFailed   7        0          2023-12-21 20:13:08 +0530 IST   29d       default            <none>
daily-backup-20231221110328   PartiallyFailed   1        159        2023-12-21 21:15:20 +0530 IST   29d       default            <none>
daily-backup-20231221110059   PartiallyFailed   1        166        2023-12-21 16:30:59 +0530 IST   28d       default            <none>
daily-backup-20231221050003   PartiallyFailed   7        99         2023-12-21 10:30:03 +0530 IST   28d       default            <none>
daily-backup-20231221000006   PartiallyFailed   13       0          2023-12-21 05:30:06 +0530 IST   28d       default            <none>
daily-backup-20231220110058   PartiallyFailed   2        322        2023-12-20 16:30:58 +0530 IST   27d       default            <none>
daily-backup-20231220050001   PartiallyFailed   7        98         2023-12-20 10:30:01 +0530 IST   27d       default            <none>
daily-backup-20231219110057   PartiallyFailed   1        716        2023-12-19 16:30:57 +0530 IST   26d       default            <none>
daily-backup-20231219050052   PartiallyFailed   7        92         2023-12-19 10:30:52 +0530 IST   26d       default            <none>
daily-backup-20231219000003   PartiallyFailed   13       0          2023-12-19 05:30:03 +0530 IST   26d       default            <none>
daily-backup-20231218110056   PartiallyFailed   5        559        2023-12-18 16:30:56 +0530 IST   25d       default            <none>
daily-backup-20231218050051   PartiallyFailed   7        546        2023-12-18 10:30:51 +0530 IST   25d       default            <none>
daily-backup-20231218000001   PartiallyFailed   13       0          2023-12-18 05:30:02 +0530 IST   25d       default            <none>
daily-backup-20231216110053   Completed         0        223        2023-12-16 16:30:53 +0530 IST   23d       default            <none>
daily-backup-20231216050048   PartiallyFailed   8        0          2023-12-16 10:30:48 +0530 IST   23d       default            <none>
daily-backup-20231216000058   PartiallyFailed   1        0          2023-12-16 05:30:58 +0530 IST   23d       default            <none>
daily-backup-20231215110052   Completed         0        504        2023-12-15 16:30:52 +0530 IST   22d       default            <none>

Environment:

• helm chart version - 5.2.0
• Kubernetes version - v1.26.6
• Cloud provider - Azure AKS

[Ilhaame]

I am experiencing a similar issue. Have you found a solution for it ?

[pschichtel]

I was able to just delete the PVCs, they were just left over from a migration. So I don't really have a "solution".

[jenting]

This repo is for helm chart issue only, it'd be good to open the issue under velero repo, thanks.