Fail get DataUploadResult for restore - multiple DataUpload result cms found

[zanderson-aim]

What steps did you take and what happened: When trying to restore a backup I get the following error:

    rc-aimit-io:  error preparing persistentvolumeclaims/rc-aimit-io/datadir-rocketchat-mongodb-0: rpc error: code = Unknown desc = fail get DataUploadResult for restore: velero-imdc-202311
05205455-20231105210928: multiple DataUpload result cms found with labels velero.io/pvc-namespace-name=rc-aimit-io.datadir-rocketchat-mongodb-0,velero.io/restore-uid=630b736f-e04a-4fb7-b937
-7654feb4062e,velero.io/resource-usage=DataUpload
                  error preparing persistentvolumeclaims/rc-aimit-io/datadir-rocketchat-mongodb-1: rpc error: code = Unknown desc = fail get DataUploadResult for restore: velero-imdc-202311
05205455-20231105210928: multiple DataUpload result cms found with labels velero.io/pvc-namespace-name=rc-aimit-io.datadir-rocketchat-mongodb-1,velero.io/restore-uid=630b736f-e04a-4fb7-b937
-7654feb4062e,velero.io/resource-usage=DataUpload

The restore command velero restore create --from-backup velero-imdc-20231105205455 --include-namespaces rc-aimit-io --include-resources persistentvolumeclaims,persistentvolumes,dataupload

What did you expect to happen: The PV/PVCs would be restored.

The following information will help us better understand what's going on: bundle-2023-11-05-23-09-42.tar.gz

Anything else you would like to add: I was able to confirm the data is there by manually mounting the bucket using Kopia tools and restoring the files from there.

Environment:

• Velero version (use velero version):

  Client:
      Version: v1.12.1
      Git commit: 5c4fdfe147357ec7b908339f4516cd96d6b97c61
  Server:
      Version: v1.12.1

• Velero features (use velero client config get features): features: <NOT SET>
• Kubernetes version (use kubectl version):

  Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.17+rke2r1", GitCommit:"a7736eaf34d823d7652415337ac0ad06db9167fc", GitTreeState:"clean", BuildDate:"2022-12-08T21:42:35Z", GoVersion:"go1.16.15b7", Compiler:"gc", Platform:"linux/amd64"}
  Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.17+rke2r1", GitCommit:"a7736eaf34d823d7652415337ac0ad06db9167fc", GitTreeState:"clean", BuildDate:"2022-12-08T21:42:35Z", GoVersion:"go1.16.15b7", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes installer & version:
• RKE2
• v1.22.17+rke2r1

[Lyndon-Li]

@zanderson-aim For a quick check, could you run below command and share the result? kubectl get cm -n velero

[blackpiglet]

{"level":"error","logSource":"pkg/controller/restore_controller.go:560","msg":"Namespace rc-aimit-io, resource restore error: error preparing persistentvolumeclaims/rc-aimit-io/datadir-rocketchat-mongodb-0: rpc error: code = Unknown desc = fail get DataUploadResult for restore: velero-imdc-20231105205455-20231105210928: multiple DataUpload result cms found with labels velero.io/pvc-namespace-name=rc-aimit-io.datadir-rocketchat-mongodb-0,velero.io/restore-uid=630b736f-e04a-4fb7-b937-7654feb4062e,velero.io/resource-usage=DataUpload","restore":"velero/velero-imdc-20231105205455-20231105210928","time":"2023-11-05T21:20:28Z"}
{"level":"error","logSource":"pkg/controller/restore_controller.go:560","msg":"Namespace rc-aimit-io, resource restore error: error preparing persistentvolumeclaims/rc-aimit-io/datadir-rocketchat-mongodb-1: rpc error: code = Unknown desc = fail get DataUploadResult for restore: velero-imdc-20231105205455-20231105210928: multiple DataUpload result cms found with labels velero.io/pvc-namespace-name=rc-aimit-io.datadir-rocketchat-mongodb-1,velero.io/restore-uid=630b736f-e04a-4fb7-b937-7654feb4062e,velero.io/resource-usage=DataUpload","restore":"velero/velero-imdc-20231105205455-20231105210928","time":"2023-11-05T21:20:28Z"}

Please also check the PVCs rc-aimit-io/datadir-rocketchat-mongodb-0 and rc-aimit-io/datadir-rocketchat-mongodb-1 are mounted multiple times.

[zanderson-aim]

@zanderson-aim For a quick check, could you run below command and share the result? kubectl get cm -n velero

This results in about 9000 lines which look like this

velero-imdc-20231031000006-55gfn-j4jk6   1      5d5h
velero-imdc-20231031000006-55gfn-ldm8d   1      5d5h
velero-imdc-20231031000006-55gfn-qf8hn   1      4d4h
velero-imdc-20231031000006-55gfn-qzkb4   1      4d6h
velero-imdc-20231031000006-7n9b9-c5ph9   1      4d4h
velero-imdc-20231031000006-7n9b9-d8w6b   1      5d5h
velero-imdc-20231031000006-7n9b9-j4gpj   1      4d6h
velero-imdc-20231031000006-7n9b9-ns4xs   1      5d5h
velero-imdc-20231031000006-7wg25-bmgxg   1      4d6h
velero-imdc-20231031000006-7wg25-rml9s   1      4d4h
velero-imdc-20231031000006-7wg25-sd85w   1      5d5h
velero-imdc-20231031000006-7wg25-zm8zq   1      5d5h
velero-imdc-20231031000006-92sqt-5dfmz   1      4d4h
velero-imdc-20231031000006-92sqt-7lhk9   1      4d6h
velero-imdc-20231031000006-92sqt-bdr7h   1      5d5h
velero-imdc-20231031000006-92sqt-kmwf9   1      5d5h
velero-imdc-20231031000006-cdqft-5smd7   1      4d4h
velero-imdc-20231031000006-cdqft-hc99l   1      5d5h
velero-imdc-20231031000006-cdqft-m2lbp   1      5d5h
velero-imdc-20231031000006-cdqft-vwmc7   1      4d6h
velero-imdc-20231031000006-cvmt7-8qx8p   1      5d5h
velero-imdc-20231031000006-cvmt7-gpr97   1      4d6h
velero-imdc-20231031000006-cvmt7-ql5zw   1      4d4h
velero-imdc-20231031000006-cvmt7-rlswx   1      5d5h
velero-imdc-20231031000006-d8zbx-5vfnz   1      4d6h
velero-imdc-20231031000006-d8zbx-cl68t   1      4d4h
velero-imdc-20231031000006-d8zbx-jwgwp   1      5d5h
velero-imdc-20231031000006-d8zbx-t9thk   1      5d5h
velero-imdc-20231031000006-dkzqm-4brdn   1      5d5h
velero-imdc-20231031000006-dkzqm-dpls4   1      4d6h
velero-imdc-20231031000006-dkzqm-vsjfd   1      4d4h
velero-imdc-20231031000006-dkzqm-wf2fr   1      5d5h
velero-imdc-20231031000006-dqqrj-pd4bx   1      5d5h
velero-imdc-20231031000006-dqqrj-pf58j   1      4d4h
velero-imdc-20231031000006-dqqrj-spw8q   1      4d6h

[zanderson-aim]

{"level":"error","logSource":"pkg/controller/restore_controller.go:560","msg":"Namespace rc-aimit-io, resource restore error: error preparing persistentvolumeclaims/rc-aimit-io/datadir-rocketchat-mongodb-0: rpc error: code = Unknown desc = fail get DataUploadResult for restore: velero-imdc-20231105205455-20231105210928: multiple DataUpload result cms found with labels velero.io/pvc-namespace-name=rc-aimit-io.datadir-rocketchat-mongodb-0,velero.io/restore-uid=630b736f-e04a-4fb7-b937-7654feb4062e,velero.io/resource-usage=DataUpload","restore":"velero/velero-imdc-20231105205455-20231105210928","time":"2023-11-05T21:20:28Z"}
{"level":"error","logSource":"pkg/controller/restore_controller.go:560","msg":"Namespace rc-aimit-io, resource restore error: error preparing persistentvolumeclaims/rc-aimit-io/datadir-rocketchat-mongodb-1: rpc error: code = Unknown desc = fail get DataUploadResult for restore: velero-imdc-20231105205455-20231105210928: multiple DataUpload result cms found with labels velero.io/pvc-namespace-name=rc-aimit-io.datadir-rocketchat-mongodb-1,velero.io/restore-uid=630b736f-e04a-4fb7-b937-7654feb4062e,velero.io/resource-usage=DataUpload","restore":"velero/velero-imdc-20231105205455-20231105210928","time":"2023-11-05T21:20:28Z"}

Please also check the PVCs rc-aimit-io/datadir-rocketchat-mongodb-0 and rc-aimit-io/datadir-rocketchat-mongodb-1 are mounted multiple times.

I deleted them from the live system to test the restore, so they don't currently exist any more.

[Lyndon-Li]

@zanderson-aim Are there running data mover restores at the time you run kubectl get cm -n velero?

[zanderson-aim]

@zanderson-aim Are there running data mover restores at the time you run kubectl get cm -n velero?

No currently nothing running

kubectl -n velero get pods
NAME                      READY   STATUS    RESTARTS      AGE
node-agent-24ct4          1/1     Running   0             11d
node-agent-6ckx4          1/1     Running   0             11d
node-agent-8f559          1/1     Running   0             11d
node-agent-b2kpg          1/1     Running   0             11d
node-agent-pzbnq          1/1     Running   0             11d
node-agent-qzlsh          1/1     Running   0             11d
velero-6798d896db-5hpnk   1/1     Running   1 (40h ago)   11d

# kubectl get cm -n velero | wc -l
9756

[Lyndon-Li]

@zanderson-aim Could you share the content of one of these configMaps? E.g., kubectl get cm -n velero -o yaml velero-imdc-20231031000006-dqqrj-spw8q Please also let us know the timestamp of the latest one of these configMaps? You can also run kubectl get cm -n velero -o yaml for the latest configMap

[Lyndon-Li]

@zanderson-aim We want to do several checks on the list of configMaps as you shared, different info are required. So how about this:

1. Run kubectl get cm -n velero and share us the entire output
2. Run kubectl get cm -n velero -o yaml and share us the entire output

2 may be very large, not sure if you can make it, but let's see it.

[zanderson-aim]

@zanderson-aim We want to do several checks on the list of configMaps as you shared, different info are required. So how about this:

1. Run kubectl get cm -n velero and share us the entire output
2. Run kubectl get cm -n velero -o yaml and share us the entire output

2 may be very large, not sure if you can make it, but let's see it.

kubectl get cm -n velero > /tmp/velero-cm-list.txt velero-cm-list.txt

kubectl get cm -n velero -o yaml > /tmp/velero-cm.yaml velero-cm.zip

[Lyndon-Li]

@zanderson-aim Looks like you have included velero namespace in the backup as what we can see in the backup describe:

velero.io/v1/Backup:
    - velero/velero-imdc-20231007180046
    - velero/velero-imdc-20231008180047
    - velero/velero-imdc-20231013060021
    - velero/velero-imdc-20231013180021
    - velero/velero-imdc-20231014120022
    - velero/velero-imdc-20231014180023
    - velero/velero-imdc-20231015000059
    - velero/velero-imdc-20231015060059
    - velero/velero-imdc-20231015120059
    - velero/velero-imdc-20231015180000
    - velero/velero-imdc-20231016000000
    - velero/velero-imdc-20231016060000
    - velero/velero-imdc-20231016120001
    - velero/velero-imdc-20231016180001
    - velero/velero-imdc-20231017000001
    - velero/velero-imdc-20231017120047
    - velero/velero-imdc-20231017180047
    - velero/velero-imdc-20231018000047
    - velero/velero-imdc-20231018060048
    - velero/velero-imdc-20231019000049
    - velero/velero-imdc-20231019060049
    - velero/velero-imdc-20231019120049
    - velero/velero-imdc-20231020000032
    - velero/velero-imdc-20231020060038
    - velero/velero-imdc-20231020120038
    - velero/velero-imdc-20231020180039
    - velero/velero-imdc-20231021000039
    - velero/velero-imdc-20231021060013
    - velero/velero-imdc-20231021120013
    - velero/velero-imdc-20231021180011
    - velero/velero-imdc-20231022000012
    - velero/velero-imdc-20231022060012
    - velero/velero-imdc-20231022120053
    - velero/velero-imdc-20231022180054
    - velero/velero-imdc-20231023000054
    - velero/velero-imdc-20231023060054
    - velero/velero-imdc-20231023120057
    - velero/velero-imdc-20231023180057
    - velero/velero-imdc-20231024000057
    - velero/velero-imdc-20231024060058
    - velero/velero-imdc-20231024120058
    - velero/velero-imdc-20231024180058
    - velero/velero-imdc-20231025000059
    - velero/velero-imdc-20231025060059
    - velero/velero-imdc-20231025120049
    - velero/velero-imdc-20231025180049
    - velero/velero-imdc-20231026000059
    - velero/velero-imdc-20231026060059
    - velero/velero-imdc-20231026120000
    - velero/velero-imdc-20231026180000
    - velero/velero-imdc-20231027000001
    - velero/velero-imdc-20231027060001
    - velero/velero-imdc-20231027120001
    - velero/velero-imdc-20231027180001
    - velero/velero-imdc-20231028000002
    - velero/velero-imdc-20231028060002
    - velero/velero-imdc-20231028120002
    - velero/velero-imdc-20231028180003
    - velero/velero-imdc-20231029000003
    - velero/velero-imdc-20231029060003
    - velero/velero-imdc-20231029120004
    - velero/velero-imdc-20231029180004
    - velero/velero-imdc-20231030000004
    - velero/velero-imdc-20231030060005
    - velero/velero-imdc-20231030120005
    - velero/velero-imdc-20231030180005
    - velero/velero-imdc-20231030183306
    - velero/velero-imdc-20231031000006
    - velero/velero-imdc-20231031060006
    - velero/velero-imdc-20231031120006
    - velero/velero-imdc-20231031180007
    - velero/velero-imdc-20231101000007
    - velero/velero-imdc-20231101060008
    - velero/velero-imdc-20231101120008
    - velero/velero-imdc-20231101180008
    - velero/velero-imdc-20231101201654
    - velero/velero-imdc-20231102000009
    - velero/velero-imdc-20231102060009
    - velero/velero-imdc-20231102120009
    - velero/velero-imdc-20231102180010
    - velero/velero-imdc-20231103000010
    - velero/velero-imdc-20231103060011
    - velero/velero-imdc-20231103120011
    - velero/velero-imdc-20231103180011
    - velero/velero-imdc-20231104000012
    - velero/velero-imdc-20231104060012
    - velero/velero-imdc-20231104120045
    - velero/velero-imdc-20231104180045
    - velero/velero-imdc-20231105000046
    - velero/velero-imdc-20231105060046
    - velero/velero-imdc-20231105120046
    - velero/velero-imdc-20231105180046
    - velero/velero-imdc-20231105205455
  velero.io/v1/BackupRepository:
    - velero/4luvofthegame-com-backblaze-imdc-velero-kopia-lgzfh
    - velero/4x5program-org-backblaze-imdc-velero-kopia-xgzg5
    - velero/alumcoprecision-com-backblaze-imdc-velero-kopia-8s4wv
    - velero/argocd-backblaze-imdc-velero-kopia-lmbz8
    - velero/blingtour-com-backblaze-imdc-velero-kopia-n9s8j
    - velero/budibase-backblaze-imdc-velero-kopia-wnccx
    - velero/camplulu-com-backblaze-imdc-velero-kopia-zt6b5
    - velero/ceph-csi-backblaze-imdc-velero-kopia-675cp
    - velero/drillrdd-com-backblaze-imdc-velero-kopia-5j7kz
    - velero/fleet-aimit-io-backblaze-imdc-velero-kopia-kht45
    - velero/grafana-agent-backblaze-imdc-velero-kopia-kvmx2
    - velero/hs-aimit-io-backblaze-imdc-velero-kopia-576hs
    - velero/ingress-contour-backblaze-imdc-velero-kopia-v85kn
    - velero/inventory-aimit-io-backblaze-imdc-velero-kopia-2tl79
    - velero/kube-system-backblaze-imdc-velero-kopia-xc97l
    - velero/pm-aimit-io-backblaze-imdc-velero-kopia-dk9q8
    - velero/raisingthebar-org-backblaze-imdc-velero-kopia-4kfpb
    - velero/rc-aimit-io-backblaze-imdc-velero-kopia-gpvx9
    - velero/test-velero-backblaze-imdc-velero-kopia-8p58x
    - velero/velero-backblaze-imdc-velero-kopia-gsnhb
    - velero/velero-test-backblaze-imdc-velero-kopia-d5z69
  velero.io/v1/BackupStorageLocation:
    - velero/backblaze-imdc-velero
  velero.io/v1/Restore:
    - velero/velero-imdc-20231030183306-20231030204827
    - velero/velero-imdc-20231030183306-20231031223223
    - velero/velero-imdc-20231030183306-20231031225141
    - velero/velero-imdc-20231030183306-20231031225242
    - velero/velero-imdc-20231030183306-20231031231546
    - velero/velero-imdc-20231030183306-20231031231633
    - velero/velero-imdc-20231030183306-20231031231704
    - velero/velero-imdc-20231030183306-20231101025845
    - velero/velero-imdc-20231030183306-20231101032154
    - velero/velero-imdc-20231031120006-20231031221148
    - velero/velero-imdc-20231031180007-20231031214800
    - velero/velero-imdc-20231101201654-20231101205439
    - velero/velero-imdc-20231101201654-20231101225032
  velero.io/v1/Schedule:
    - velero/velero-imdc
  velero.io/v2alpha1/DataDownload:
    - velero/velero-imdc-20231030183306-20231030204827-hcrbx
    - velero/velero-imdc-20231031180007-20231031214800-cssf6
    - velero/velero-imdc-20231101201654-20231101205439-hvhjs
    - velero/velero-imdc-20231101201654-20231101205439-m7fpx
    - velero/velero-imdc-20231101201654-20231101205439-tvrnb
  velero.io/v2alpha1/DataUpload:
    - velero/velero-imdc-20231014120022-22nb7
    - velero/velero-imdc-20231014120022-26cqh
    - velero/velero-imdc-20231014120022-2jj5l
    - velero/velero-imdc-20231014120022-2wtwh
    - velero/velero-imdc-20231014120022-2zrlt
    - velero/velero-imdc-20231014120022-428xg
    - velero/velero-imdc-20231014120022-5wztf
    - velero/velero-imdc-20231014120022-62sb9
    - velero/velero-imdc-20231014120022-77k7p
    - velero/velero-imdc-20231014120022-7fzjj
    - velero/velero-imdc-20231014120022-8mqzs
    - velero/velero-imdc-20231014120022-9jdtp
    - velero/velero-imdc-20231014120022-b8gm4
    - velero/velero-imdc-20231014120022-bcsvm
    - velero/velero-imdc-20231014120022-d275t
    - velero/velero-imdc-20231014120022-d5dv7
    - velero/velero-imdc-20231014120022-d8xc7
    - velero/velero-imdc-20231014120022-df8h8
    - velero/velero-imdc-20231014120022-drxv4
    - velero/velero-imdc-20231014120022-dtj4z
    - velero/velero-imdc-20231014120022-dwhv9
    - velero/velero-imdc-20231014120022-hfmlh
    - velero/velero-imdc-20231014120022-hj9b8
    - velero/velero-imdc-20231014120022-jgdj5
    - velero/velero-imdc-20231014120022-k57c9
    - velero/velero-imdc-20231014120022-kg59k
    - velero/velero-imdc-20231014120022-lgcxg
    - velero/velero-imdc-20231014120022-ls8s5
    - velero/velero-imdc-20231014120022-m4wmc
    - velero/velero-imdc-20231014120022-mrksz
    - velero/velero-imdc-20231014120022-n6nbs
    - velero/velero-imdc-20231014120022-p6hmd
    - velero/velero-imdc-20231014120022-pcbpj
    - velero/velero-imdc-20231014120022-qkxpf

Most resources in the velero namespace cannot be backed up, if you want to backup the entire cluster, please exclude velero namespace.

[zanderson-aim]

Makes sense, I will update the job and test again. I guess these makes all the "old" backups that include velero already unrestorable?

[Lyndon-Li]

I guess these makes all the "old" backups that include velero already unrestorable?

yes

[shawn-hurley]

If this is true, can we change the default to permanently exclude the velero namespace, or even better, maybe, always exclude velero resources by default?

[zanderson-aim]

I can confirm it worked after I excluded the namespace.

[zanderson-aim]

I did tried some older backups before 1.12 and they worked fine with the namespace included. Must be something with changes in 1.12.

[Lyndon-Li]

I did tried some older backups before 1.12 and they worked fine with the namespace included. Must be something with changes in 1.12.

OK, I will keep this issue open and see if we can make some improvement around DataUpload CRs. However, we will not take this as high priority since we don't see too much value to include velero namespace into the backup, and even it is required, the DataUpload CRs could be filtered out as a workaround.

[shawn-hurley]

@shubham-pampattiwar @sseago We probably want to make sure that this is documented in our downstream docs

[oboudry-mvp]

For info I had exactly the same issue with v.12.1, restore of snapshots (Kopia to Minio) were failing when velero was included in the backup. I excluded all kube- namespaces but left velero in.

New backups without velero namespace restore fine.

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days. If a Velero team member has requested log or more information, please provide the output of the shared commands.

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days. If a Velero team member has requested log or more information, please provide the output of the shared commands.

[filipe-silva-magalhaes-alb]

I have the same issue.

I solved this by cleaning the s3 bucket, installing the Velero doing the backup again, and restoring.

Kopia might get confused when the bucket has multiple backups.

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days. If a Velero team member has requested log or more information, please provide the output of the shared commands.

[dharanui]

Facing same issue when trying to restore with velero version 1.14