search

vmware-tanzu / velero

Backup and migrate Kubernetes applications and their persistent volumes
https://velero.io
Apache License 2.0
8.73k stars 1.41k forks source link

The kopia snapshots are not removed after deleting the backup #6575

Closed ywk253100 closed 1 year ago

ywk253100 commented 1 year ago

The backups before deleting:

velero  backup get
NAME    STATUS            ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
dm-02   PartiallyFailed   1        0          2023-07-31 16:28:39 +0800 CST   29d       default            <none>
dm-03   PartiallyFailed   1        0          2023-07-31 17:20:38 +0800 CST   29d       default            <none>
dm-04   PartiallyFailed   1        0          2023-07-31 17:27:27 +0800 CST   29d       default            <none>
dm-05   PartiallyFailed   1        0          2023-07-31 17:29:56 +0800 CST   29d       default            <none>
dm-06   Completed         0        0          2023-08-01 08:19:49 +0800 CST   29d       default            <none>
dm-07   Completed         0        0          2023-08-01 08:22:40 +0800 CST   29d       default            <none>
dm-08   Completed         0        0          2023-08-01 08:28:54 +0800 CST   29d       default            <none>
dm-09   Completed         0        0          2023-08-01 08:29:41 +0800 CST   29d       default            <none>
dm-10   Completed         0        0          2023-08-01 14:06:31 +0800 CST   29d       default            <none>
dm-11   Completed         0        0          2023-08-01 15:14:56 +0800 CST   29d       default            <none>

The kopia snapshots are still there after deleting the backup:

kopia snapshot list --all
...
default@default:default/etcd0-pv-claim
  2023-08-01 08:23:09 CST k7738a8843e1b833272635aa59fe3a628 128 MB drwxr-xr-x files:3 dirs:5 (latest-6)
  2023-08-01 08:24:10 CST k8ff6c8e82a8a954821cd3026419a5c51 128 MB drwxr-xr-x files:3 dirs:5 (latest-5)
  2023-08-01 08:29:20 CST k6b8515b71eb19f2de6984d32643b3cdc 128 MB drwxr-xr-x files:3 dirs:5 (latest-4)
  2023-08-01 08:30:10 CST kd4b1a06cda551f2449a1f34df0c64412 128 MB drwxr-xr-x files:3 dirs:5 (latest-3,hourly-3)
  2023-08-01 14:07:06 CST k85eb0566905f3ae4f835ca1c96e71d8b 128 MB drwxr-xr-x files:3 dirs:5 (latest-2,hourly-2)
  2023-08-01 15:15:31 CST k3e03908dee8c6197574629cc573c184e 128 MB drwxr-xr-x files:3 dirs:5 (latest-1,hourly-1,daily-1,weekly-1,monthly-1,annual-1)

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

blackpiglet commented 1 year ago

Put some more information found during the test.

time="2023-08-02T03:37:39Z" level=info msg="Deleted snapshot 8ef2d8da1c81e345cb00f9f1394d1eee, namespace: upgrade, repo type: kopia" logSource="pkg/controller/backup_deletion_controller.go:532"
>>> kopia manifest list
......
eb43a4a85762cfb854effb75b053b8ae        798 2023-08-02 10:16:41 CST type:snapshot hostname:default path:upgrade/pvc2 snapshot-requester:snapshot-data-upload-download snapshot-uploader:kopia username:default velero.io/async-operation-id:du-37896da0-a5ba-4c62-a851-59c7d9197f91.0489457e-8359-4bb6259e9
9c903666f45f9219a0adb8eb89c94fae        835 2023-08-02 10:34:34 CST type:snapshot hostname:default path:upgrade/pvc1 snapshot-requester:snapshot-data-upload-download snapshot-uploader:kopia username:default velero.io/async-operation-id:du-9554778f-37f7-4b30-b1a1-89e39717006d.6fe77d24-d6e9-479752b28
5a06e90711ba686415a81e835d3105e9        798 2023-08-02 10:35:25 CST type:snapshot hostname:default path:upgrade/pvc2 snapshot-requester:snapshot-data-upload-download snapshot-uploader:kopia username:default velero.io/async-operation-id:du-9554778f-37f7-4b30-b1a1-89e39717006d.0489457e-8359-4bbe70373
2e4e8bf7f177ef2a78f282efbbf0b463        829 2023-08-02 11:06:01 CST type:snapshot hostname:default path:upgrade/kibishii-data-kibishii-deployment-0 snapshot-requester:snapshot-data-upload-download snapshot-uploader:kopia username:default velero.io/async-operation-id:du-5dc7f5aa-cdcb-48bf-a203-b383869425fb.a6583d4a-20aa-4cd77feff
8ef2d8da1c81e345cb00f9f1394d1eee        829 2023-08-02 11:36:37 CST type:snapshot hostname:default path:upgrade/kibishii-data-kibishii-deployment-0 snapshot-requester:snapshot-data-upload-download snapshot-uploader:kopia username:default velero.io/async-operation-id:du-1e4dcd6f-f4e4-49a8-a509-7cd96b5db521.a6583d4a-20aa-4cdc70e52
17b778e489605e0097e1b4f7c6cbc8d6        867 2023-08-02 11:43:01 CST type:snapshot hostname:default path:upgrade/kibishii-data-kibishii-deployment-0 snapshot-requester:snapshot-data-upload-download snapshot-uploader:kopia username:default velero.io/async-operation-id:du-0564981b-4ebd-4080-8457-ffe4a3426802.a6583d4a-20aa-4cdcf74d6
5f64868ae26ffa606fc8135687cfbbfd        867 2023-08-02 11:48:40 CST type:snapshot hostname:default path:upgrade/kibishii-data-kibishii-deployment-0 snapshot-requester:snapshot-data-upload-download snapshot-uploader:kopia username:default velero.io/async-operation-id:du-78f19f6a-5569-41ad-a432-c2768b97cec0.a6583d4a-20aa-4cd037c4c
Lyndon-Li commented 1 year ago

This is a generic problem for features that are using unified repo, specifically, FSB and data mover. So we will also cherry-pick this to 1.10 and 1.11