search

openebs / zfs-localpv

Dynamically provision Stateful Persistent Node-Local Volumes & Filesystems for Kubernetes that is integrated with a backend ZFS data storage stack.
https://openebs.io
Apache License 2.0
439 stars 105 forks source link

Restore is impossible ? cannot receive incremental stream: most recent snapshot of XXXXX does not match incremental source #429

Open JBaptisme33 opened 1 year ago

JBaptisme33 commented 1 year ago

What steps did you take and what happened:

I have 2 servers running k3s with zfs on local partition. I've configured cross-replications between these 2 servers like with velero + minio Each server have a minio instance (minio namespace / pod is excluded from velero backup) , and each day Velero located in each server send a backup to each other (manifests backup + volume snapshot backup)

In each backup location, for a specific PVC, I have zfs backup like follow (which seem reprensant à zfs incremental volume

➜  ~ ls -alh /mnt/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX-20221226044520*
-rw-r--r--. 1 XXXXXXXXX XXXXXXXXX 35K Dec 26 05:49 /mnt/XXXXXXXXXXXXXXXXXXXXXXX/XXXXXXXXXXXXXXX-pvc-1ab8af82-181d-40eb-975c-5150b0be5586-XXXXXXXXXXXXXXXXXXXXXXXXXXXX
-rw-r--r--. 1 XXXXXXXXX XXXXXXXXX 1.4K Dec 26 05:49/mnt/XXXXXXXXXXXXXXXXXXXXXXX/XXXXXXXXXXXXXXX-pvc-1ab8af82-181d-40eb-975c-5150b0be5586--XXXXXXXXXXXXXXXXXXXXXXXXXXXX.zfsvol

What did you expect to happen:

I need to restore a destroyed PVC by misstake on server 1 (zfs volume into server 1 was removed to beause of Retain Policy to Delete). So I tried to restore PVC from Minio remote backup via Velero tool Restore example is as follow:

velero   restore create --from-backup  XXXXXXXXXXXX-20221224044517    --restore-volumes=true  -l app.kubernetes.io/name=grafana -n XXXXXXXXXXXXX`

Restore is impossible because of incremental zfs snapshot it seem ?

First error when trying to restore PVC from remote minio instance:

E1226 15:22:42.398226       1 zfs_util.go:892] zfs: could not restore the volume XXXXXXXXX/pvc-1ab8af82-181d-40eb-975c-5150b0be5586 cmd [-c nc -w 3 192.168.168.109 9010 | zfs recv -o quota=4294967296 -o recordsize=4k -o mountpoint=legacy -o dedup=off -o compression=lz4 -F XXXXXXXXX/pvc-1ab8af82-181d-40eb-975c-5150b0be5586] error: cannot receive incremental stream: destination 'XXXXXXXXX/pvc-1ab8af82-181d-40eb-975c-5150b0be5586' does not exist

So I create zvol called pvc-1ab8af82-181d-40eb-975c-5150b0be5586 Afterthat, I have error regarding incremental snapshot:

E1226 14:04:25.819776       1 zfs_util.go:892] zfs: could not restore the volume XXXXXXXXX/pvc-1ab8af82-181d-40eb-975c-5150b0be5586 cmd [-c nc -w 3 192.168.168.109 9010 | zfs recv -o quota=4294967296 -o recordsize=4k -o mountpoint=legacy -o dedup=off -o compression=lz4 -F XXXXXXXXX/pvc-1ab8af82-181d-40eb-975c-5150b0be5586] error: cannot receive incremental stream: most recent snapshot of XXXXXXXXX/pvc-1ab8af82-181d-40eb-975c-5150b0be5586 does not
match incremental source

For info, incrBackupCount: was configured to "10" and velero backup ttl was configured to 240h

Is my setup broken by design ? Is it possible to make a cross replication setup with only zfs-localpv / minio and velero ? Does I need to restore some another ressources like zfsvolumes.zfs.openebs.io manifests before restore my complete backup ?

Thanks

JBaptisme33 commented 1 year ago

Anyone have an idea :) ?

1lann commented 1 year ago

I have the same problem, I believe the implementation of backups are buggy as some of my full backups seem to be too small to be possible. I also believe this is the case because there is no checksum verification of backups sent over the cluster network. If the connection is interrupted, the driver assumes the backup is complete.