Hybrid VolumeSnapshot/Restic backup (consistent online backup)

[asoltesz]

Started as: "Store VolumeSnapshots in remote repository"

Based on the guides, it seems, that Stash cannot store VolumeSnapshots in the backup repository, thus it cannot be used for inter-cloud migrations and not even for VolumeSnapshot based disaster recovery with several cloud providers like Hetzner Cloud.

Use case:

I use Rook/Ceph as the distributed cloud storage on my k8s cluster. This does not depend on the cloud provider (Hetzner in my case) but is internal of the cluster and has the same volume snapshotting capability as say GCE. (My Rook/Ceph storage cluster is always part of my Kubernetes cluster.)

I want to be able to take snapshots of my Ceph backed PVs and store them in my remote backup repository (e.g.: a Backblaze B2 bucket).

I want to be able to rebuild my k8s cluster from backups in case of the complete loss of a cluster. My cluster is single-datacenter so the loss of a datacenter would be reason enough for rebuilding in another datacenter from B2 or S3 backup repositories.

Note: similar software like Velero supports this functionality. (Turns out Velero doesn't, as of v1.4)

[hossainemruz]

Based on the guides, it seems, that Stash cannot store VolumeSnapshots in the backup repository,

Stash utilizes Kubernetes VolumeSnapshot API. So, it is dependent on the respective CSI driver. Hence, it is not able to store the respective data in a different cloud storage. You can use Restic driver for this purpose.

I want to be able to rebuild my k8s cluster from backups in case of the complete loss of a cluster. My cluster is single-datacenter so the loss of a datacenter would be reason enough for rebuilding in another datacenter from B2 or S3 backup repositories.

We have plan to support this feature in future. We need time to implement this. You can check the respective addon here.

[asoltesz]

Stash utilizes Kubernetes VolumeSnapshot API. So, it is dependent on the respective CSI driver. Hence, it is not able to store the respective data in a different cloud storage. You can use Restic driver for this purpose.

Ok, reading up a bit, I get that the VolumeSnapshotContent cannot be moved directly between cloud providers. Actually, this wouldn't be an issue for me because I would always use my own Rook/Ceph CSI for these kinds of snapshot backups but I understand that this would be hard / impossible to generalize.

How about using volume snapshotting in combination with Restic like this? :

• A BackupConfiguration could contain settings for both VolumeSnapshot and Restic backup.

• Stash takes a VolumeSnapshot according to the BackupConfiguration schedule, so a new VolumeSnapshotContent is created by the CSI driver

• Stash creates a standalone, temporary Restic pod and in it uses a newly created PVC which is sourced from the newly taken VolumeSnapshot

• Stash runs Restic on the new PV and stores the files in the Repository according to the BackupConfiguration settings

• After successful backup, Stash drops the the temporary Restic pod and also drops the temporary backup PV/PVC and the VolumeSnapshot(Content).

Restore would be the same as a normal Restic restore.

This would allow Stash to take fully-consistent, online backups from any kind of workload which is placed on a volume of a VolumeSnapshot-capable CSI driver. Additionally, the backup would be out-of-cluster and out-of-datacenter due to the remote backup repo.

Is this behavior possible with Stash at the moment? Maybe it is, only not documented in the guides.

We have plan to support this feature in future. We need time to implement this. You can check the respective addon here.

Thanks, I will take a look at that addon.

[asoltesz]

I imagine that all necessary code is already available in Stash for this, since all elements of it are already supported.

Only the way of applying those already implemented functions need to be organized into a backup variation.

[rofra]

I really appreciate this feature proposal, as it would join the best world: VolumeSnapshot + Restic

[hossainemruz]

I think this types of features should be implemented through Addons. We don't want to make Stash's internal logic more complicated (already fairly complicated). If any of you want to contribute an Addon for this feature, I will be happy to guide.

[asoltesz]

@hossainemruz I think this is a major feature that should not be implemented by some external addon but by Stash itself. Without this, you can only take consistent online backups with snapshots which are VERY limited in capability compared to Restic backups (incremental backups, storage space utilization, migration between PV storageclasses, migration between cloud regions...etc).

[murkylife]

I agree with the previous comments. I love the approach of Stash, but i think this is a pretty needed feature. The number of snapshots you can have varies on your filesystem and cloud, and in some cases you'll be paying alot of money or IOPS for the convenience.

Being able to take a snapshot instantly via the VolumeSnapshot and then back it up to s3/b2 is a great solution.