Data Mover - preserve local snapshot when snapshot data movement is enabled

[Lyndon-Li]

Since local snapshot restoration is faster than data movement, it is a useful feature to preserve several local snapshots when snapshot data movement is enabled. The count of the preservation should be configurable by users. Moreover, if the data movement fails, the local snapshot may also be kept.

There will be two configurable options:

• Keep the last X local snapshot
• If data movement fails, always keep local snapshot

There are some details to cover in order to manage the local snapshots together with the remote data movement data, let's see how we can implement this.

[sseago]

We should make sure that if we want to keep last X local snapshot that "0" is an allowed value for X. Some users would rather not keep any artifacts in-cluster for things that are fully restorable from backup, since this increases storage costs.

[Lyndon-Li]

@sseago Sure, will do this.

For most of the on-premise storages, preserving local snapshots is either impossible or too costing. This requirement is primarily from the users who are running clusters in a public cloud and who also want to move data across cloud providers or who require some advanced data retention/management that the local snapshot cannot provide.

The major benefit to keep the local snapshots is that the restore is much faster from local snapshots.

Regarding to the cost for this kind of single-public-cloud-provider usage, the cost for local snapshot vs. data mover + simple object store is an interesting topic:

• The local snapshot data is stored in the object store by the cloud provider, the storing will be charged
• The management, manipulation of the local snapshot is charged additionally
• For data mover + simple object store: the storing of the backup data is charged and the data downloading is also charged (we may be able to optimize this)
• The comparison result of who stores more data in the object store for the same volume is unclear -- most cloud providers store the snapshot data incrementally from block level vs. Velero's repository (Kopia repository) has data deduplication
• Different cloud providers have different charging themes

[reasonerjt]

We should make sure that if we want to keep last X local snapshot that "0" is an allowed value for X. Some users would rather not keep any artifacts in-cluster for things that are fully restorable from backup, since this increases storage costs.

This is the current behavior of v1.12.0

How to expose the setting, i.e. is it per backup or per {PV, storageclass...} involves the decision on #6638

But we may start a design without touching exposing the setting, for example PoC reading the number for local snapshot from environment variable.