oetiker / znapzend

zfs backup with remote capabilities and mbuffer integration.
www.znapzend.org
GNU General Public License v3.0
607 stars 137 forks source link

Feature request: Option to only destroy snapshots created by znapzend itself #489

Closed rholighaus closed 3 years ago

rholighaus commented 4 years ago

We have been using znapzend for remote backups on a Proxmox PVE cluster. Znapzend is a wonderful system, thank you so much for it. Proxmox is also great and one of the big pushers for ZFS on Linux. While PVE has a great built-in option for file system replication within a cluster, it lacks sophisticated off-site replication features, which is why we introduced znapzend for this purpose.

Lately, however, znapzend seems to interfere performance on our two production servers. There is also a problem when migrating containers from one host to the other and keep znapzend settings up and running - they have to migrated to, otherwise the same happens as described below.

The two production servers are regularly replicated using PVE's ZFS replication feature to a host that is not performance critical. I would love to just replicate the file systems from this non-critical system to a remote site using znapzend, but znapzend regularly breaks PVE's replication by destroying snapshots required by PVE. Here's why - and I have a suggestion for a work-around:

To align its replication, PVE looks for snapshots on the destination that it created with the last replication. They have a format like

rpool/data/subvol-130-disk-0@__replicate_130-1_1591224303__

When znapzend deletes one of those snapshots on the source system, this will destroy PVE replication.

To prevent this from happening, It would help if either znapzend either only deletes snapshots in its pre-configured tsformat and hence skips those snapshots not created by itself (unless they are using the same time format) - or it introduces a blacklist option to blacklist snapshots of a specific naming from being destroyed.

jimklimov commented 4 years ago

It actually should filter the snapshot names that it deletes on destination to only match its timestamp based pattern, see Time.pm. Would not hurt to verify that this code only matches this pattern as the snapshot name (and not e.g. a subset like something that looks like ten digits by default).

I am not sure about out-of-the-box settings currently, my setups explicitly define the znapzend pattern like znapzend-auto-%Y-%m-%dT%H:%M:%SZ thus by chance ensuring a unique namespace for the automatic snapshot cleanup - and more for me to differentiate these vs. ones I make by hand. This certainly works to not remove manually-made snapshots. (Note also that if I want to keep some automatically made snapshots, I can just recursively rename them in the dataset tree).

psy0rz commented 3 years ago

ZFS-autobackup does this correctly by default, please check it out at: https://github.com/psy0rz/zfs_autobackup

Install it with: pip install --upgrade --pre zfs-autobackup

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.