Backup once a day, but znapzend is sending snapshots more often

[Dacesilian]

I have specified three remote destination and want to save bandwidth, so for dst_c, I want to backup only once a day. So I have this configuration. But snapshots are being sent very often, I think each 15 minutes and then gets deleted from remote. How can I send snapshots only once a day to that dst_c destination? Thanks.

dst_c=root@server.com:tank/backup/container
dst_c_plan=30days=>12hours
enabled=on
mbuffer=/usr/bin/mbuffer
mbuffer_size=1G
post_znap_cmd=off
pre_znap_cmd=off
recursive=on
src=tank/container
src_plan=8hours=>15minutes,6days=>2hours,30days=>12hours
tsformat=%Y-%m-%d-%H%M%S
zend_delay=420

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Dacesilian]

Please don't thank me for my contribution since it's useless, because no one has responded to it yet 👎

[Dacesilian]

It's probably synchronizing each 15 minutes, I get that. Can we for example add another parameter to sync only once a defined period?

[jochendemuth]

Same use case here.

According to znapzendzetup manual "pre-send-command [...] Run command/script before and after sending the snapshot to the destination. Intended to run a remote script via ssh on the destination, e.g. to bring up a backup disk or server. Or to put a zpool online/offline:"

In my mind this is extremely useful if znapzend were to allow different backup schedules across destinations. E.g. frequent (15min/hourly) local snapshots to fix user errors, less frequent (daily/weekly) snapshots to remote locations for disaster recovery. Bringing up a backup remote server (as mentioned in the documentation) every 15 minutes is highly inefficient if it is intended to hold daily snapshots. When I tried this out I expected that the destination-specific pre/post send commands would only execute on the destination-specific schedule, not on the local schedule.

I consider the current behavior a bug.

[user@localhost ~]# dnf list znapzend Last metadata expiration check: 1:02:43 ago on Fri 29 Oct 2021 09:36:58 AM EDT. Installed Packages znapzend.x86_64 0.20.0-1.fc32 @copr:copr.fedorainfracloud.org:spike:znapzend

[jochendemuth]

Noticed that I was running quite an old version of znapzend. Now I am using what looks like the most current release version. The upgrade did not change the behavior described above.

[user@localhost ~]# dnf list znapzend Last metadata expiration check: 0:17:23 ago on Fri 29 Oct 2021 03:41:35 PM EDT. Installed Packages znapzend.x86_64 0.21.0-1.fc34 @copr:copr.fedorainfracloud.org:spike:znapzend

[jochendemuth]

I spent some quality time with the sources.

Starting with line 1202 in sub createWorkers the next execution timestamp is calculated based on the src backup plan. zTime->getNextSnapshotTimestamp I cannot find a place where the next execution timestamp for a destination backup plan is calculated.

The snapWorker (sub starting 1142) first takes the local snapshot then starts the process of sending to destinations.

Local/new snapshots are created using sub createSnapshot starting at line 983. Sub sendRecvCleanup (starting line 264) is apparently responsible for the sending of snapshots to destination(s). It loops through all destinations starting line 301.

I can see that in ZnapZend.pm lines 318-334 of sub sendRecvCleanup the pre-send-cmd is executed. Until this point the destination backup schedule has not been taken into account.

I propose a refactoring that

1. Changes sub sendRecvCleanup to send/receive/cleanup only for a single destination. This requires adding destination as a parameter to sub sendRecvCleanup and removal of the loop in line 301.
2. Divides src snapshot creation and sendRecvCleanup for each destination onto separate schedules as indicated by their respective backup plans. This involves looping through destinations in sub createWorkers, looking up the repective next execution timestamp for src and each destination and scheduling execution separately.

I would argue against an alternative approach that would add another condition to the already lengthy sub sendRecvCleanup checking just ahead of line 318 trying to determine if anything should be sent to the current destination based on current timestamp and "next execution timestamp" for the current destination.

I am sure there are challenges along the way, but it doesn't look conceptually difficult.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.