znapzend being weirdly selective about syncing snapshots

[saurabhnanda]

I have the following backup plan for a dataset:

dst_rsynctank_plan= 26weeks=>1week,28days=>1day,3days=>15minutes,1day=>5minutes
src_plan        = 26weeks=>1week,28days=>1day,3days=>15minutes,1day=>5minutes

The DST was some other server and I changed it to a new one, with a complete blank ZFS filesystem. The first invocation of znapzend kept running for 2-3 days and managed to sync ~530GB of data.

Here's what got synced with the new DST:

• the current/live ZFS dataset
• All 1day => 5min snapshots
• All 3day => 15min snapshots

Here's what has not been synced with the new DST, and it seems that znapzend is not even attempting to sync them (by looking at the logs):

• 26weeks=>1week
• 28days=>1day

What's going on, and how do I debug this further?

[saurabhnanda]

I enabled syncing for two new ZFS datasets:

• One has a minimum snapshot frequency of 1 hour
• The other has a minimum snapshot frequency of 1 day

Both have not even begun syncing yet. Is this because the snapshot get synced with DST only when a new remote snapshot actually gets created?

So, all hourly snapshots for the first dataset should start syncing to DST when the next hourly snapshot is created at SRC?

Similarly, all daily snapshots for the second dataset should start syncing to DST when the next daily snapshot is created at SRC?

[jimklimov]

By the code logic and with default CLI arguments, znapzend essentially:

• snapshots source
• sends the latest snapshot to each destination (as an increment with intermediates from latest common snapshot, or from scratch... probably also with intermediates... if it has just autoCreated the destination)
• cleans up older snapshots on destinations by retention schedule
• cleans up older snapshots on source by retention schedule

By default it sends all intermediate snapshots (between latest common and newly created) and then weeds out the obsoleted ones (note this also includes snapshots that could have been made manually outside znapzend - they would stay on destination until cleaned manually, and can serve as backup last-common snapshots for manual fixes of replication failures).

It also does not send any snapshots older than the last common one.

You can have an option to --skipIntermediates to cover such gap in one hop and just not have the snapshots in-between on the destination.

So any of these can contribute to not-seeing snapshots on the destination... are the older (26weeks=>1week and 28days=>1day) snapshots still present on source? I would expect these to have been sent over when znapzend autocreated the destinations on your new server, at least with intermediates enabled...

As for debugging the situation, there is a --debug flag that logs a lot of data about the inputs and decisions at different stages in the script.

[jimklimov]

UPDATE: Actually no, for an initial send (no snapshots on destination), there is no increment to go from, so znapzend just rolls it back and sends the last snapshot (newly created) as the first common checkpoint - so this may be what happens to your setup with a new clean destination. I've posted #513 to track this nuance from development perspective.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.