search

oetiker / znapzend

zfs backup with remote capabilities and mbuffer integration.
www.znapzend.org
GNU General Public License v3.0
613 stars 137 forks source link

feature discussion: automatically create bookmarks for snapshots that have been sent #330

Closed JMoVS closed 3 years ago

JMoVS commented 6 years ago

Hi,

I've recently been bitten by my DST being down and the source being cleanup up to the point where there were no common snapshots left.

Even with #329 and hence #322 gets solved, it would cost basically nothing to bookmark all snapshots that have been sent to the dst.

Bookmarks are only a pointer to a specific TxG group so don't consume any space but allow to send a snapshot from source to target even if there are no common snapshots left. The dst needs the corresponding snapshot to the original one from the bookmark - but as the problem is that the dst was down - it is unlikely that it has been deleted.

My naive way of implementing this would be to just create a bookmark after a successfull send on the source and then if ever needed let the user manually send (an automatic send would be nice but is above my capabilities in PERL).

My first attempt can be found at #331 and I need help getting it to work and figure out how exactly this could be integrated into znapzend.

papamoose commented 6 years ago

I've been in the same situation before and would love if we can get this working. Since this is a ZFS feature we should make sure to check that it is available and enabled before attempting to create a bookmark.

# zpool get feature@bookmarks tank -H -o value
enabled
JMoVS commented 6 years ago

problem is, I don't really have an idea on how to proceed.

@papamoose I suggest the following strategy (to be implemented): When znapzend detects that the destination is unreachable, create a bookmark for every existing snapshot (bookmarks are super cheap in contrast to snapshots). When the destination becomes available/is available but no common snapshots found, find the newest common bookmark and resume sending from there.

redmop commented 6 years ago

Why do it on fail? Why not do it on success? The "I successfully synced everything to the destination up to here" bookmark. That also gives us something to query to see how backups are going.

JMoVS commented 6 years ago

true. Bookmark should be the oldest snapshot on the source that also exists on the destination.

But znapzend is written in PERL and I have literally no idea how and where I would hook that code up

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

mishehu commented 3 years ago

Seeing if by commenting on this issue if it flips back from closed. I do have an interest in this sort of feature, as I rotate backup media in and out, and I end up in this sort of a position as well. So consider this a ping...