Open bobobo1618 opened 5 years ago
Have my backup process pipe all confirmed-uploaded bytes to zstreamdump and use the resume token to resume the backup when interrupted.
That won't necessarily work, because you don't know if the target has actually written the entire contents of the stream.
Read my backup using a cloud server provided by the hosting platform and produce a resume token
That make sense - reading the zstream file and producing a resume token.
That won't necessarily work, because you don't know if the target has actually written the entire contents of the stream.
True that this won't work in all cases but it can work in some. For example one backup tool I use breaks the sent stream into chunks. Those chunks can be written to stdout when completed and it will be guaranteed that they were successfully stored.
AWS (FSx for OpenZFS) would also absolutely love it if this functionality were implemented.
This is an Enhancement.
My primary use-case is resuming failed backups to offline storage (i.e.
zfs send
to Amazon S3 or a similar service).The problem is that right now, if a transfer is interrupted, there's no way to resume it:
receive
on the other side so I can't get the token from the destination filesystem.receive
it to. Doing the receive will cost a ton of bandwidth (expensive at cloud providers) and/or I/O.But from my quick reading on the internals of
receive_resume_token
, it mostly boils down to an (object, offset) pair, which should be pretty easy to produce.I think these problems could be solved with a simple tool that reads in a
send
stream and produces a resume token. I think this might be best added tozstreamdump
as an option.I would then be able to:
zstreamdump
and use the resume token to resume the backup when interrupted. or