search

openzfs / zfs

OpenZFS on Linux and FreeBSD
https://openzfs.github.io/openzfs-docs
Other
10.38k stars 1.72k forks source link

Add receive_resume_token easier to zstreamdump #8658

Open bobobo1618 opened 5 years ago

bobobo1618 commented 5 years ago

This is an Enhancement.

My primary use-case is resuming failed backups to offline storage (i.e. zfs send to Amazon S3 or a similar service).

The problem is that right now, if a transfer is interrupted, there's no way to resume it:

But from my quick reading on the internals of receive_resume_token, it mostly boils down to an (object, offset) pair, which should be pretty easy to produce.

I think these problems could be solved with a simple tool that reads in a send stream and produces a resume token. I think this might be best added to zstreamdump as an option.

I would then be able to:

ahrens commented 5 years ago

Have my backup process pipe all confirmed-uploaded bytes to zstreamdump and use the resume token to resume the backup when interrupted.

That won't necessarily work, because you don't know if the target has actually written the entire contents of the stream.

Read my backup using a cloud server provided by the hosting platform and produce a resume token

That make sense - reading the zstream file and producing a resume token.

bobobo1618 commented 5 years ago

That won't necessarily work, because you don't know if the target has actually written the entire contents of the stream.

True that this won't work in all cases but it can work in some. For example one backup tool I use breaks the sent stream into chunks. Those chunks can be written to stdout when completed and it will be guaranteed that they were successfully stored.

bethtirado commented 4 months ago

AWS (FSx for OpenZFS) would also absolutely love it if this functionality were implemented.