search

digint / btrbk

Tool for creating snapshots and remote backups of btrfs subvolumes
https://digint.ch/btrbk/
GNU General Public License v3.0
1.67k stars 122 forks source link

Enable --progress-total option using pv. #547

Open kpoeppel opened 1 year ago

kpoeppel commented 1 year ago

Add a --progress-total option that internally runs btrbk diff first to get a total size to be transferred (per target). This option is an alternative to --progress and shows a ETA time estimate as well as progress bar via pv.

Aims to resolve #543 .

kpoeppel commented 1 year ago

Thanks for the MR, looks promising. I've had a brief look at it, and did not have time to test it yet (I'm quite busy at the moment, this might take a while sorry).

Do you have any numbers on how good are the estimates are and how long it takes to calculate the diffs? I'm especially curious as the btrfs send-stream adds quite some metadata.

Speed-wise it was pretty fast / negligible compared to the transfer time for all my cases.

kpoeppel commented 1 year ago

Another general comment: Handling the stderr output is not optimal in the current code, and I used a workaround to feed stderr of the subprocess directly to stderr of the main script. This way errors are not captured. Maybe the chomp + readline combination should be replaced by something involving select. Also handling the stderr updates of pv or mbuffer (removing parts of their outputs over time) makes it difficult probably.

kpoeppel commented 8 months ago

Any update / further requests for the PR?

tbertels commented 5 months ago

I'd suggest updating the --progress entry: --progress show progress bar on send-receive operation There's no progress bar with --progress since the switch to mbuffer, just transferred size and speed.

A man entry should be added too.

Note that there wasn't any commit here since August 2023, so digint is probably busy.

It would be nice to show total progress even for the initial snapshot, at least if quotas are enabled (otherwise it can be pretty slow to compute for HDDs with lots of data). sudo btrfs subvolume show [snapshot] then displays the snapshot size.

Still, it's pretty useful already.