digint / btrbk

Tool for creating snapshots and remote backups of btrfs subvolumes
https://digint.ch/btrbk/
GNU General Public License v3.0
1.58k stars 116 forks source link

Not optimal behavior for low quality network connections #584

Open mio-19 opened 3 months ago

mio-19 commented 3 months ago

After experiencing numerous failures due to unstable network connections, I discovered that btrbk is set to send a relatively large incremental backup instead of smaller incremental backups, making it challenging to complete the backup process under unstable network conditions. Here is what btrbk currently do (I used a port forward program because one machine doesn't have public ip address)

btrbk run -n
WARNING: Target subvolume "127.0.0.1[25566]:/mnt/Rescue1/HDDPool-Y.20240314T0030" exists, but is not a receive target of "/mnt/HDDPool-Y/btrbk_snapshots
/HDDPool-Y.20240314T0030"
WARNING: Please delete stray subvolumes: "btrbk clean 127.0.0.1[25566]:/mnt/Rescue1"
WARNING: Skipping backup of: /mnt/HDDPool-Y/btrbk_snapshots/HDDPool-Y.20240314T0030                                                                     --------------------------------------------------------------------------------
Backup Summary (btrbk command line client, version 0.32.6)                                                                                                                                                                                                                                                          Date:   Sat Mar 16 13:09:36 2024
    Config: /etc/btrbk/btrbk.conf                                                                                                                           Dryrun: YES
                                                                                                                                                        Legend:                                                                                                                                                     ===  up-to-date subvolume (source snapshot)                                                                                                             +++  created subvolume (source snapshot)                                                                                                                ---  deleted subvolume
    ***  received subvolume (non-incremental)
    >>>  received subvolume (incremental)
--------------------------------------------------------------------------------
/mnt/HDDPool-Y
+++ /mnt/HDDPool-Y/btrbk_snapshots/HDDPool-Y.20240316T1309
--- /mnt/HDDPool-Y/btrbk_snapshots/HDDPool-Y.20240226T0032
--- /mnt/HDDPool-Y/btrbk_snapshots/HDDPool-Y.20240227T0032                                                                                              --- /mnt/HDDPool-Y/btrbk_snapshots/HDDPool-Y.20240228T0128
--- /mnt/HDDPool-Y/btrbk_snapshots/HDDPool-Y.20240229T0430                                                                                              --- /mnt/HDDPool-Y/btrbk_snapshots/HDDPool-Y.20240301T0430                                                                                              --- /mnt/HDDPool-Y/btrbk_snapshots/HDDPool-Y.20240309T0436
--- /mnt/HDDPool-Y/btrbk_snapshots/HDDPool-Y.20240309T1230                                                                                              --- /mnt/HDDPool-Y/btrbk_snapshots/HDDPool-Y.20240309T1652                                                                                              --- /mnt/HDDPool-Y/btrbk_snapshots/HDDPool-Y.20240310T0433
--- /mnt/HDDPool-Y/btrbk_snapshots/HDDPool-Y.20240310T0506                                                                                              --- /mnt/HDDPool-Y/btrbk_snapshots/HDDPool-Y.20240310T0830                                                                                              --- /mnt/HDDPool-Y/btrbk_snapshots/HDDPool-Y.20240310T1230                                                                                              --- /mnt/HDDPool-Y/btrbk_snapshots/HDDPool-Y.20240310T1630                                                                                              --- /mnt/HDDPool-Y/btrbk_snapshots/HDDPool-Y.20240310T2030
--- /mnt/HDDPool-Y/btrbk_snapshots/HDDPool-Y.20240311T0430
--- /mnt/HDDPool-Y/btrbk_snapshots/HDDPool-Y.20240311T0830
--- /mnt/HDDPool-Y/btrbk_snapshots/HDDPool-Y.20240311T1230
--- /mnt/HDDPool-Y/btrbk_snapshots/HDDPool-Y.20240311T1630
--- /mnt/HDDPool-Y/btrbk_snapshots/HDDPool-Y.20240311T1729
--- /mnt/HDDPool-Y/btrbk_snapshots/HDDPool-Y.20240312T0905
--- /mnt/HDDPool-Y/btrbk_snapshots/HDDPool-Y.20240312T0915
--- /mnt/HDDPool-Y/btrbk_snapshots/HDDPool-Y.20240312T1912
--- /mnt/HDDPool-Y/btrbk_snapshots/HDDPool-Y.20240312T1915
!!! 127.0.0.1[25566]:/mnt/Rescue1/HDDPool-Y.20240314T0030
>>> 127.0.0.1[25566]:/mnt/Rescue1/HDDPool-Y.20240315T0103
>>> 127.0.0.1[25566]:/mnt/Rescue1/HDDPool-Y.20240315T0222
>>> 127.0.0.1[25566]:/mnt/Rescue1/HDDPool-Y.20240315T0426
>>> 127.0.0.1[25566]:/mnt/Rescue1/HDDPool-Y.20240315T0430
>>> 127.0.0.1[25566]:/mnt/Rescue1/HDDPool-Y.20240315T0434
>>> 127.0.0.1[25566]:/mnt/Rescue1/HDDPool-Y.20240315T0830
>>> 127.0.0.1[25566]:/mnt/Rescue1/HDDPool-Y.20240315T1943
>>> 127.0.0.1[25566]:/mnt/Rescue1/HDDPool-Y.20240316T0403
>>> 127.0.0.1[25566]:/mnt/Rescue1/HDDPool-Y.20240316T0430
>>> 127.0.0.1[25566]:/mnt/Rescue1/HDDPool-Y.20240316T1257
>>> 127.0.0.1[25566]:/mnt/Rescue1/HDDPool-Y.20240316T1309
--- 127.0.0.1[25566]:/mnt/Rescue1/HDDPool-Y.20240220T0032
--- 127.0.0.1[25566]:/mnt/Rescue1/HDDPool-Y.20240221T0032
--- 127.0.0.1[25566]:/mnt/Rescue1/HDDPool-Y.20240222T0032
--- 127.0.0.1[25566]:/mnt/Rescue1/HDDPool-Y.20240223T0032
--- 127.0.0.1[25566]:/mnt/Rescue1/HDDPool-Y.20240224T0032
--- 127.0.0.1[25566]:/mnt/Rescue1/HDDPool-Y.20240309T0436
--- 127.0.0.1[25566]:/mnt/Rescue1/HDDPool-Y.20240309T1230
--- 127.0.0.1[25566]:/mnt/Rescue1/HDDPool-Y.20240309T1652
--- 127.0.0.1[25566]:/mnt/Rescue1/HDDPool-Y.20240310T0433
--- 127.0.0.1[25566]:/mnt/Rescue1/HDDPool-Y.20240310T0506
--- 127.0.0.1[25566]:/mnt/Rescue1/HDDPool-Y.20240310T0830
--- 127.0.0.1[25566]:/mnt/Rescue1/HDDPool-Y.20240310T1230
--- 127.0.0.1[25566]:/mnt/Rescue1/HDDPool-Y.20240310T1630
--- 127.0.0.1[25566]:/mnt/Rescue1/HDDPool-Y.20240310T2030
--- 127.0.0.1[25566]:/mnt/Rescue1/HDDPool-Y.20240311T0430
--- 127.0.0.1[25566]:/mnt/Rescue1/HDDPool-Y.20240311T0830
--- 127.0.0.1[25566]:/mnt/Rescue1/HDDPool-Y.20240311T1230
--- 127.0.0.1[25566]:/mnt/Rescue1/HDDPool-Y.20240311T1630
--- 127.0.0.1[25566]:/mnt/Rescue1/HDDPool-Y.20240311T1729
--- 127.0.0.1[25566]:/mnt/Rescue1/HDDPool-Y.20240312T0905

NOTE: Some errors occurred, which may result in missing backups!
Please check warning and error messages above.
Please delete stray subvolumes: "btrbk clean 127.0.0.1[25566]:/mnt/Rescue1"

NOTE: Dryrun was active, none of the operations above were actually executed!

For a better experience on low quality network, a strategy is to send 20240312 snapshots first