tlaurion
commented
6 months ago Open tlaurion opened 7 months ago
tlaurion
commented
6 months ago tlaurion
commented
6 months ago Some performance notes on backuping the local qube backup to qube-ssh remote archive
user@wyng-wdcloud:~$ rsync --archive --compress --whole-file --sparse --hard-links --human-readable --info=progress2 --exclude='*.tmp' /home/user/nv41/ root@192.168.3.1:/mnt/Backups/nv41/
465.03M 4% 5.96MB/s 0:01:14 (xfr#5450, ir-chk=2697/104473)^C
rsync error: unexplained error (code 255) at rsync.c(713) [sender=3.2.7]
user@wyng-wdcloud:~$ rsync --archive --sparse --hard-links --human-readable --info=progress2 --exclude='*.tmp' /home/user/nv41/ root@192.168.3.1:/mnt/Backups/nv41/
485.39M 4% 5.74MB/s 0:01:20 (xfr#4774, ir-chk=3579/110136)^C
rsync error: unexplained error (code 255) at rsync.c(713) [sender=3.2.7]
user@wyng-wdcloud:~$ rsync --archive --compress --sparse --hard-links --human-readable --info=progress2 --exclude='*.tmp' /home/user/nv41/ root@192.168.3.1:/mnt/Backups/nv41/
514.04M 4% 6.05MB/s 0:01:21 (xfr#4532, ir-chk=2402/113490)^C
rsync error: unexplained error (code 255) at rsync.c(713) [sender=3.2.7]
user@wyng-wdcloud:~$ rsync --archive --compress --whole-file --sparse --hard-links --human-readable --info=progress2 --exclude='*.tmp' /home/user/nv41/ root@192.168.3.1:/mnt/Backups/nv41/
372.23M 3% 5.88MB/s 0:01:00 (xfr#6202, ir-chk=1195/118485)^C
rsync error: unexplained error (code 255) at rsync.c(713) [sender=3.2.7]
user@wyng-wdcloud:~$ rsync --archive --compress --whole-file --hard-links --human-readable --info=progress2 --exclude='*.tmp' /home/user/nv41/ root@192.168.3.1:/mnt/Backups/nv41/
117.62M 1% 5.93MB/s 0:00:18 (xfr#1294, ir-chk=3839/122422)^C
rsync error: unexplained error (code 255) at rsync.c(713) [sender=3.2.7]
user@wyng-wdcloud:~$ rsync --archive --compress --hard-links --human-readable --info=progress2 --exclude='*.tmp' /home/user/nv41/ root@192.168.3.1:/mnt/Backups/nv41/
525.93M 4% 6.24MB/s 0:01:20 (xfr#5013, ir-chk=2895/127825)^C
rsync error: unexplained error (code 255) at rsync.c(713) [sender=3.2.7]
user@wyng-wdcloud:~$ rsync --archive --hard-links --human-readable --info=progress2 --exclude='*.tmp' /home/user/nv41/ root@192.168.3.1:/mnt/Backups/nv41/
509.16M 3% 6.06MB/s 0:01:20 (xfr#4838, ir-chk=2174/134879)^C
rsync error: unexplained error (code 255) at rsync.c(713) [sender=3.2.7]
user@wyng-wdcloud:~$ rsync --hard-links --human-readable --info=progress2 --exclude='*.tmp' /home/user/nv41/ root@192.168.3.1:/mnt/Backups/nv41/
skipping directory .
0 100% 0.00kB/s 0:00:00 (xfr#0, to-chk=0/0)
user@wyng-wdcloud:~$ rsync --archive --hard-links --human-readable --info=progress2 --exclude='*.tmp' /home/user/nv41/ root@192.168.3.1:/mnt/Backups/nv41/
162.15M 1% 6.06MB/s 0:00:25 (xfr#1318, ir-chk=2187/136211)^C
rsync error: unexplained error (code 255) at rsync.c(713) [sender=3.2.7]
user@wyng-wdcloud:~$ rsync --archive --compress --hard-links --human-readable --info=progress2 --exclude='*.tmp' /home/user/nv41/ root@192.168.3.1:/mnt/Backups/nv41/
197.29M 1% 6.20MB/s 0:00:30 (xfr#1788, ir-chk=1634/137446)^C
rsync error: unexplained error (code 255) at rsync.c(713) [sender=3.2.7]For low performance router (ACM3200 being openwrt first candidate) over wifi to softraid 5:
rsync --archive --compress --hard-links --human-readable --info=progress2 --exclude='*.tmp' /home/user/nv41/ root@192.168.3.1:/mnt/Backups/nv41/
Is the winner. Couterintuitively, compression helps, while whole-file and sparse doesn't.
root@Insurgo-Lab:~# iostat
Linux 5.15.137 (Insurgo-Lab) 05/10/24 _armv7l_ (2 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
1.54 0.02 2.09 2.71 0.00 93.64
Device tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd
md0 14.96 18.86 683.45 0.00 102129 3701100 0
mtdblock0 0.02 0.06 0.00 0.00 351 0 0
mtdblock1 0.01 0.04 0.00 0.00 199 0 0
mtdblock10 0.02 0.07 0.00 0.00 367 0 0
mtdblock2 0.01 0.04 0.00 0.00 207 0 0
mtdblock3 0.01 0.09 0.00 0.00 463 0 0
mtdblock4 0.02 0.06 0.00 0.00 351 0 0
mtdblock5 0.03 0.41 0.00 0.00 2215 0 0
mtdblock6 0.02 0.06 0.00 0.00 335 0 0
mtdblock7 0.03 0.41 0.00 0.00 2215 0 0
mtdblock8 0.02 0.06 0.00 0.00 335 0 0
mtdblock9 0.02 0.06 0.00 0.00 335 0 0
sda 40.32 42.46 242.13 0.00 229944 1311223 0
sdb 41.04 38.99 245.76 0.00 211157 1330839 0
sdc 40.65 39.19 244.12 0.00 212233 1321955 0
sdd 40.33 41.98 242.16 0.00 227353 1311383 0
ubiblock0_0 0.17 3.26 0.00 0.00 17633 0 0
Nope. In the longer run, compress is not helping
user@wyng-wdcloud:~$ rsync --archive --compress --hard-links --human-readable --info=progress2 --exclude='*.tmp' /home/user/nv41/ root@192.168.3.1:/mnt/Backups/nv41/
7.40G 18% 3.60MB/s 0:32:36 (xfr#67561, ir-chk=1971/389789)^C
rsync error: unexplained error (code 255) at rsync.c(713) [sender=3.2.7]
user@wyng-wdcloud:~$ rsync --archive --hard-links --human-readable --info=progress2 --exclude='*.tmp' /home/user/nv41/ root@192.168.3.1:/mnt/Backups/nv41/
11.79G 22% 5.70MB/s 0:32:53 (xfr#196247, ir-chk=2468/588821)
rsync error: unexplained error (code 255) at rsync.c(713) [sender=3.2.7]
So winning is rsync --archive --hard-links --human-readable --info=progress2 --exclude='*.tmp' /home/user/nv41/ root@192.168.3.1:/mnt/Backups/nv41/
tlaurion
commented
6 months ago some other raw notes on using openwrt router (basic now, still struggling a bit to have softraid5 survive usb reset as of now)
tlaurion
commented
6 months ago @tasket the socket filename is too big when using host name of tor v3 hostname. Could that be truncated to max size (100/108 chars from what I get?)
@tasket #191 also gets in the way. When changing exposed tor hidden service port from 2345 to 22 (mitigation) we see this blocking next.
EDIT: opened #194 to track this
tasket
commented
6 months ago template on which qube-ssh qube depends
* python3 * python3-pycryptodome python3-pycryptodome-ex (yes. Its the qube that needs this, not the archive machine)
@tlaurion None of the pycryptodome packages should be needed in the helper VM, unless you are running Wyng itself in the VM (thus its not a helper) and backing up volumes from there. (Note: this is possible with Wyng but not the util. FWIW, setting up Btrfs or LVM in an appvm isn't too difficult. IDK why anyone would use Wyng this way outside of testing purposes.)
dom0 ...
- python3-zlib
The correct library name is python3-zstd and its optional. zlib is already built into Python.
Further note on compression: If you want med-high compression ratios around 3.8:1 then bz2 can be competitive with zstd in both speed and size ratio (at least it was some years ago). This is bz2 level 8 vs zstd level 17. But they're comparable only in that narrow band.
tlaurion
commented
6 months ago My own researches (in compression of initrd content for firmware space economy) is that zstd is now par if not better then bz2 depending of comoressable content.
tlaurion
commented
6 months ago Testing after removing all openwrt unneeded deps (I have coreutils under openwrt which is pretty massive) and also python3 instead of python3-light lead to https://github.com/tasket/wyng-backup/issues/195
This is raw notes and will finalize later modifying OP
Documentation is not so clear on distinctions of dependencies of each part implied:
Notes:
With a setup like this, if qube has public key copied on openwrt server without password on key, qube can talk to openwrt through dropbear with what it seems no additional dependencies there, to be confirmed.