PythonNut / virtualbox-remote-snapshots

:clock4: A snapshot system for VirtualBox which supports remote storage

3 stars 0 forks source link

Be "fast" #22

Open PythonNut opened 7 years ago

PythonNut commented 7 years ago

This is a meta-issue that tracks making differential extraction faster. Currently, they're pretty fast. But they can go faster.

[ ] Look into alternatives/enhancements to rsync, namely ones that support multithreading.
[ ] Look into alternatives for compression, namely lz4, since our "slow" WiFi is only slow relative to Gigabit Ethernet.
[x] Look into automatically tuning the compression we use for rsync based on profiling information.
[ ] Wait patiently for borg to support differential extraction.

PythonNut commented 7 years ago

My conclusion from testing rsync compression is that we may already get a nice speedup from using zlib level 3.

PythonNut commented 7 years ago

Compression Benchmark

Virtual Machine: Arch Linux Aurora [41.85 GB Split2G VMDK w/ 1 snapshot]

Full backup times (laptop)

i5-6200U with --chunker-params 13,19,16,4095

Algorithm	Time	Archive Size	Compression Ratio	Network Speed	Mean CPU Load
`lz4`	`28:13.88`	`17.79 GB`	`2.35`	`84.0 Mbps`	`28%`
`zlib,3`	`33:49.80`	`15.56 GB`	`2.69`	`61.3 Mbps`	`35%`
`lzma,1`	`2:00:40.27`	`14.31 GB`	`2.92`	`15.8 Mbps`	`35%`

PythonNut commented 7 years ago

Might want to look into tuning the mount options for borg mount and see if we can't eek out a little more performance. Currently, we seem to be CPU bound.

EDIT: Can't seem to pass fuse options to borg mount. :/