tasket
commented
3 years ago There is already an issue for generic CPU optimization. But you should know that receive has gotten very little of it so far. Probably after v0.5 we'll see some receive CPU optimization including parallel processing.
(BTW, I was able to make the wyng-extract.sh script somewhat parallel for compression because the data caching in /tmp presented the opportunity to do this easily. But Wyng itself doesn't cache data this way.)
The doc for --sparse-write doesn't mention that it introduces an extra step: comparison with the local volume data. Despite that, all chunks are retrieved from remote. Therefore, this option should be used specifically to optimize local disk space.
Re-Compression dominates receive --sparse out of necessity & bzip2 is CPU intensive. In the future we could use more efficient compression like zstd and lbzip2, as well as find ways to do more work in parallel. But for now this option is strictly "use CPU bandwidth to avoid costs on slow/expensive network".
And the respective efficiency of both options depends on just how much the archive copy differs with the local volume being over-written. (More difference = less efficiency.)
This type of issue is why backup tools like Wyng try to migrate to efficient compression libraries as soon as they can, bc uncompressed data chunks cannot be safely compared for --sparse use case. v0.3 now has Zstandard, which is fast but requires fairly recent OS version (Qubes 4.0 has lousy support for zstd). The picture for Wyng is quite complex bc libraries for new tools/formats come to Python slowly, and the decisions of repo managers in this respect is often shockingly bad; there is also a format stabilization issue w zstd that can defeat a sparse processing feature (IIRC this has also negatively impacted other backup tools) but that is slowly improving.
Hashtype is already fastest type, sha256. Chunk size IIRC was chosen to reduce the amount of metadata being sent over a slow network and reduce metadata that had to be verified in Heads env. You might try an archive configured with smaller chunk sizes (the default is 65536) to see how that impacts send/receive ops.
tlaurion
We see a small difference in local processing time, where bandwidth used is mostly the same.
Sorry if I am a bit rigid on documentation. I have a hard time wrapping my head on current, where QubesOS costs on AppVM's LVM disk for storage for backup storage has known high overhead, so I thought of using
--sparse-writeto spread the cpu pinning costs over 2 CPUs, giving edge over--sparse, but don't see direct benefit.Let me explain:
Doc today says:
Where detailed doc says:
My understanding is that present code is not paralleling any work (in neither modes, think its another tticket), so one core performance would be the limitation of combined dom0 + qube virtualized IO backup storage in my use case (which happens over wyng-backups-vm storage).
I would still have expected
--sparse-write(50% storage qubes, 50% dom0) to fasten the receive operation over--sparse(100% CPU hit for local calculation and less pulling over qubes's stored backups data) where the results seem to be as equal.Maybe you could clarify or give a bit more of insights? Otherwise I will put timestamps in my scripts.
Pertinent notes on current archive.ini conf:
Where bz2 was chosen for Head's current busybox support and where I lost track of the chunksize and hashtype costs, so you may shed some lights if you will! :)
Also note that https://git.busybox.net/buildroot/commit/?id=6bccac75ea3f8cd66bcde3747067add14b0c4f2c relies on python script... so not gonna happen soon under Heads.