fcorbelli / zpaqfranz

Deduplicating archiver with encryption and paranoid-level tests. Swiss army knife for the serious backup and disaster recovery manager. Ransomware neutralizer. Win/Linux/Unix
MIT License
278 stars 25 forks source link

Zpaq 7.15 (2016) is 5.62% faster than Zpaqfranz v54.9-experimental (2021) (can u improve it?) #11

Closed pdr76 closed 2 years ago

pdr76 commented 2 years ago

zpaq64

zpaq v7.15 journaling archiver, compiled Aug 17 2016 Creating util-zpaq64.zpaq at offset 0 + 0 Adding 15.834624 MB in 1 files -method 56 -threads 4 at 2021-12-07 23:44:21. 100.00% 0:00:00 + Util.tar 15834624 100.00% 0:00:00 [1..235] 15835572 -method 56,18,0 1 +added, 0 -removed.

0.000000 + (15.834624 -> 15.834624 -> 14.094668) = 14.094668 MB 109.547 seconds (all OK)

zpaqfranz

zpaqfranz v54.9-experimental (HW BLAKE3), SFX64 v52.15, compiled Nov 4 2021 Integrity check type: XXHASH64+CRC-32 + CRC-32 by fragments Creating util-zpaqfranz.zpaq at offset 0 + 0 Adding 15.834.624 (15.10 MB) in 1 files at 2021-12-07 23:46:10

1 +added, 0 -removed.

0 + (15.834.624 -> 15.834.624 -> 14.094.701) = 14.094.701

115.703 seconds (000:01:55) (all OK)

fcorbelli commented 2 years ago

Well... in fact... no No, because zpaqfranz does more "background things" than 7.15 (calc CRC-32 and one hash, xxhash64 for default compatibility with 32bit-version) The key is here

Integrity check type: XXHASH64+CRC-32 + CRC-32 by fragments

More details on this topic (https://encode.su/threads/3658-How-big-can-the-hash-slowdown-in-an-archiver-be-tolerable)

This will detect (not fix) SHA-1 collisions, and made much faster the v (verify) command against the filesystem, that can be run in multithreaded mode (-all) In other words, in real life NVMe scenarios, verify data @2GB/s sustained

It is also possible to use safer-level hashes (BLAKE3, SHA-2 and even SHA-3) which give certainty in storage (SHA-2-256 is a legal standard in Europe)

Short version: zpaqfranz is developed to increase security in copies, check and verify, not for speed (a typical execution takes tens of minutes or even hours)

The difference in philosophy is

The matter is moot (as in the forum thread): what is your opinion?

Here some data

C:\zpaqfranz\release\54_9>zpaqfranz a z:\uno j:\Image2.nrg
zpaqfranz v54.9-experimental (HW BLAKE3), SFX64 v52.15, compiled Nov  4 2021
Integrity check type: XXHASH64+CRC-32 + CRC-32 by fragments
Creating z:/uno.zpaq at offset 0 + 0
Adding 4.291.166.334 (4.00 GB) in 1 files  at 2021-12-08 12:42:13
 94.65% 00:00:00 (   3.78 GB) -> (   3.76 GB) of (   4.00 GB)  227.84 MB/sec
1 +added, 0 -removed.

0 + (4.291.166.334 -> 4.289.765.038 -> 4.288.313.145) = 4.288.313.145

19.328 seconds (000:00:19)  (all OK)

You can disable zpaqfranz's extension, running just like 7.15

C:\zpaqfranz\release\54_9>zpaqfranz a z:\uno j:\Image2.nrg -715
zpaqfranz v54.9-experimental (HW BLAKE3), SFX64 v52.15, compiled Nov  4 2021
franz:mode -715 activated
**** Activated V7.15 mode ****
T forcezfs,donotforcexls,forcewindows; F crc32,checksum,filelist,xxhash,xxh3,fixeml,fix255,utf,flat
Integrity check type: NOTHING (LIKE 7.15)
Creating z:/uno.zpaq at offset 0 + 0
Adding 4.291.166.334 (4.00 GB) in 1 files  at 2021-12-08 12:43:41
 93.86% 00:00:00 (   3.75 GB) -> (   3.73 GB) of (   4.00 GB)  256.08 MB/sec
1 +added, 0 -removed.

0 + (4.291.166.334 -> 4.289.765.038 -> 4.288.313.109) = 4.288.313.109

16.937 seconds (000:00:16)  (all OK)

As you can see "zpaqfranz 7.15" is faster (just a bit) than the original 7.15

C:\zpaqfranz\release\54_9>c:\zpaqfranz\715 a z:\uno j:\Image2.nrg -summary 1
zpaq v7.15 journaling archiver, compiled Aug 21 2020
Creating z:/uno.zpaq at offset 0 + 0
Adding 4291.166334 MB in 1 files -method 14 -threads 32 at 2021-12-08 12:44:44.
1 +added, 0 -removed.

0.000000 + (4291.166334 -> 4289.765038 -> 4288.313109) = 4288.313109 MB
18.500 seconds (all OK)

Or you can retain some of zpaqfranz enhancements, but without -nochecksums

C:\zpaqfranz\release\54_9>zpaqfranz a z:\uno j:\Image2.nrg -nochecksum
zpaqfranz v54.9-experimental (HW BLAKE3), SFX64 v52.15, compiled Nov  4 2021
franz:NO checksum (-nochecksum)
Integrity check type: NOTHING (LIKE 7.15)
Creating z:/uno.zpaq at offset 0 + 0
Adding 4.291.166.334 (4.00 GB) in 1 files  at 2021-12-08 13:00:58
 94.25% 00:00:00 (   3.77 GB) -> (   3.75 GB) of (   4.00 GB)  257.14 MB/sec
1 +added, 0 -removed.

0 + (4.291.166.334 -> 4.289.765.038 -> 4.288.313.109) = 4.288.313.109

17.078 seconds (000:00:17)  (all OK)