Closed CoodyLing closed 6 months ago
If i modify test08.zpag content just 1 bit and run zpaqfranz v test??.zpaq to verify then show all ok
If run zpaqfranz t test??.zpaq to test then show error
Wow, a single question that raise 4 different answers 😄 First of all, thanks for you report. I'll explain all of your concers (no keyboard now) The differences of t test and v verify (2 different things) Why chuncked archives cannot be used like not chuncked Why they get the same extension (zpaq) And how to use backup (eventually) instead "When" use chunk, and "why"
Now the answers, broken down by topic Let's start with chunked archives
zpaqfranz a z:\full_?? *.cpp
zpaqfranz a z:\full_?? *.exe
This will create TWO zpaqs
17/01/2024 11:39 1.253.507 full_01.zpaq
17/01/2024 11:39 12.304.926 full_02.zpaq
Of different size. You can list even one by one
C:\zpaqfranz>zpaqfranz l z:\full_01.zpaq
zpaqfranz v59.1a-JIT-GUI-L,HW SHA1/2,SFX64 v55.1,(2024-01-16)
franz:-hw
z:/full_01.zpaq:
1 versions, 4 files, 1.253.507 bytes (1.20 MB)
- 2024-01-15 16:19:22 3.492.792 A 59_1.cpp
- 2024-01-15 18:03:00 3.494.187 A va01.cpp
- 2024-01-16 14:54:03 2.513.087 A zpaqfranz-linux.cpp
- 2024-01-16 14:53:45 3.494.167 A zpaqfranz.cpp
12.994.233 (12.39 MB) of 12.994.233 (12.39 MB) in 4 files shown
1.253.507 compressed Ratio 0.096 <<z:/full_01.zpaq>>
0.047 seconds (000:00:00) (all OK)
And the second
C:\zpaqfranz>zpaqfranz l z:\full_02.zpaq
zpaqfranz v59.1a-JIT-GUI-L,HW SHA1/2,SFX64 v55.1,(2024-01-16)
franz:-hw
z:/full_02.zpaq:
1 versions, 16 files, 12.304.926 bytes (11.73 MB)
- 2023-11-10 16:38:26 negative A 12.exe
- 2024-01-15 14:26:16 negative A 13.exe
- 2023-04-21 17:28:19 negative A 5.exe
- 2023-04-21 17:28:19 negative A 6.exe
- 2018-12-30 21:56:20 968.192 A CppCommentRemover.exe
- 2017-10-05 08:27:44 86.546 A dd.exe
- 2023-10-04 14:43:51 negative A exp01.exe
- 2023-12-16 14:38:40 negative A exp03-Wunused-parameter.exe
- 2023-12-16 15:01:40 negative A exp03.exe
- 2023-10-25 15:53:54 307.200 A unz.exe
- 2023-12-09 18:44:23 635.392 A zpaq715.exe
- 2016-08-17 23:07:33 984.576 A zpaqd.exe
- 2024-01-16 14:52:40 negative A zpaqfranz.exe
- 2024-01-16 14:57:46 negative A zpaqfranz32.exe
- 2024-01-16 14:58:55 negative A zpaqfranzhw.exe
- 2023-07-23 11:40:15 999.264 A zpaqlist.exe
3.981.160 (3.80 MB) of 39.895.113 (38.05 MB) in 16 files shown
12.304.926 compressed Ratio 0.308 <<z:/full_02.zpaq>>
0.062 seconds (000:00:00) (all OK)
Using the right wildcard you'll get back everything
C:\zpaqfranz>zpaqfranz l z:\full_??.zpaq
zpaqfranz v59.1a-JIT-GUI-L,HW SHA1/2,SFX64 v55.1,(2024-01-16)
franz:-hw
z:/full_??.zpaq:
2 versions, 20 files, 13.558.433 bytes (12.93 MB)
- 2023-11-10 16:38:26 3.263.488 A 12.exe
- 2024-01-15 14:26:16 3.331.584 A 13.exe
- 2023-04-21 17:28:19 3.148.288 A 5.exe
- 2024-01-15 16:19:22 3.492.792 A 59_1.cpp
- 2023-04-21 17:28:19 3.148.288 A 6.exe
- 2018-12-30 21:56:20 968.192 A CppCommentRemover.exe
- 2017-10-05 08:27:44 86.546 A dd.exe
- 2023-10-04 14:43:51 3.244.032 A exp01.exe
- 2023-12-16 14:38:40 5.859.195 A exp03-Wunused-parameter.exe
- 2023-12-16 15:01:40 3.014.144 A exp03.exe
- 2023-10-25 15:53:54 307.200 A unz.exe
- 2024-01-15 18:03:00 3.494.187 A va01.cpp
- 2023-12-09 18:44:23 635.392 A zpaq715.exe
- 2016-08-17 23:07:33 984.576 A zpaqd.exe
- 2024-01-16 14:54:03 2.513.087 A zpaqfranz-linux.cpp
- 2024-01-16 14:53:45 3.494.167 A zpaqfranz.cpp
- 2024-01-16 14:52:40 6.109.764 A zpaqfranz.exe
- 2024-01-16 14:57:46 3.074.048 A zpaqfranz32.exe
- 2024-01-16 14:58:55 3.120.128 A zpaqfranzhw.exe
- 2023-07-23 11:40:15 999.264 A zpaqlist.exe
54.288.362 (51.77 MB) of 54.288.362 (51.77 MB) in 20 files shown
13.558.433 compressed Ratio 0.250 <<z:/full_??.zpaq>>
0.078 seconds (000:00:00) (all OK)
This mode (the original) imposes no control over individual .zpaqs. You can replace test_27.zpaq with a new zpaq and the program will not notice it Similarly you can have "holes" (test_01...02...03...05...06...) with a missing 04
The problem increases more if the files (which make up the .zpaq file sequence) have different passwords
Now split @ 1MB the same example
C:\zpaqfranz>zpaqfranz a z:\piece_?? *.cpp -chunk 1m
zpaqfranz v59.1a-JIT-GUI-L,HW SHA1/2,SFX64 v55.1,(2024-01-16)
franz:-chunk 1.000.000
(...)INFO: The last chunk is: z:/piece_02.zpaq
C:\zpaqfranz>zpaqfranz a z:\piece_?? *.exe -chunk 1m
zpaqfranz v59.1a-JIT-GUI-L,HW SHA1/2,SFX64 v55.1,(2024-01-16)
franz:-chunk 1.000.000
(...)
INFO: The last chunk is: z:/piece_14.zpaq
Here's the files. Note that "piece_01" and 02 are the FIRST version, the other the 2nd
17/01/2024 11:47 1.048.576 piece_01.zpaq
17/01/2024 11:47 204.931 piece_02.zpaq
17/01/2024 11:47 1.048.576 piece_03.zpaq
17/01/2024 11:47 1.048.576 piece_04.zpaq
17/01/2024 11:47 1.048.576 piece_05.zpaq
17/01/2024 11:47 1.048.576 piece_06.zpaq
17/01/2024 11:47 1.048.576 piece_07.zpaq
17/01/2024 11:47 1.048.576 piece_08.zpaq
17/01/2024 11:47 1.048.576 piece_09.zpaq
17/01/2024 11:47 1.048.576 piece_10.zpaq
17/01/2024 11:47 1.048.576 piece_11.zpaq
17/01/2024 11:47 1.048.576 piece_12.zpaq
17/01/2024 11:47 1.048.576 piece_13.zpaq
17/01/2024 11:47 770.590 piece_14.zpaq
You cannot list, for example, the piece_12.zpaq as before: zpaqfranz cannot recognize the "piece" (as a zpaq archive), then "thinks" it is an encrypted one, then you must try a password
C:\zpaqfranz>zpaqfranz l z:\piece_12.zpaq
zpaqfranz v59.1a-JIT-GUI-L,HW SHA1/2,SFX64 v55.1,(2024-01-16)
franz:-hw
Archive seems encrypted (or corrupted)
Enter password :
This, of course, don't work
You must use the wildcard, as standard for zpaq, to list (or extract or whatever)
C:\zpaqfranz>zpaqfranz l z:\piece_??.zpaq
zpaqfranz v59.1a-JIT-GUI-L,HW SHA1/2,SFX64 v55.1,(2024-01-16)
franz:-hw
z:/piece_??.zpaq:
2 versions, 20 files, 13.558.433 bytes (12.93 MB)
- 2023-11-10 16:38:26 3.263.488 A 12.exe
- 2024-01-15 14:26:16 3.331.584 A 13.exe
- 2023-04-21 17:28:19 3.148.288 A 5.exe
- 2024-01-15 16:19:22 3.492.792 A 59_1.cpp
- 2023-04-21 17:28:19 3.148.288 A 6.exe
- 2018-12-30 21:56:20 968.192 A CppCommentRemover.exe
- 2017-10-05 08:27:44 86.546 A dd.exe
- 2023-10-04 14:43:51 3.244.032 A exp01.exe
- 2023-12-16 14:38:40 5.859.195 A exp03-Wunused-parameter.exe
- 2023-12-16 15:01:40 3.014.144 A exp03.exe
- 2023-10-25 15:53:54 307.200 A unz.exe
- 2024-01-15 18:03:00 3.494.187 A va01.cpp
- 2023-12-09 18:44:23 635.392 A zpaq715.exe
- 2016-08-17 23:07:33 984.576 A zpaqd.exe
- 2024-01-16 14:54:03 2.513.087 A zpaqfranz-linux.cpp
- 2024-01-16 14:53:45 3.494.167 A zpaqfranz.cpp
- 2024-01-16 14:52:40 6.109.764 A zpaqfranz.exe
- 2024-01-16 14:57:46 3.074.048 A zpaqfranz32.exe
- 2024-01-16 14:58:55 3.120.128 A zpaqfranzhw.exe
- 2023-07-23 11:40:15 999.264 A zpaqlist.exe
54.288.362 (51.77 MB) of 54.288.362 (51.77 MB) in 20 files shown
13.558.433 compressed Ratio 0.250 <<z:/piece_??.zpaq>>
0.078 seconds (000:00:00) (all OK)
Look at this example
C:\zpaqfranz>zpaq64 l z:\piece_??
zpaq v7.15 journaling archiver, compiled Aug 17 2016
z:/piece_??.zpaq: 2 versions, 20 files, 577 fragments, 13.558433 MB
- 2023-11-10 15:38:26 3263488 A 12.exe
- 2024-01-15 13:26:16 3331584 A 13.exe
- 2023-04-21 15:28:19 3148288 A 5.exe
- 2024-01-15 15:19:22 3492792 A 59_1.cpp
- 2023-04-21 15:28:19 3148288 A 6.exe
- 2018-12-30 20:56:20 968192 A CppCommentRemover.exe
- 2017-10-05 06:27:44 86546 A dd.exe
- 2023-10-04 12:43:51 3244032 A exp01.exe
- 2023-12-16 13:38:40 5859195 A exp03-Wunused-parameter.exe
- 2023-12-16 14:01:40 3014144 A exp03.exe
- 2023-10-25 13:53:54 307200 A unz.exe
- 2024-01-15 17:03:00 3494187 A va01.cpp
- 2023-12-09 17:44:23 635392 A zpaq715.exe
- 2016-08-17 21:07:33 984576 A zpaqd.exe
- 2024-01-16 13:54:03 2513087 A zpaqfranz-linux.cpp
- 2024-01-16 13:53:45 3494167 A zpaqfranz.cpp
- 2024-01-16 13:52:40 6109764 A zpaqfranz.exe
- 2024-01-16 13:57:46 3074048 A zpaqfranz32.exe
- 2024-01-16 13:58:55 3120128 A zpaqfranzhw.exe
- 2023-07-23 09:40:15 999264 A zpaqlist.exe
54.288362 MB of 54.288362 MB (20 files) shown
-> 42.161910 MB (813 refs to 577 of 577 frags) after dedupe
-> 13.558433 MB compressed.
0.031 seconds (all OK)
Changing the extension of the chunks (e.g., from .zpaq to .zcp) so that zpaqfranz would recognize them for what they are (multipart chunked and not multipart) would have prevented use with zpaq 7.15 (incidentally costing me a lot of time and effort)
It is possible to convert a multipart archive to a single .zpaq file in essentially two ways. The first is to use the m (merge) command
C:\zpaqfranz>zpaqfranz m z:\full_?? z:\full-one-piece.zpaq
zpaqfranz v59.1a-JIT-GUI-L,HW SHA1/2,SFX64 v55.1,(2024-01-16)
franz:-hw
*** Merge (consolidate) ***
Chunk 00000000 1.253.507 <<z:/full_01.zpaq>>
Chunk 00000001 12.304.926 <<z:/full_02.zpaq>>
--------------------------------------------------------------------------------------------------------------------
Total 2 13.558.433 (12.93 MB)
Outfile <<z:/full-one-piece.zpaq>>
Free space 85.772.365.824
Needed 13.558.433
(...)
Done
Written 13.558.433
Expected 13.558.433
0.062 seconds (000:00:00) (all OK)
Same thing for the chunked
C:\zpaqfranz>zpaqfranz m z:\piece_?? z:\piece-one-piece.zpaq
zpaqfranz v59.1a-JIT-GUI-L,HW SHA1/2,SFX64 v55.1,(2024-01-16)
franz:-hw
*** Merge (consolidate) ***
Chunk 00000000 1.048.576 <<z:/piece_01.zpaq>>
Chunk 00000001 204.931 <<z:/piece_02.zpaq>>
Chunk 00000002 1.048.576 <<z:/piece_03.zpaq>>
Chunk 00000003 1.048.576 <<z:/piece_04.zpaq>>
Chunk 00000004 1.048.576 <<z:/piece_05.zpaq>>
Chunk 00000005 1.048.576 <<z:/piece_06.zpaq>>
Chunk 00000006 1.048.576 <<z:/piece_07.zpaq>>
Chunk 00000007 1.048.576 <<z:/piece_08.zpaq>>
Chunk 00000008 1.048.576 <<z:/piece_09.zpaq>>
Chunk 00000009 1.048.576 <<z:/piece_10.zpaq>>
Chunk 00000010 1.048.576 <<z:/piece_11.zpaq>>
Chunk 00000011 1.048.576 <<z:/piece_12.zpaq>>
Chunk 00000012 1.048.576 <<z:/piece_13.zpaq>>
Chunk 00000013 770.590 <<z:/piece_14.zpaq>>
--------------------------------------------------------------------------------------------------------------------
Total 14 13.558.433 (12.93 MB)
Outfile <<z:/piece-one-piece.zpaq>>
Free space 85.758.803.968
Needed 13.558.433
(...)
Done
Written 13.558.433
Expected 13.558.433
0.110 seconds (000:00:00) (all OK)
Now
C:\zpaqfranz>dir z:\*one*.zpaq
Il volume nell'unità Z è RamDisk
Numero di serie del volume: 40FE-1005
Directory di z:\
17/01/2024 11:59 13.558.433 full-one-piece.zpaq
17/01/2024 12:00 13.558.433 piece-one-piece.zpaq
You can also do the merge with normal commands, such as copy /b, I won't go into it here
The v command has a different purpose than the one for which I think you want to use it
That is you do with the t command (or p or w)
Verify compares, possibly in parallel on SSD drives (with the -ssd switch) that the hashes of the files that zpaqfranz has processed match those of the files in the original folders (or wherever you want with the -to, -find, -replace switches)
The point is related to the possibility of changes DURING the procedure. I give an example that I hope will clarify it (!) Suppose we have a Windows file server with a number of users accessing files from their clients Suppose we want to back them up somewhot Let's run it and get a .zpaq file, perhaps with a 15-minute time frame What if a user had opened, say, the foo.doc file (while the backup was running), modifies it, and saves it while the backup was running?
The foo.doc file, currently in the file server folder, may be DIFFERENT from the one inside the .zpaq file. Not because of some kind malfunction, but precisely because the compression operation is not atomic, it takes time. You start with a 100 bytes-long foo.doc, but you store a 200 bytes-long foo.doc (just an example) zpaqfranz deals with this possibility in two ways: the first is with the zfs support (on non-Windows machines), where you can be assured that the files will NOT be modified during execution (taken from a snapshot) The second, on Windows, is to do a v (verification) at the end of archiving. On SSD drives this is very fast (even more than 1GB/s in the real world).
To do this you would have to extract the archive (somewhere) and then check the hashes there. That's exactly the t command with the -paranoid switch
First of all I suggest to... read the wiki, or the embedded manual
zpaqfranz h t
looking at the examples
Last version: t z:\1.zpaq
All versions: t z:\1.zpaq -all
Compare against filesystem: t z:\1.zpaq -verify
Against filesystem, 4 threads: t z:\1.zpaq -verify -ssd -t4
Real paranoid: extract all t z:\1.zpaq -to z:\knb -paranoid
Fast-SHA1 (nz the source dir): t z:\1.zpaq c:\nz
Cnk-SHA1+hash (nz the source dir): t z:\1.zpaq c:\nz -checksum
Multiple paranoid check (Win): t *.zpaq -to z:\temp\ -paranoid -longpath -big
Multiple test (*NIX): t "/copie/*.zpaq"
All version SHA-1 collisions: t z:\1.zpaq -collision -all
Lets pick some of them
To test that the .zpaq file IN THE LAST VERSION is not corrupted, you use this
C:\zpaqfranz\spaz>zpaqfranz t intermedie.zpaq
zpaqfranz v59.1a-JIT-GUI-L,HW SHA1/2,SFX64 v55.1,(2024-01-16)
franz:-hw
intermedie.zpaq:
7350 versions, 7.631 files, 431.966.159 bytes (411.95 MB)
To be checked 351.508.130 in 102 files (32 threads)
7.15 stage time 0.86 no error detected (RAM ~130.07 MB), try CRC-32 (if any)
Checking 4.592 blocks with CRC-32 (351.508.130 not-0 bytes)
CRC-32 time 0.12s
Blocks 351.508.130 ( 4.592)
Zeros 0 ( 0) 0.000000 s
Total 351.508.130 speed 2.492.965.460/sec (2.32 GB/s)
GOOD : 00000102 of 00000102 (stored=decompressed)
VERDICT : OK (CRC-32 stored vs decompressed)
1.000 seconds (000:00:01) (all OK)
Then you know that the very last version (7350) is good, with (about) 351MB
If you want to test EVERY part of the file, you do -all
C:\zpaqfranz\spaz>zpaqfranz t intermedie.zpaq -all
zpaqfranz v59.1a-JIT-GUI-L,HW SHA1/2,SFX64 v55.1,(2024-01-16)
franz:-all 4
franz:-hw
intermedie.zpaq:
7350 versions, 7.631 files, 431.966.159 bytes (411.95 MB)
To be checked 25.314.671.936 in 7.629 files (32 threads)
7.15 stage time 15.47 no error detected (RAM ~258.07 MB), try CRC-32 (if any)
Checking 303.287 blocks with CRC-32 (25.314.671.936 not-0 bytes)
Block 00275K 21.54 GB
CRC-32 time 8.89s
Blocks 25.314.671.936 ( 303.287)
Zeros 0 ( 0) 0.000000 s
Total 25.314.671.936 speed 2.846.904.176/sec (2.65 GB/s)
GOOD : 00007629 of 00007629 (stored=decompressed)
VERDICT : OK (CRC-32 stored vs decompressed)
24.359 seconds (000:00:24) (all OK)
In this example about 25GB of data is checked, two times
The first is fragment-SHA1 test (just like zpaq 7.15) The second stage is global (file level) CRC-32 (zpaqfranz's)
Now you know that the archive contains valid data, however, you DO NOT KNOW if it matches the original data, and you DO NOT know if it is restorable
Can you, concretely, recreate all the files that are present, and do these files contain exactly the data they should contain (i.e., are the hashes OK)?
If you are really paranoid, that is, you want to make sure that really everything is okay, you need to extract the contents of the archive, in some temporary folder, and then check each file. This will be a lengthy test (depending on the power of your computer of course) and is normally done on RAMDISK or solid-state disks
Now extract everything (from the intermedie.zpaq) because of -all, inside the temp folder z:\knb, and test everything -paranoid
C:\zpaqfranz\spaz>zpaqfranz t intermedie.zpaq -to z:\knb -all -paranoid
zpaqfranz v59.1a-JIT-GUI-L,HW SHA1/2,SFX64 v55.1,(2024-01-16)
franz:-all 4
franz:-hw -paranoid
intermedie.zpaq:
7350 versions, 7.631 files, 431.966.159 bytes (411.95 MB)
Extract 25.314.671.936 bytes (23.58 GB) in 14.981 files (0 folders) / 32 T
97.27% 00:00:00 ( 22.93 GB)=>( 23.58 GB) 670.94 MB/sec
FULL-extract hashing check (aka:paranoid)
Total bytes 25.314.671.936 (should be 25.314.671.936)
Bytes checked 25.314.671.936 (should be 25.314.671.936)
Files to be checked 7.629
Files == 7.629 (should be 7.629)
Files != 0 (should be zero)
Files deleted 7.629 (should be 7.629)
52.156 seconds (000:00:52) (all OK)
If, for some reason, the temp folder is NOT EMPTY (after the execution) THEN some file will not match, and this is very bad
If you don't want to extract the data to a physical drive (this slows down and also makes tear), but you have large amounts of free RAM, you can use the w command (call help with zpaqfranz h w for some examples) In this case we want to extract to RAM "fake -ramdisk" and -test with -checksum in multithread (-ssd) -all files with -verbose You cannot operate this way if you have a file larger, in the archive, than the available RAM (or rather than what zpaqfranz will use)
Suppose you have 16GB of free RAM, and every file in the archive is <10GB. Then you can use (zpaqfranz will use 16*0.75=12GB at most). If, on the other hand, you have a 20GB file in the archive, you will have to operate on a disk drive (i.e. t -paranoid on temporary folder) since there is not enough RAM. You can use the -frugal switch (of the w command) to minimize RAM usage, or leave it to zpaqfranz (which will take 75% of that reported free by the operating system). Obviously the more RAM, the faster The free RAM in my PC is ~32GB, we can continue (minimum ram needed is 5.411.726 bytes)
C:\zpaqfranz\spaz>zpaqfranz w intermedie.zpaq -ramdisk -test -checksum -ssd -all -verbose
zpaqfranz v59.1a-JIT-GUI-L,HW SHA1/2,SFX64 v55.1,(2024-01-16)
DETECTED SHA1/2 HW INSTRUCTIONS
franz:-all 4
franz:-checksum -hw -ramdisk -ssd -test -verbose
**** CHUNKED EXTRACTION/TEST ****
intermedie.zpaq:
7350 vers, 7.631 files, 9.715 frags, 21.805 blks, 431.966.159 bytes (411.95 MB)
Minimum needed (+10%) 5.411.726 0763/zpaqfranz.cpp
Free RAM (-25%) 32.891.433.984 (as reported by OS)
Chunks 0001 x 25.314.671.936 (total decompressed size 25.314.671.936)
====================================================================================================================
Chunk 001/001 25.314.671.936 bytes (23.58 GB) in 7.628 files by 32 threads
RAMDISK 25.314.671.936 bytes (23.58 GB) time 9.16 s @ 2.764.214.013 (2.57 GB/s)
Running 32 threads on 7.628 files
CRC-32 25.314.671.936 [OK] ( 36.60 s) @ 659.67 MB/s UNKN 0 [OK] GOOD 7.628 [OK] BAD 0 [OK]
HASHes 25.314.671.936 [OK] ( 22.12 s) @ 1.07 GB/s UNKN 0 [OK] GOOD 7.628 [OK] BAD 0 [OK]
--------------------------------------------------------------------------------------------------------------------
Bytes expected 25.314.671.936 ( 1.88 s) @ 0 B/s
RAMDISK releasing 25.314.671.936 bytes to 0 (should be zero)
Stage VEF 0001 : errors 0 (0=good)
====================================================================================================================
12.312 seconds (000:00:12) (all OK)
Then there are numerous other verification mechanisms, to be adopted if the original files are available, or using other algorithms etc. You can find a good part of the explanation in the wiki, or you can ask me directly
Last, but not least, to merge a multipart archive the 2nd way, the repack In this example x means... extract "" means everything (you want only cpp? do .cpp or whatever) -to pippo is a dummy (necessary to do not overwrite locally!) -space is do-not-care-for-pippo
zpaqfranz x z:\piece_?? * -repack z:\all-in-one.zpaq -to pippo -space
Actually, this repack requires an understanding and familiarity with zpaq archives, let's say at the "poweruser" level, so I don't suggest it for the novice
After run the command : zpaqfranz a test??.zpaq -chunk 1m I got 13 files : test01.zpaq ~ test13.zpaq if run the command : zpaqfranz v test01.zpaq I got error : zpaqfranz error: archive not found if run the command : zpaqfranz v test02.zpaq I got ask me to input password ??