search

fcorbelli / zpaqfranz

Deduplicating archiver with encryption and paranoid-level tests. Swiss army knife for the serious backup and disaster recovery manager. Ransomware neutralizer. Win/Linux/Unix
MIT License
278 stars 25 forks source link

Windows: error shown with symlinked files #15

Closed houbit closed 2 years ago

houbit commented 2 years ago

At first I have to thank you for [maintaining] this excellent tool!

Here a little issue using zpaqfranz v54.12, packing symlinked files under NTFS I get

"38385: WARN expected 0 getted for ".

Further on this results in "38721: HOUSTON something seems wrong: expected , done + " "..." "... almost certainly incomplete" (that is definetly wrong, the archive is fine)

Kind regards

fcorbelli commented 2 years ago

Thank you for your recommendation. Could you please do an actual example, even just for a single symlinked file?

"38385: WARN expected 0 getted for ".

should be a little different

38385: WARN expected 0 getted 34 for thepippo.txt

It is indeed important to understand in which portion of the program (I think I understand it, but not for sure) the difference is generated In this example I made a symlink to a folder...

Z:\>dir
 Il volume nell'unità Z è RamDisk
 Numero di serie del volume: 8A05-AD69

 Directory di Z:\

27/04/2022  14:35    <DIR>          nza_copy
               0 File              0 byte
               1 Directory  84.879.110.144 byte disponibili

Z:\>mklink collegamento2 z:\nza_copy /d
collegamento simbolico creato per collegamento2 <<===>> z:\nza_copy

Z:\>dir
 Il volume nell'unità Z è RamDisk
 Numero di serie del volume: 8A05-AD69

 Directory di Z:\

27/04/2022  14:45    <SYMLINKD>     collegamento2 [z:\nza_copy]
27/04/2022  14:35    <DIR>          nza_copy
               0 File              0 byte
               2 Directory  84.879.110.144 byte disponibili

and it works OK

Z:\>zpaqfranz a 1 z:\collegamento2\*
zpaqfranz v54.12-experimental (HW BLAKE3), SFX64 v52.15, compiled Mar 20 2022
Integrity check type: XXHASH64+CRC-32 + CRC-32 by fragments
Creating 1.zpaq at offset 0 + 0
Adding 909.162.975 (867.04 MB) in 1.558 files  at 2022-04-27 14:46:55
Non-latin (UTF-8)             2
 82.09% 00:00:00 ( 711.73 MB) -> ( 342.98 MB) of ( 867.04 MB)  177.93 MB/sec
1.786 +added, 0 -removed.

0 + (909.162.975 -> 826.472.213 -> 437.368.167) = 437.368.167
Forced XLS/PPT has included 17.408 bytes in 1 files

6.078 seconds (000:00:06)  (all OK)

Maybe you use FILE symlinks instead of folder? Can you please post a run with the -debug switch? Maybe you get this ARCHIVE;REPARSE_POINT;

Can I prepare, in case, a specific debug version, so that you can test it, by putting it here (Windows 64 bit)?

houbit commented 2 years ago

Sorry my text was incomplete, used angle brackets without thinkin:

NTFS with symbolic link on a FILE:

"38385: WARN expected 0 getted for N".

Further on this results in "38721: HOUSTON something seems wrong: expected M, done M+N" "..." "... almost certainly incomplete" (that is definetly wrong, the archive is fine)`

Seems like the symlinked file was read (N bytes) but counted as 0 bytes.

Full example, zpaqfranz unpacked, symlink notes.txt.symlnk -> notes.txt: 'dir' => Verzeichnis von A:_zbackup\zpaq_test 27.04.2022 14:51

.
27.04.2022 14:51 ..
20.03.2022 18:41 293.202 blake3_windows_gnu.S
20.03.2022 18:41 5.677 CHANGELOG.md
20.03.2022 18:41 15.391 differences715.txt
20.03.2022 18:41 470 Makefile
20.03.2022 18:41 43.609 notes.txt
27.04.2022 14:51 notes.txt.symlnk [A:_zbackup\zpaq_test\notes.txt]
20.03.2022 18:41 27.754 README.md
27.04.2022 14:51 1.065.373 x.zpaq
20.03.2022 18:41 209.486 zpaq206.pdf
20.03.2022 18:41 1.841.725 zpaqfranz.cpp
10 Datei(en), 3.502.687 Bytes
2 Verzeichnis(se), xxxxxxxx Bytes frei

"zpaq a test " (sorry, renamed it to zpaq for my own convinience only) => A:_zbackup\zpaq_test>zpaq a test
zpaqfranz v54.12-experimental (HW BLAKE3), SFX64 v52.15, compiled Mar 20 2022
Integrity check type: XXHASH64+CRC-32 + CRC-32 by fragments
Creating test.zpaq at offset 0 + 0
Adding 2.437.314 (2.32 MB) in 9 files at 2022-04-27 15:07:40
38385: WARN expected 0 getted 43.609 for notes.txt.symlnk 9 +added, 0 -removed. 0 + (2.437.314 -> 2.437.314 -> 1.065.373) = 1.065.373 38271: HOUSTON something seems wrong: expected 2.437.314, done 2.480.923
38271: Corrupted source files? Lost connection? Cannot access? Media full?
38271: =>The updated .zpaq archive is almost certainly incompleted 0.234 seconds (00:00:00) (with errors)

fcorbelli commented 2 years ago

Sorry my text was incomplete, used angle brackets without thinkin:

NTFS with symbolic link on a FILE:

OK, this make sense (I never use symlink to file, only to folders :-)

"38385: WARN expected 0 getted for N".

Further on this results in "38721: HOUSTON something seems wrong: expected M, done M+N" "..." "... almost certainly incomplete" (that is definetly wrong, the archive is fine)`

Seems like the symlinked file was read (N bytes) but counted as 0 bytes.

Exactly. The file size is taken the first time from the filesystem (and in this function, scandir (), will need to be corrected) and a second time after the block reading (to calculate the hash). In a normal situation the two data coincide. The foo.txt file was 1000 bytes BEFORE and will be 1000 AFTER

However, this does not always happen, as "someone" may change a file while the compression is in progress (can takes time, ever hours to complete) So the file pippo.txt FIRST could be 1000 bytes large (... someone changes it ...) and THEN become let's say 1001. In this case the warning is shown

As mentioned, I would need, please, a run with -debug to understand if my interpretation is correct. Thanks

houbit commented 2 years ago

Same constallations, "zpaq a test * -debug | tee test.log"

test.log: ` zpaqfranz v54.12-experimental (HW BLAKE3), SFX64 v52.15, compiled Mar 20 2022 franz:DEBUG very verbose (-debug) 21026: check_if_password of test.zpaq Integrity check type: XXHASH64+CRC-32 + CRC-32 by fragments Creating test.zpaq at offset 0 + 0 00000010 MY new t . DIRECTORY;

00000010 MY new t .. DIRECTORY;

00000020 MY new t blake3_windows_gnu.S ARCHIVE;

00000020 MY new t CHANGELOG.md ARCHIVE;

00000020 MY new t differences715.txt ARCHIVE;

00000020 MY new t Makefile ARCHIVE;

00000020 MY new t notes.txt ARCHIVE;

00000420 MY new t notes.txt.symlnk ARCHIVE;REPARSE_POINT;

00000020 MY new t README.md ARCHIVE;

00000020 MY new t zpaq206.pdf ARCHIVE;

00000020 MY new t zpaqfranz.cpp ARCHIVE;

Default sort Adding 2.437.314 (2.32 MB) in 9 files 20006: converting to localtime 2022-04-27 13:31:56 20028: localtime is 2022-04-27 15:31:56 at 2022-04-27 15:31:56 Model2: XXHASH64 2CB19869E0642796 CHANGELOG.md Mode2: XXHASH64: <<2CB19869E0642796>> CRC32 <> CHANGELOG.md Model2: XXHASH64 5046DFD533DC8DA5 Makefile Mode2: XXHASH64: <<5046DFD533DC8DA5>> CRC32 <> Makefile Model2: XXHASH64 51D9BB16C4688B8C README.md Mode2: XXHASH64: <<51D9BB16C4688B8C>> CRC32 <> README.md Model2: XXHASH64 AB2D326F6ADC598D blake3_windows_gnu.S Mode2: XXHASH64: <> CRC32 <<15956CE2>> blake3_windows_gnu.S Model2: XXHASH64 D27913528DDE6FA4 differences715.txt Mode2: XXHASH64: <> CRC32 <<1030B35B>> differences715.txt Model2: XXHASH64 C498B2481EA2C766 notes.txt Mode2: XXHASH64: <> CRC32 <<010AA2D6>> notes.txt Model2: XXHASH64 C498B2481EA2C766 notes.txt.symlnk 38385: WARN expected 0 getted 43.609 for notes.txt.symlnk Mode2: XXHASH64: <> CRC32 <<010AA2D6>> notes.txt.symlnk Model2: XXHASH64 EA4527C120B47032 zpaq206.pdf Mode2: XXHASH64: <> CRC32 <<26121D8D>> zpaq206.pdf Model2: XXHASH64 8DCD3E429C38D817 zpaqfranz.cpp Mode2: XXHASH64: <<8DCD3E429C38D817>> CRC32 <<4ACAE94D>> zpaqfranz.cpp

9 +added, 0 -removed.

38271: HOUSTON something seems wrong: expected 2.437.314, done 2.480.923 38271: Corrupted source files? Lost connection? Cannot access? Media full? 38271: =>The updated .zpaq archive is almost certainly incompleted 33026: call xcommand on errorcode==2 Entering xcommand() i_command || i_parameter |38271: HOUSTON something seems wrong expected vs done| Exit because empty i_command ` If you need something from a 'special/prepared' version of zpaqfranz let me know.

Thank you

fcorbelli commented 2 years ago

54_13j.zip Can you please check the attached pre-release, with and without -debug?

Thanks

houbit commented 2 years ago

I am VERY sorry about the delay, was in really bad health troubles.

..\54_13j a test | tee ..\54_13j.log ..\54_13j a test -debug | tee ..\54_13j-debug.log

54_13j.log: `zpaqfranz v54.13j-experimental (HW BLAKE3), SFX64 v52.15, compiled Apr 28 2022 Integrity check type: XXHASH64+CRC-32 + CRC-32 by fragments Creating test.zpaq at offset 0 + 0 Adding 2.437.314 (2.32 MB) in 9 files at 2022-06-13 11:47:12 38385: WARN expected 0 getted 43.609 for notes.txt.symlnk

9 +added, 0 -removed.

38271: HOUSTON something seems wrong: expected 2.437.314, done 2.480.923 38271: Corrupted source files? Lost connection? Cannot access? Media full? 38271: =>The updated .zpaq archive is almost certainly incompleted 54_13j-debug.log zpaqfranz v54.13j-experimental (HW BLAKE3), SFX64 v52.15, compiled Apr 28 2022 franz:DEBUG very verbose (-debug) 21026: check_if_password of test.zpaq Integrity check type: XXHASH64+CRC-32 + CRC-32 by fragments Creating test.zpaq at offset 0 + 0 00000010 MY new t . DIRECTORY;

00000010 MY new t .. DIRECTORY;

00000020 MY new t blake3_windows_gnu.S ARCHIVE;

00000020 MY new t CHANGELOG.md ARCHIVE;

00000020 MY new t differences715.txt ARCHIVE;

00000020 MY new t Makefile ARCHIVE;

00000020 MY new t notes.txt ARCHIVE;

00000420 MY new t notes.txt.symlnk ARCHIVE;REPARSE_POINT;

29300: maybe Windows-symlinked file, fake size 0 real size 0 * 00000020 MY new t README.md ARCHIVE;

00000020 MY new t zpaq206.pdf ARCHIVE;

00000020 MY new t zpaqfranz.cpp ARCHIVE;

Default sort Adding 2.437.314 (2.32 MB) in 9 files 20006: converting to localtime 2022-06-13 09:50:50 20028: localtime is 2022-06-13 11:50:50 at 2022-06-13 11:50:50 Model2: XXHASH64 2CB19869E0642796 CHANGELOG.md Mode2: XXHASH64: <<2CB19869E0642796>> CRC32 <> CHANGELOG.md Model2: XXHASH64 5046DFD533DC8DA5 Makefile Mode2: XXHASH64: <<5046DFD533DC8DA5>> CRC32 <> Makefile Model2: XXHASH64 51D9BB16C4688B8C README.md Mode2: XXHASH64: <<51D9BB16C4688B8C>> CRC32 <> README.md Model2: XXHASH64 AB2D326F6ADC598D blake3_windows_gnu.S Mode2: XXHASH64: <> CRC32 <<15956CE2>> blake3_windows_gnu.S Model2: XXHASH64 D27913528DDE6FA4 differences715.txt Mode2: XXHASH64: <> CRC32 <<1030B35B>> differences715.txt Model2: XXHASH64 C498B2481EA2C766 notes.txt Mode2: XXHASH64: <> CRC32 <<010AA2D6>> notes.txt Model2: XXHASH64 C498B2481EA2C766 notes.txt.symlnk 38385: WARN expected 0 getted 43.609 for notes.txt.symlnk Mode2: XXHASH64: <> CRC32 <<010AA2D6>> notes.txt.symlnk Model2: XXHASH64 EA4527C120B47032 zpaq206.pdf Mode2: XXHASH64: <> CRC32 <<26121D8D>> zpaq206.pdf Model2: XXHASH64 8DCD3E429C38D817 zpaqfranz.cpp Mode2: XXHASH64: <<8DCD3E429C38D817>> CRC32 <<4ACAE94D>> zpaqfranz.cpp

9 +added, 0 -removed.

38271: HOUSTON something seems wrong: expected 2.437.314, done 2.480.923 38271: Corrupted source files? Lost connection? Cannot access? Media full? 38271: =>The updated .zpaq archive is almost certainly incompleted 33026: call xcommand on errorcode==2 Entering xcommand() i_command || i_parameter |38271: HOUSTON something seems wrong expected vs done| Exit because empty i_command `

fcorbelli commented 2 years ago

I hope you health is OK

Model2: XXHASH64 C498B2481EA2C766 notes.txt.symlnk
38385: WARN expected 0 getted 43.609 for notes.txt.symlnk
(...)
38271: HOUSTON something seems wrong: expected 2.437.314, done 2.480.923

How do you have created notes.txt.symlnk?

Thank you PS the latest pre-release is this one

54_16e.zip

houbit commented 2 years ago

Thank You, I am fine now.

Just made a batch to create and run a test case:

! The symlinks are created by Windows own mklink.com. ! Sadly you need administrator privileges to use this and also for the batch.

here the results: zpaqfranz_test.zip

Anycase I hope this helps. Kind regards

fcorbelli commented 2 years ago

Please test the current 55.4 release
Thank you