Open mirogeorg opened 1 week ago
a) ZPAQFRANZ A, when detecting an "Incomplete transaction ignored," should attempt to "repair" by deleting back to a healthy archive. You can. With the trim command There is not a "trim" command for zpaq
b) ZPAQFRANZ T and L, when detecting an "Incomplete transaction ignored," should set a Warning status and the corresponding ERRORLEVEL.
This can make sense
ZPAQFRANZ recognizes the incomplete transaction but does not remove it. It detects that something is wrong, continues without successfully handling the situation, and as a result, all subsequent operations effectively cannot detect or recover from the issue. You cannot recover from a missing part It is just a "hole" zpaq does detect this situation, and work in the same way "opppssss..."
Suppose you have
piece0001 piece0002 piece0003 piece0004 piece0005 piece0006 piece0008 piece0009 piece0010 piece0011 You can extract upto piece0006. Everything after can, or cannot be, extractable (you can try extracting directly, if not encrypted) You must be very lucky (no spiegone here)
What you essentially need to do for a NON-encrypted multipart archive is replace the file with the incomplete transaction with a zero-length file. At that point, "magically," everything will start working again.
touch piece0007.zpaq
Alright, it can be fixed with ZPAQFRANZ trim, but how can someone know this needs to be done if all the tests pass without errors?
The alternative is the same as with the old ZPAQ: capture the output and search for strings. ;(
This is precisely the issue — ZPAQFRANZ continues to append parts, but they effectively remain inaccessible. The chain is broken. In my opinion, since ZPAQFRANZ is a backup tool, this behavior is unacceptable. I know this is inherited from ZPAQ, but it's a pretty bad legacy and shouldn’t continue like this.
Ideally, ZPAQFRANZ should detect that. In my view, there are two proper solutions: a) It should determine that the chain is broken and stop further additions to the archive, simply returning an error. b) It should allow further additions but return ERRORLEVEL <> 0. The rationale for this option is if the first one is too difficult to implement. It would serve as a quick workaround.
Work in progress 1 bigger advice, showing part hole 2 fix for unencrypted multipart archives (instead of manually renaming) 3 freezer for multipart incomplete
The longer part will be a incomplete enumerator
Doable
The current state is the new fix command, which, on NON-encrypted multipart archives with incomplete transactions, detects and resolves them (at least in theory), allowing the content to be extracted and a new "clean" archive to be created. Essentially, it's a patch job. It also displays a huge warning in case of incomplete transactions, making it easier to understand that the archive is corrupted.
haha. That is cool. I'll test when you release version for test
Franco, from the screenshots you’ve taken, I see that the FIX command is attempting to salvage as much as possible from the archives, showing a warning that the data may not be okay.
Perhaps such a command would be better named ZPAQFRANZ salvage.
Wouldn't it be better if ZPAQFRANZ fix were a fast command that quickly repairs the archive and trims it down to what's intact? It would also be good if it set the ERRORLEVEL as follows:
0 when the archive is intact and hasn't been modified, 1 when the repair was successful, and
Obviously, it already exists 😄 For monolithic files, that is, single .zpaq files, there is the trim command (specific to zpaqfranz). It exists for single files, implemented but disabled by default. Essentially, if you press Control-C while adding data, it can automatically trim. Since I’m cautious, I didn’t set it as default. But the point is different. While I can roughly do whatever I want for a Control-C, I can’t do the same for a process kill. In this case, I COULD detect the transaction during the next read_archive and remove it mercilessly. Or mercilessly delete the last part of the multipart file. This would make the archive consistent. BUT it involves deleting pieces of data, which I generally dislike doing. Keep in mind that there can be multiple gaps, i.e., incomplete transactions across more than one file. In short, a very aggressive approach might result in data loss.
I will probably (I need to think about it) adopt a mixed procedure. In the case of multipart archives, if it detects a gap in the last file, it simply renames it to .spaz. This way, by continuing to add data, the archive remains intact. If, however, the gap is NOT in the last piece, it displays an error message and advises you to proceed with a fix (which will actually be called trim later. Essentially, there will be a single trim command that will work differently for single and multipart files).
Then there's the issue of chunk archives, for which this simply can’t be done. And encrypted ones, which can’t be managed either. And those with an index, which makes everything much more complicated. Nothing is easy with zpaq. Therefore, it won’t be a universal command but specific to simple multipart files.
As you can see, these are big problems, very big problems to solve. In fact, Mahoney didn’t even tackle them.
Finally, I think I’ll add a detailed explanation on a wiki page, and zpaqfranz will provide the link. If you want to know what to try in your case, click here.
My recommendation is in two directions:
It should be implemented in a way that is user-friendly and clear for an inexperienced user who is not even conceptually familiar with this issue. In my opinion, A / BACKUP should independently detect and resolve the problem if one is identified, without asking any questions. If it’s not too difficult to implement in the code, I don’t see an issue, especially since the original ZPAQ automatically trims corrupted transactions without options and without asking (we’re talking about single-part archives). Therefore, the current multipart implementation should (finally) align the behavior with that of single-part archives.
It should be convenient for scripting. When scripting, it’s particularly difficult and unreliable to check statuses, various scenarios, and whether a TRIM is needed after each operation. This significantly decreases reliability.
In my opinion, reliability (and simplicity) should take priority above all else.
What do you think?
My recommendation is in two directions:
- It should be implemented in a way that is user-friendly and clear for an inexperienced user who is not even conceptually familiar with this issue. In my opinion, A / BACKUP should independently detect and resolve the problem if one is identified, without asking any questions.
It is risky
If it’s not too difficult to implement in the code, I don’t see an issue, especially since the original ZPAQ automatically trims corrupted transactions without options and without asking (we’re talking about single-part archives).
No, zpaq does not trim corrupted transactions, at all
Therefore, the current multipart implementation should (finally) align the behavior with that of single-part archives.
It is impossible, because a missing part can be anywhere Not just in the very last chunk
- It should be convenient for scripting. When scripting, it’s particularly difficult and unreliable to check statuses, various scenarios, and whether a TRIM is needed after each operation. This significantly decreases reliability.
In my opinion, reliability (and simplicity) should take priority above all else.
What do you think?
That I do not design zpaq's file format in such a bizzarre way
Therefore a lot of things are simply impossible, without breaking backward compatibility
This is zpaq 7.15
// Truncate empty update from archive (if not indexed)
if (!index) {
if (added+removed==0 && archive_end-header_pos==104) // no update
archive_end=header_pos;
if (archive_end<archive_size) {
if (archive_end>0) {
printf("truncating archive from %1.0f to %1.0f\n",
double(archive_size), double(archive_end));
if (truncate(arcname.c_str(), archive_end)) printerr(archive.c_str());
}
else if (archive_end==0) {
if (delete_file(arcname.c_str())) {
printf("deleted ");
printUTF8(arcname.c_str());
printf("\n");
}
}
}
}
This is zpaqfranz The same
// Truncate empty update from archive (if not indexed)
if (!index)
{
if (g_chunk_size>0)
{
if (flagverbose)
{
printbar('-');
for (unsigned int i=0;i<out.filepartnames.size();i++)
myprintf("02136: Chunk %08d %21s %s\n",i+1,migliaia(prendidimensionefile(out.filepartnames[i].c_str())),out.filepartnames[i].c_str());
printbar('=');
}
string thelast=out.lastfilename();
myprintf("02137: INFO: The last chunk is: %Z\n",thelast.c_str());
if (prendidimensionefile(thelast.c_str())==0)
if (delete_file(thelast.c_str()))
{
myprintf("02138: deleted %Z : no data to be keeped\n",thelast.c_str());
}
if (flagdebug3)
myprintf("02139: thecdatasize %21s\n",migliaia(thecdatasize));
}
else //default, not chunked
{
if (flagdebug3)
myprintf("02140: arcname %s\n",arcname.c_str());
if (added+removed==0 && archive_end-header_pos==104) // no update
archive_end=header_pos;
if ((archive_end<archive_size) && (g_chunk_size==0))
{
if (archive_end>0)
{
if (flagverbose)
myprintf("02141: truncating archive from %s to %s\n",migliaia(archive_size), migliaia2(archive_end));
if (truncate(arcname.c_str(), archive_end))
printerr("17092",archive.c_str(),0);
if (fasttxt!="")
{
myprintf("02142: Turning off fasttxt due to truncation [archive not changed]\n");
flagfasttxt=false; // we do not want to update CRC-32!
fasttxt="";
}
if (g_starting_zpaqdate>0)
{
if (flagverbose)
myprintf("02143: touching back to %s\n",dateToString(false,g_starting_zpaqdate).c_str());
if (!touch(arcname.c_str(),g_starting_zpaqdate,g_starting_zpaqattr))
myprintf("02144$ WARNING trouble in touching\n");
}
}
else
if (archive_end==0)
{
if (delete_file(arcname.c_str()))
if (flagverbose)
{
myprintf("02145: deleted %Z: no data to be archived\n",arcname.c_str());
}
}
}
}
}
Short version: I’m working on a trimmer for multipart archives that, if the gap is in the very last piece, renames it, simulating truncation. If the gap isn’t in the last part, it’s your problem.
You’re saying it’s risky if you have old, potentially corrupted archives… But they’re still old AND corrupted. Isn’t it even riskier for new ones to keep getting corrupted… Especially in a script, where there are no eyes to see the big warning sign? ;)
Isn’t it better for the future to be brighter and more promising than the past? ;)
OK. thank you for the effort!
Sadly, it is impossible, due to the -chunk new option Did I already mention that I didn’t design the data storage format? 😄
60_10b.zip You can try this (highly instable) pre-release
This should:
The trim
command now (perhaps) also works for multipart archives (non-chunked and non-encrypted). When launched normally, like zpaqfranz trim z:\uno_????
, it shows a report. With the -kill
switch, it attempts, as much as possible, to repair the archive. If the erroneous piece is the last one, the archive is valid. If not, you'll need to extract the data and start over, as the internal structure is compromised.
60_10c.zip This pre-release add the -trim switch (in add and backup) that (should) trim incomplete transaction, if possible, deleting the last chunk
The current ongoing "superspiegone"
https://github.com/fcorbelli/zpaqfranz/wiki/Real-world:-corruption
I ran some tests with 60.10c and am providing feedback so you can decide if this is the result you were expecting.
To remind you - the test archive is corrupted in part 5 or 6 out of a total of 40+ parts.
Results: ZPAQFRANZHW trim - kill: It zeros out only the corrupted part, but the archive remains problematic. Displays "(with warnings)" and returns errorlevel = 1. From your plan, I understand this behavior will remain in the final version: In other cases (e.g., incomplete transaction not at the end), suggest performing a trim and terminate.
When running ZPAQFRANZHW trim - kill again: Displays "(all OK)", returns errorlevel = 0, but the archive remains problematic.
ZPAQFRANZHW t (test) on an unrepaired archive: Displays large "INCOMPLETE" during the process, but at the end shows "(all OK)" and returns errorlevel = 0.
ZPAQFRANZHW t (test) on an repaired archive: Displays lines like: Unordered fragment tables: expected >= 860399 found 796835 Ends with "(with errors)" and returns errorlevel = 2.
ZPAQFRANZHW a (add) on an unrepaired archive: Shows a large "INCOMPLETE", adds files, ends with "(all OK)", and returns errorlevel = 0. Continues adding files to the corrupted archive, just like in older versions.
ZPAQFRANZHW a (add) on an repaired archive: Displays errors such as: Unordered fragment tables: expected >= 1108171 found 1107945 Ends with "(all OK)", returns errorlevel = 0, and continues adding files to the corrupted archive as in previous versions.
Comments: For "t" (test) on an unrepaired archive, it should display something like "(with errors)" or "(with warnings)" and return errorlevel <> 0. For "a" (add) in both cases, "(all OK)" and errorlevel = 0 are misleading. The status should be at least "(with warnings)" or "(with errors)", and errorlevel <> 0.
I ran some tests with 60.10c and am providing feedback so you can decide if this is the result you were expecting.
To remind you - the test archive is corrupted in part 5 or 6 out of a total of 40+ parts.
Results: ZPAQFRANZHW trim - kill: It zeros out only the corrupted part, but the archive remains problematic. Displays "(with warnings)" and returns errorlevel = 1. From your plan, I understand this behavior will remain in the final version: In other cases (e.g., incomplete transaction not at the end), suggest performing a trim and terminate.
When running ZPAQFRANZHW trim - kill again: Displays "(all OK)", returns errorlevel = 0, but the archive remains problematic.
The archive cannot be fixed The fragment sequence is broken, that's it
You should extract all data, ASAP, and rebuild from scratch another archive The goal is TO EXTRACT as much as possible in this scenario
ZPAQFRANZHW t (test) on an unrepaired archive: Displays large "INCOMPLETE" during the process, but at the end shows "(all OK)" and returns errorlevel = 0. I'll fix this
ZPAQFRANZHW t (test) on an repaired archive: Displays lines like: Unordered fragment tables: expected >= 860399 found 796835 Ends with "(with errors)" and returns errorlevel = 2. In fact, those are warnings, not error
And the spiegone... The presence of an unordered fragment table may be caused by archive corruption, which is a true error (data extraction will fail sooner or later), or by the presence of gaps, that is, incomplete transactions that are "skipped." In the latter case—at least in theory—the data can be extracted, all of it if you are lucky. There is no easy way to distinguish between the two cases.
ZPAQFRANZHW a (add) on an unrepaired archive: Shows a large "INCOMPLETE", adds files, ends with "(all OK)", and returns errorlevel = 0. Continues adding files to the corrupted archive, just like in older versions.
You shoud use the -trim switch
ZPAQFRANZHW a (add) on an repaired archive: Displays errors such as: Unordered fragment tables: expected >= 1108171 found 1107945 Ends with "(all OK)", returns errorlevel = 0, and continues adding files to the corrupted archive as in previous versions.
It's the (lack of) -trim
Comments: For "t" (test) on an unrepaired archive, it should display something like "(with errors)" or "(with warnings)" and return errorlevel <> 0. For "a" (add) in both cases, "(all OK)" and errorlevel = 0 are misleading. The status should be at least "(with warnings)" or "(with errors)", and errorlevel <> 0.
I am not ready to change the default behavior of zpaq (and previous versions of zpaqfranz). You can see the "new" behavior by using the temporary -trim switch (which I think I will later remove, or rather reverse its meaning. By default, it will get strict, and with -trim, it will behave like zpaq).
Short version: try -trim
Here you can see zpaq 7.15 happily update a broken multipart, return code 0
C:\zpaqfranz>dir z:\
Il volume nell'unità Z è RamDisk
Numero di serie del volume: EEEC-70CA
Directory di z:\
18/11/2024 17:49 920.718.464 uno_0001.zpaq
18/11/2024 17:49 32.556.353 uno_0002.zpaq
18/11/2024 17:50 1.429.352.890 uno_0003.zpaq
01/01/1980 01:00 223.019.008 uno_0004.zpaq
25/11/2024 16:40 4.401.879 uno_0005.zpaq
5 File 2.610.048.594 byte
0 Directory 83.189.440.512 byte disponibili
C:\zpaqfranz>zpaq64 a z:\uno_???? *.cpp
zpaq v7.15 journaling archiver, compiled Aug 17 2016
z:/uno_????.zpaq: Incomplete transaction ignored
3 versions, 11591 files, 54895 fragments, 2382.627707 MB
Creating z:/uno_0006.zpaq at offset 0 + 2610048594
Adding 88.558771 MB in 51 files -method 14 -threads 32 at 2024-11-25 15:49:20.
4.48% 0:00:01 + nonva01.cpp 3963195 -> 3823584
8.95% 0:00:00 + va02.cpp 3963568 -> 1457576
13.42% 0:00:00 + va03.cpp 3957196 -> 158122
17.89% 0:00:00 + va04.cpp 3957228 -> 65832
(...)
100.00% 0:00:00 + turbo2.cpp 4649
100.00% 0:00:00 [55264..55301] 887433 -method 14,167,0
- c:/nz/freearcnext/_CLS/complex_codec.cpp
(...)
51 +added, 14 -removed.
0.000000 + (88.558771 -> 30.253242 -> 4.400803) = 4.400803 MB
0.672 seconds (all OK)
C:\zpaqfranz>zpaqfranz trim z:\uno_????
zpaqfranz v60.10e-JIT,-GUI,-L,HW SHA1/2,4,SFX64 v55.1,(2024-11-24)
franz:trim 4 - command
franz:-hw
Part count 6 total size 2.614.449.397 bytes
COMPLETE 0 jDC20241118164910c0000000001 <<z:/uno_0001.zpaq>>
COMPLETE 0 jDC20241118164946c0000026258 <<z:/uno_0002.zpaq>>
COMPLETE 0 jDC20241118165004c0000027842 <<z:/uno_0003.zpaq>>
INCOMPLETE 0 jDC20241120114454c0000054896 <<z:/uno_0004.zpaq>>
COMPLETE 0 jDC20241125154014c0000054896 <<z:/uno_0005.zpaq>>
COMPLETE 5 jDC20241125154920c0000054896 <<z:/uno_0006.zpaq>>
Incomplete transactions 1
Complete transactions 5
Incomplete transaction founded, NOT on the last chunk (this is bad)
Using trim -kill you can try to fix to extract data ASAP (and rebuild)
0.125s (00:00:00,heap 9.93MB|array 873.03KB|dt 0=>10.78MB) (with warnings)9```
And here you can see a happy zpaq 7.15 test
C:\zpaqfranz>zpaq64 x z:\uno_???? -test -summary 1
zpaq v7.15 journaling archiver, compiled Aug 17 2016
z:/uno_????.zpaq: Incomplete transaction ignored
3 versions, 11591 files, 54895 fragments, 2382.627707 MB
Extracting 3403.806136 MB in 11112 files -threads 32
1.078 seconds (all OK)
OK you can try an update with v. 60.10f with default autotrim (can be DISABLED with -notrim)
I can't update.
zpaqfranzhw says hash mismatch. zpaqfranz hangs... Isn't this the same bug that was recently fixed?
So in this version what should I test? "zpaqfranz a" should do autotrim by default unless -notrim is specified. Did I understand correctly?
Yes you get It right By default is now very picky -notrim to revert to zpaq style
PS i made no check on p7m at all during upload. I will
Franco, I want to ask you why you care so much about backward compatibility? Even when the behavior is buggy, I see a desire in you to preserve it, and then you possibly introduce a new option specifically to handle the bug.
I’m just curious about the reason for this. ;) -notrim is a beautiful exception ;)
It seems that ZPAQFRANZ.EXE freezes during the update, but it still managed to update successfully.
sorry, ZPAQFRANZ still an older version (60.10c) ... the update wasn't successful.
Backward compatibility is why you can send and receive emails with attachments 50 years after 7bit-style
Changing something is very risky
Do not use relative paths
OK, you can now update to 60.10g. Fixed the bug in batch upload
Hi Franco,
I'm not sure exactly which part of my question about relative paths you were answering, but I tested zpaqfranz update -force using both relative and absolute paths. It hangs in both cases. When I press CTRL-C and try again, it turns out the update actually succeeded the first time.
Do you know why it hangs like that?
OK one thing at time
About the updater, I don’t know what more to say. It just hangs, though it used to work fine until recently.
Renaming <
In my opinion, when the issue is SUCCESSFULLY resolved by add command (trim is performed), the errorlevel should always be 0. The big "INCOMPLETE" message is better displayed only if the gap is further back and hasn’t been fixed, leaving the archive damaged. In fact, in such cases, I would personally halt any further addition of data to the archive, as it becomes pointless. It just wastes space and time. Even if it doesn't add anything and immediately stops execution, there's a higher chance the issue will be noticed because it becomes obvious that no data is being added, as the process finishes so quickly.
Update process:
1 try -debug3. Maybe you get an antisomething update 2 please explain more 3 it is hard, because the check is done before the fix. Doable but a pain in the ass. But After a successful fix the output can be all ok.
- in the output, just below an yellow line, there is: Renaming <
> to <</DC7_0006_00004.spaz>> I found the .spaz files, but they are being created in the root of D:\ instead of in D:\BAD - Copy as expected.
This is a bug to be fixed
If you e-mail me, I can try a teamviewer session to look for missing update
You can try the new pre-release 60.10h A very strange pipe-back-to-father to write on the console And you can try update -force -debug3 to get the script, not launched
Hello Franco, First, I want to tell you that I still haven't started backing up using the method zpaqfranz backup.
Instead, I am still using the classic method
zpaqfranz a _????,
and the bug I want to discuss does not concern it.
there are 52 versions but only 5 are visible, and no errors reported:
...
This issue reveals an insidious problem that can arise unnoticed and lead to data loss. Unlike ZPAQ, ZPAQFRANZ is designed as a backup tool, which is why I think it's important to address this problem and resolve it properly.
By "properly resolved," I mean it should be addressed by default (without requiring specialized options). The standard behavior should either prevent this problem entirely or, at the very least, make it clearly noticeable.
ZPAQFRANZ recognizes the incomplete transaction but does not remove it. It detects that something is wrong, continues without successfully handling the situation, and as a result, all subsequent operations effectively cannot detect or recover from the issue.
The algorithm, as implemented, is not safe.
I have two questions:
Do you think it's a good idea to leave it as is, considering this is likely the most common way users will utilize ZPAQ? Will zpaqfranz backup handle such a scenario? It seems to me that the behavior should be different: a) ZPAQFRANZ A, when detecting an "Incomplete transaction ignored," should attempt to "repair" by deleting back to a healthy archive. b) ZPAQFRANZ T and L, when detecting an "Incomplete transaction ignored," should set a Warning status and the corresponding ERRORLEVEL.