fcorbelli / zpaqfranz

Deduplicating archiver with encryption and paranoid-level tests. Swiss army knife for the serious backup and disaster recovery manager. Ransomware neutralizer. Win/Linux/Unix
MIT License
279 stars 25 forks source link

Problem with "Incomplete transaction ignored9 (block/s)" #146

Open mirogeorg opened 1 week ago

mirogeorg commented 1 week ago

Hello Franco, First, I want to tell you that I still haven't started backing up using the method zpaqfranz backup . Instead, I am still using the classic method zpaqfranz a _????, and the bug I want to discuss does not concern it.


D:\vmware\DC7>dir \\bkpsrv\bkp\DC7_????.zpaq
 Volume in drive \\bkpsrv\bkp has no label.
 Volume Serial Number is 520E-E9B4

 Directory of \\bkpsrv\bkp

10/06/2024  04:13 AM    18,324,189,445 DC7_0001.zpaq
10/07/2024  06:43 AM     1,650,504,616 DC7_0002.zpaq
10/07/2024  08:14 PM       266,201,925 DC7_0003.zpaq
10/09/2024  07:39 AM       608,692,303 DC7_0004.zpaq
10/09/2024  09:20 PM       845,123,078 DC7_0005.zpaq
10/11/2024  11:00 PM     1,743,847,424 DC7_0006.zpaq
10/12/2024  01:37 PM     1,893,937,951 DC7_0007.zpaq
10/13/2024  10:09 AM     2,157,025,042 DC7_0008.zpaq
10/14/2024  06:47 PM     2,358,943,608 DC7_0009.zpaq
10/15/2024  11:20 PM     2,544,744,903 DC7_0010.zpaq
10/16/2024  09:48 PM     2,693,264,666 DC7_0011.zpaq
10/16/2024  09:57 PM     2,833,260,873 DC7_0012.zpaq
10/18/2024  07:24 PM     2,829,669,991 DC7_0013.zpaq
10/18/2024  07:25 PM     2,682,992,600 DC7_0014.zpaq
10/20/2024  02:14 AM     2,905,579,886 DC7_0015.zpaq
10/20/2024  02:07 AM     2,682,846,539 DC7_0016.zpaq
10/20/2024  10:49 PM     2,932,014,256 DC7_0017.zpaq
10/20/2024  11:00 PM     3,171,180,778 DC7_0018.zpaq
10/21/2024  09:16 PM     2,921,751,091 DC7_0019.zpaq
10/21/2024  09:22 PM     3,178,289,994 DC7_0020.zpaq
10/23/2024  02:03 AM     3,171,197,260 DC7_0021.zpaq
10/23/2024  01:59 AM     2,922,808,827 DC7_0022.zpaq
10/24/2024  01:50 AM     3,192,315,865 DC7_0023.zpaq
10/24/2024  01:45 AM     2,924,153,389 DC7_0024.zpaq
10/25/2024  01:17 AM     2,946,048,574 DC7_0025.zpaq
10/25/2024  01:20 AM     3,189,817,451 DC7_0026.zpaq
10/26/2024  01:28 PM     3,202,702,860 DC7_0027.zpaq
10/26/2024  01:32 PM     2,947,486,223 DC7_0028.zpaq
10/27/2024  02:11 AM     2,948,551,025 DC7_0029.zpaq
10/27/2024  02:21 AM     3,151,993,797 DC7_0030.zpaq
10/29/2024  01:21 AM     3,174,010,907 DC7_0031.zpaq
10/29/2024  01:25 AM     2,947,850,430 DC7_0032.zpaq
10/30/2024  12:58 AM     3,168,998,017 DC7_0033.zpaq
10/30/2024  12:53 AM     2,946,487,538 DC7_0034.zpaq
10/31/2024  12:37 AM     3,315,517,740 DC7_0035.zpaq
10/31/2024  12:34 AM     2,947,311,936 DC7_0036.zpaq
11/01/2024  12:28 AM     3,335,537,308 DC7_0037.zpaq
11/01/2024  12:27 AM     2,926,347,335 DC7_0038.zpaq
11/03/2024  04:04 AM     3,384,367,145 DC7_0039.zpaq
11/03/2024  04:05 AM     2,943,661,640 DC7_0040.zpaq
11/05/2024  03:08 AM     3,416,929,483 DC7_0041.zpaq
11/06/2024  03:02 AM     3,416,686,597 DC7_0042.zpaq
11/07/2024  01:23 AM     3,422,982,632 DC7_0043.zpaq
11/10/2024  01:07 AM     3,396,852,605 DC7_0044.zpaq
11/11/2024  01:01 AM     3,386,543,623 DC7_0045.zpaq
11/12/2024  01:03 AM     3,460,696,114 DC7_0046.zpaq
11/13/2024  01:11 AM     3,642,649,741 DC7_0047.zpaq
11/13/2024  11:59 PM     5,082,128,266 DC7_0048.zpaq
11/15/2024  01:31 AM    12,558,138,703 DC7_0049.zpaq
11/17/2024  01:30 AM    12,685,380,813 DC7_0050.zpaq
11/18/2024  01:25 AM    12,834,029,886 DC7_0051.zpaq
11/18/2024  02:13 PM    13,117,342,230 DC7_0052.zpaq

there are 52 versions but only 5 are visible, and no errors reported:

image ... image

image

image This issue reveals an insidious problem that can arise unnoticed and lead to data loss. Unlike ZPAQ, ZPAQFRANZ is designed as a backup tool, which is why I think it's important to address this problem and resolve it properly.

By "properly resolved," I mean it should be addressed by default (without requiring specialized options). The standard behavior should either prevent this problem entirely or, at the very least, make it clearly noticeable.

ZPAQFRANZ recognizes the incomplete transaction but does not remove it. It detects that something is wrong, continues without successfully handling the situation, and as a result, all subsequent operations effectively cannot detect or recover from the issue.

The algorithm, as implemented, is not safe.

I have two questions:

Do you think it's a good idea to leave it as is, considering this is likely the most common way users will utilize ZPAQ? Will zpaqfranz backup handle such a scenario? It seems to me that the behavior should be different: a) ZPAQFRANZ A, when detecting an "Incomplete transaction ignored," should attempt to "repair" by deleting back to a healthy archive. b) ZPAQFRANZ T and L, when detecting an "Incomplete transaction ignored," should set a Warning status and the corresponding ERRORLEVEL.

fcorbelli commented 1 week ago

a) ZPAQFRANZ A, when detecting an "Incomplete transaction ignored," should attempt to "repair" by deleting back to a healthy archive. You can. With the trim command There is not a "trim" command for zpaq

b) ZPAQFRANZ T and L, when detecting an "Incomplete transaction ignored," should set a Warning status and the corresponding ERRORLEVEL.

This can make sense

ZPAQFRANZ recognizes the incomplete transaction but does not remove it. It detects that something is wrong, continues without successfully handling the situation, and as a result, all subsequent operations effectively cannot detect or recover from the issue. You cannot recover from a missing part It is just a "hole" zpaq does detect this situation, and work in the same way "opppssss..."

Suppose you have

piece0001 piece0002 piece0003 piece0004 piece0005 piece0006 piece0008 piece0009 piece0010 piece0011 You can extract upto piece0006. Everything after can, or cannot be, extractable (you can try extracting directly, if not encrypted) You must be very lucky (no spiegone here)

fcorbelli commented 1 week ago

What you essentially need to do for a NON-encrypted multipart archive is replace the file with the incomplete transaction with a zero-length file. At that point, "magically," everything will start working again.

touch piece0007.zpaq
mirogeorg commented 1 week ago

Alright, it can be fixed with ZPAQFRANZ trim, but how can someone know this needs to be done if all the tests pass without errors?

The alternative is the same as with the old ZPAQ: capture the output and search for strings. ;(

This is precisely the issue — ZPAQFRANZ continues to append parts, but they effectively remain inaccessible. The chain is broken. In my opinion, since ZPAQFRANZ is a backup tool, this behavior is unacceptable. I know this is inherited from ZPAQ, but it's a pretty bad legacy and shouldn’t continue like this.

Ideally, ZPAQFRANZ should detect that. In my view, there are two proper solutions: a) It should determine that the chain is broken and stop further additions to the archive, simply returning an error. b) It should allow further additions but return ERRORLEVEL <> 0. The rationale for this option is if the first one is too difficult to implement. It would serve as a quick workaround.

fcorbelli commented 1 week ago

Work in progress 1 bigger advice, showing part hole 2 fix for unencrypted multipart archives (instead of manually renaming) 3 freezer for multipart incomplete

The longer part will be a incomplete enumerator

Doable

fcorbelli commented 1 week ago

The current state is the new fix command, which, on NON-encrypted multipart archives with incomplete transactions, detects and resolves them (at least in theory), allowing the content to be extracted and a new "clean" archive to be created. Essentially, it's a patch job. It also displays a huge warning in case of incomplete transactions, making it easier to understand that the archive is corrupted.

1 2 3 4

mirogeorg commented 1 week ago

haha. That is cool. I'll test when you release version for test

mirogeorg commented 1 week ago

Franco, from the screenshots you’ve taken, I see that the FIX command is attempting to salvage as much as possible from the archives, showing a warning that the data may not be okay.

Perhaps such a command would be better named ZPAQFRANZ salvage.

Wouldn't it be better if ZPAQFRANZ fix were a fast command that quickly repairs the archive and trims it down to what's intact? It would also be good if it set the ERRORLEVEL as follows:

0 when the archive is intact and hasn't been modified, 1 when the repair was successful, and

fcorbelli commented 1 week ago

Obviously, it already exists 😄 For monolithic files, that is, single .zpaq files, there is the trim command (specific to zpaqfranz). It exists for single files, implemented but disabled by default. Essentially, if you press Control-C while adding data, it can automatically trim. Since I’m cautious, I didn’t set it as default. But the point is different. While I can roughly do whatever I want for a Control-C, I can’t do the same for a process kill. In this case, I COULD detect the transaction during the next read_archive and remove it mercilessly. Or mercilessly delete the last part of the multipart file. This would make the archive consistent. BUT it involves deleting pieces of data, which I generally dislike doing. Keep in mind that there can be multiple gaps, i.e., incomplete transactions across more than one file. In short, a very aggressive approach might result in data loss.

I will probably (I need to think about it) adopt a mixed procedure. In the case of multipart archives, if it detects a gap in the last file, it simply renames it to .spaz. This way, by continuing to add data, the archive remains intact. If, however, the gap is NOT in the last piece, it displays an error message and advises you to proceed with a fix (which will actually be called trim later. Essentially, there will be a single trim command that will work differently for single and multipart files).

Then there's the issue of chunk archives, for which this simply can’t be done. And encrypted ones, which can’t be managed either. And those with an index, which makes everything much more complicated. Nothing is easy with zpaq. Therefore, it won’t be a universal command but specific to simple multipart files.

As you can see, these are big problems, very big problems to solve. In fact, Mahoney didn’t even tackle them.

Finally, I think I’ll add a detailed explanation on a wiki page, and zpaqfranz will provide the link. If you want to know what to try in your case, click here.

mirogeorg commented 1 week ago

My recommendation is in two directions:

  1. It should be implemented in a way that is user-friendly and clear for an inexperienced user who is not even conceptually familiar with this issue. In my opinion, A / BACKUP should independently detect and resolve the problem if one is identified, without asking any questions. If it’s not too difficult to implement in the code, I don’t see an issue, especially since the original ZPAQ automatically trims corrupted transactions without options and without asking (we’re talking about single-part archives). Therefore, the current multipart implementation should (finally) align the behavior with that of single-part archives.

  2. It should be convenient for scripting. When scripting, it’s particularly difficult and unreliable to check statuses, various scenarios, and whether a TRIM is needed after each operation. This significantly decreases reliability.

In my opinion, reliability (and simplicity) should take priority above all else.

What do you think?

fcorbelli commented 1 week ago

My recommendation is in two directions:

  1. It should be implemented in a way that is user-friendly and clear for an inexperienced user who is not even conceptually familiar with this issue. In my opinion, A / BACKUP should independently detect and resolve the problem if one is identified, without asking any questions.

It is risky

If it’s not too difficult to implement in the code, I don’t see an issue, especially since the original ZPAQ automatically trims corrupted transactions without options and without asking (we’re talking about single-part archives).

No, zpaq does not trim corrupted transactions, at all

Therefore, the current multipart implementation should (finally) align the behavior with that of single-part archives.

It is impossible, because a missing part can be anywhere Not just in the very last chunk

  1. It should be convenient for scripting. When scripting, it’s particularly difficult and unreliable to check statuses, various scenarios, and whether a TRIM is needed after each operation. This significantly decreases reliability.

In my opinion, reliability (and simplicity) should take priority above all else.

What do you think?

That I do not design zpaq's file format in such a bizzarre way

Therefore a lot of things are simply impossible, without breaking backward compatibility

fcorbelli commented 1 week ago

This is zpaq 7.15

// Truncate empty update from archive (if not indexed)
  if (!index) {
    if (added+removed==0 && archive_end-header_pos==104) // no update
      archive_end=header_pos;
    if (archive_end<archive_size) {
      if (archive_end>0) {
        printf("truncating archive from %1.0f to %1.0f\n",
            double(archive_size), double(archive_end));
        if (truncate(arcname.c_str(), archive_end)) printerr(archive.c_str());
      }
      else if (archive_end==0) {
        if (delete_file(arcname.c_str())) {
          printf("deleted ");
          printUTF8(arcname.c_str());
          printf("\n");
        }
      }
    }
  }

This is zpaqfranz The same

// Truncate empty update from archive (if not indexed)
    if (!index)
    {
        if (g_chunk_size>0)
        {
            if (flagverbose)
            {
                printbar('-');
                for (unsigned int i=0;i<out.filepartnames.size();i++)
                    myprintf("02136: Chunk %08d %21s %s\n",i+1,migliaia(prendidimensionefile(out.filepartnames[i].c_str())),out.filepartnames[i].c_str());
                printbar('=');
            }
            string thelast=out.lastfilename();
            myprintf("02137: INFO: The last chunk is: %Z\n",thelast.c_str());

            if (prendidimensionefile(thelast.c_str())==0)
                if (delete_file(thelast.c_str()))
                {
                    myprintf("02138: deleted %Z : no data to be keeped\n",thelast.c_str());

                }
            if (flagdebug3)
                myprintf("02139: thecdatasize %21s\n",migliaia(thecdatasize));

        }
        else //default, not chunked
        {
            if (flagdebug3)
                myprintf("02140: arcname %s\n",arcname.c_str());
            if (added+removed==0 && archive_end-header_pos==104) // no update
                archive_end=header_pos;
            if ((archive_end<archive_size) && (g_chunk_size==0))
            {
                if (archive_end>0)
                {
                    if (flagverbose)
                        myprintf("02141: truncating archive from %s to %s\n",migliaia(archive_size), migliaia2(archive_end));
                    if (truncate(arcname.c_str(), archive_end))
                        printerr("17092",archive.c_str(),0);
                    if (fasttxt!="")
                    {
                        myprintf("02142: Turning off fasttxt due to truncation [archive not changed]\n");
                        flagfasttxt=false; // we do not want to update CRC-32!
                        fasttxt="";
                    }
                    if (g_starting_zpaqdate>0)
                    {
                        if (flagverbose)
                            myprintf("02143: touching back to %s\n",dateToString(false,g_starting_zpaqdate).c_str());
                        if (!touch(arcname.c_str(),g_starting_zpaqdate,g_starting_zpaqattr))
                            myprintf("02144$ WARNING trouble in touching\n");
                    }
                }
                else
                if (archive_end==0)
                {
                    if (delete_file(arcname.c_str()))
                        if (flagverbose)
                        {
                            myprintf("02145: deleted %Z: no data to be archived\n",arcname.c_str());
                        }
                }
            }
        }
    }
fcorbelli commented 1 week ago

Short version: I’m working on a trimmer for multipart archives that, if the gap is in the very last piece, renames it, simulating truncation. If the gap isn’t in the last part, it’s your problem.

mirogeorg commented 1 week ago

You’re saying it’s risky if you have old, potentially corrupted archives… But they’re still old AND corrupted. Isn’t it even riskier for new ones to keep getting corrupted… Especially in a script, where there are no eyes to see the big warning sign? ;)

Isn’t it better for the future to be brighter and more promising than the past? ;)

OK. thank you for the effort!

fcorbelli commented 1 week ago

Sadly, it is impossible, due to the -chunk new option Did I already mention that I didn’t design the data storage format? 😄

fcorbelli commented 1 week ago

60_10b.zip You can try this (highly instable) pre-release

This should:

  1. If it finds a non-chunked, non-encrypted multipart archive with an incomplete transaction right at the last piece, rename it and continue.
  2. In any case, display a very large warning.
  3. In other cases (e.g., incomplete transaction not at the end), suggest performing a trim and terminate.

The trim command now (perhaps) also works for multipart archives (non-chunked and non-encrypted). When launched normally, like zpaqfranz trim z:\uno_????, it shows a report. With the -kill switch, it attempts, as much as possible, to repair the archive. If the erroneous piece is the last one, the archive is valid. If not, you'll need to extract the data and start over, as the internal structure is compromised.

fcorbelli commented 6 days ago

60_10c.zip This pre-release add the -trim switch (in add and backup) that (should) trim incomplete transaction, if possible, deleting the last chunk

fcorbelli commented 4 days ago

The current ongoing "superspiegone"

https://github.com/fcorbelli/zpaqfranz/wiki/Real-world:-corruption

mirogeorg commented 2 days ago

I ran some tests with 60.10c and am providing feedback so you can decide if this is the result you were expecting.

To remind you - the test archive is corrupted in part 5 or 6 out of a total of 40+ parts.

Results: ZPAQFRANZHW trim - kill: It zeros out only the corrupted part, but the archive remains problematic. Displays "(with warnings)" and returns errorlevel = 1. From your plan, I understand this behavior will remain in the final version: In other cases (e.g., incomplete transaction not at the end), suggest performing a trim and terminate.

When running ZPAQFRANZHW trim - kill again: Displays "(all OK)", returns errorlevel = 0, but the archive remains problematic.

ZPAQFRANZHW t (test) on an unrepaired archive: Displays large "INCOMPLETE" during the process, but at the end shows "(all OK)" and returns errorlevel = 0.

ZPAQFRANZHW t (test) on an repaired archive: Displays lines like: Unordered fragment tables: expected >= 860399 found 796835 Ends with "(with errors)" and returns errorlevel = 2.

ZPAQFRANZHW a (add) on an unrepaired archive: Shows a large "INCOMPLETE", adds files, ends with "(all OK)", and returns errorlevel = 0. Continues adding files to the corrupted archive, just like in older versions.

ZPAQFRANZHW a (add) on an repaired archive: Displays errors such as: Unordered fragment tables: expected >= 1108171 found 1107945 Ends with "(all OK)", returns errorlevel = 0, and continues adding files to the corrupted archive as in previous versions.

Comments: For "t" (test) on an unrepaired archive, it should display something like "(with errors)" or "(with warnings)" and return errorlevel <> 0. For "a" (add) in both cases, "(all OK)" and errorlevel = 0 are misleading. The status should be at least "(with warnings)" or "(with errors)", and errorlevel <> 0.

fcorbelli commented 1 day ago

I ran some tests with 60.10c and am providing feedback so you can decide if this is the result you were expecting.

To remind you - the test archive is corrupted in part 5 or 6 out of a total of 40+ parts.

Results: ZPAQFRANZHW trim - kill: It zeros out only the corrupted part, but the archive remains problematic. Displays "(with warnings)" and returns errorlevel = 1. From your plan, I understand this behavior will remain in the final version: In other cases (e.g., incomplete transaction not at the end), suggest performing a trim and terminate.

When running ZPAQFRANZHW trim - kill again: Displays "(all OK)", returns errorlevel = 0, but the archive remains problematic.

The archive cannot be fixed The fragment sequence is broken, that's it

You should extract all data, ASAP, and rebuild from scratch another archive The goal is TO EXTRACT as much as possible in this scenario

fcorbelli commented 1 day ago

ZPAQFRANZHW t (test) on an unrepaired archive: Displays large "INCOMPLETE" during the process, but at the end shows "(all OK)" and returns errorlevel = 0. I'll fix this

ZPAQFRANZHW t (test) on an repaired archive: Displays lines like: Unordered fragment tables: expected >= 860399 found 796835 Ends with "(with errors)" and returns errorlevel = 2. In fact, those are warnings, not error

And the spiegone... The presence of an unordered fragment table may be caused by archive corruption, which is a true error (data extraction will fail sooner or later), or by the presence of gaps, that is, incomplete transactions that are "skipped." In the latter case—at least in theory—the data can be extracted, all of it if you are lucky. There is no easy way to distinguish between the two cases.

ZPAQFRANZHW a (add) on an unrepaired archive: Shows a large "INCOMPLETE", adds files, ends with "(all OK)", and returns errorlevel = 0. Continues adding files to the corrupted archive, just like in older versions.

You shoud use the -trim switch

fcorbelli commented 1 day ago

ZPAQFRANZHW a (add) on an repaired archive: Displays errors such as: Unordered fragment tables: expected >= 1108171 found 1107945 Ends with "(all OK)", returns errorlevel = 0, and continues adding files to the corrupted archive as in previous versions.

It's the (lack of) -trim

Comments: For "t" (test) on an unrepaired archive, it should display something like "(with errors)" or "(with warnings)" and return errorlevel <> 0. For "a" (add) in both cases, "(all OK)" and errorlevel = 0 are misleading. The status should be at least "(with warnings)" or "(with errors)", and errorlevel <> 0.

I am not ready to change the default behavior of zpaq (and previous versions of zpaqfranz). You can see the "new" behavior by using the temporary -trim switch (which I think I will later remove, or rather reverse its meaning. By default, it will get strict, and with -trim, it will behave like zpaq).

fcorbelli commented 1 day ago

Short version: try -trim

fcorbelli commented 1 day ago

Here you can see zpaq 7.15 happily update a broken multipart, return code 0


C:\zpaqfranz>dir z:\
 Il volume nell'unità Z è RamDisk
 Numero di serie del volume: EEEC-70CA

 Directory di z:\

18/11/2024  17:49       920.718.464 uno_0001.zpaq
18/11/2024  17:49        32.556.353 uno_0002.zpaq
18/11/2024  17:50     1.429.352.890 uno_0003.zpaq
01/01/1980  01:00       223.019.008 uno_0004.zpaq
25/11/2024  16:40         4.401.879 uno_0005.zpaq
               5 File  2.610.048.594 byte
               0 Directory  83.189.440.512 byte disponibili

C:\zpaqfranz>zpaq64 a z:\uno_???? *.cpp
zpaq v7.15 journaling archiver, compiled Aug 17 2016
z:/uno_????.zpaq: Incomplete transaction ignored
3 versions, 11591 files, 54895 fragments, 2382.627707 MB
Creating z:/uno_0006.zpaq at offset 0 + 2610048594
Adding 88.558771 MB in 51 files -method 14 -threads 32 at 2024-11-25 15:49:20.
 4.48% 0:00:01 + nonva01.cpp 3963195 -> 3823584
 8.95% 0:00:00 + va02.cpp 3963568 -> 1457576
13.42% 0:00:00 + va03.cpp 3957196 -> 158122
17.89% 0:00:00 + va04.cpp 3957228 -> 65832
(...)
100.00% 0:00:00 + turbo2.cpp 4649
100.00% 0:00:00 [55264..55301] 887433 -method 14,167,0
- c:/nz/freearcnext/_CLS/complex_codec.cpp
(...)
51 +added, 14 -removed.

0.000000 + (88.558771 -> 30.253242 -> 4.400803) = 4.400803 MB
0.672 seconds (all OK)

C:\zpaqfranz>zpaqfranz trim z:\uno_????
zpaqfranz v60.10e-JIT,-GUI,-L,HW SHA1/2,4,SFX64 v55.1,(2024-11-24)
franz:trim                                      4 - command
franz:-hw
Part count 6 total size 2.614.449.397 bytes
COMPLETE         0 jDC20241118164910c0000000001  <<z:/uno_0001.zpaq>>
COMPLETE         0 jDC20241118164946c0000026258  <<z:/uno_0002.zpaq>>
COMPLETE         0 jDC20241118165004c0000027842  <<z:/uno_0003.zpaq>>
INCOMPLETE       0 jDC20241120114454c0000054896  <<z:/uno_0004.zpaq>>
COMPLETE         0 jDC20241125154014c0000054896  <<z:/uno_0005.zpaq>>
COMPLETE         5 jDC20241125154920c0000054896  <<z:/uno_0006.zpaq>>
Incomplete transactions         1
Complete   transactions         5
Incomplete transaction founded, NOT on the last chunk     (this is bad)
Using trim -kill you can try to fix to extract data ASAP  (and rebuild)
0.125s (00:00:00,heap 9.93MB|array 873.03KB|dt 0=>10.78MB) (with warnings)9```
fcorbelli commented 1 day ago

And here you can see a happy zpaq 7.15 test

C:\zpaqfranz>zpaq64 x  z:\uno_???? -test -summary 1
zpaq v7.15 journaling archiver, compiled Aug 17 2016
z:/uno_????.zpaq: Incomplete transaction ignored
3 versions, 11591 files, 54895 fragments, 2382.627707 MB
Extracting 3403.806136 MB in 11112 files -threads 32
1.078 seconds (all OK)
fcorbelli commented 1 day ago

OK you can try an update with v. 60.10f with default autotrim (can be DISABLED with -notrim)

mirogeorg commented 1 day ago

I can't update. image

zpaqfranzhw says hash mismatch. zpaqfranz hangs... Isn't this the same bug that was recently fixed?

So in this version what should I test? "zpaqfranz a" should do autotrim by default unless -notrim is specified. Did I understand correctly?

fcorbelli commented 1 day ago

Yes you get It right By default is now very picky -notrim to revert to zpaq style

PS i made no check on p7m at all during upload. I will

mirogeorg commented 1 day ago

Franco, I want to ask you why you care so much about backward compatibility? Even when the behavior is buggy, I see a desire in you to preserve it, and then you possibly introduce a new option specifically to handle the bug.

I’m just curious about the reason for this. ;) -notrim is a beautiful exception ;)

It seems that ZPAQFRANZ.EXE freezes during the update, but it still managed to update successfully.

mirogeorg commented 1 day ago

sorry, ZPAQFRANZ still an older version (60.10c) ... the update wasn't successful.

fcorbelli commented 1 day ago

Backward compatibility is why you can send and receive emails with attachments 50 years after 7bit-style

Changing something is very risky

fcorbelli commented 1 day ago

Do not use relative paths

fcorbelli commented 1 day ago

OK, you can now update to 60.10g. Fixed the bug in batch upload

mirogeorg commented 22 hours ago

Hi Franco,

I'm not sure exactly which part of my question about relative paths you were answering, but I tested zpaqfranz update -force using both relative and absolute paths. It hangs in both cases. When I press CTRL-C and try again, it turns out the update actually succeeded the first time.

Do you know why it hangs like that?

fcorbelli commented 22 hours ago

OK one thing at time update

mirogeorg commented 20 hours ago

image

  1. About the updater, I don’t know what more to say. It just hangs, though it used to work fine until recently.

  2. Renaming <> to <</DC7_0006_00004.spaz>> – such a file didn’t appear. Just for info...

  3. In my opinion, when the issue is SUCCESSFULLY resolved by add command (trim is performed), the errorlevel should always be 0. The big "INCOMPLETE" message is better displayed only if the gap is further back and hasn’t been fixed, leaving the archive damaged. In fact, in such cases, I would personally halt any further addition of data to the archive, as it becomes pointless. It just wastes space and time. Even if it doesn't add anything and immediately stops execution, there's a higher chance the issue will be noticed because it becomes obvious that no data is being added, as the process finishes so quickly.

mirogeorg commented 20 hours ago

Update process: image

fcorbelli commented 19 hours ago

1 try -debug3. Maybe you get an antisomething update 2 please explain more 3 it is hard, because the check is done before the fix. Doable but a pain in the ass. But After a successful fix the output can be all ok.

mirogeorg commented 9 hours ago
  1. in the output, just below an yellow line, there is: Renaming <> to <</DC7_0006_00004.spaz>> I found the .spaz files, but they are being created in the root of D:\ instead of in D:\BAD - Copy as expected.
fcorbelli commented 4 hours ago
  1. in the output, just below an yellow line, there is: Renaming <> to <</DC7_0006_00004.spaz>> I found the .spaz files, but they are being created in the root of D:\ instead of in D:\BAD - Copy as expected.

This is a bug to be fixed

fcorbelli commented 4 hours ago

If you e-mail me, I can try a teamviewer session to look for missing update

fcorbelli commented 3 hours ago

You can try the new pre-release 60.10h A very strange pipe-back-to-father to write on the console And you can try update -force -debug3 to get the script, not launched