search

fcorbelli / zpaqfranz

Deduplicating archiver with encryption and paranoid-level tests. Swiss army knife for the serious backup and disaster recovery manager. Ransomware neutralizer. Win/Linux/Unix
MIT License
279 stars 25 forks source link

Issue with the -stdin option #40

Closed Erol-2022 closed 1 year ago

Erol-2022 commented 1 year ago

Hello Mr. Corbelli,

The new -stdin option introduced with the release v56.1j does not work . Here is a quick test.

Creating a simple file to test dd for Windows :

D:\>echo This is a test. > sample.txt

D:\>dd if=sample.txt of=test.txt
rawwrite dd for windows version 0.6beta3.
Written by John Newbigin <jn@it.swin.edu.au>
This program is covered by terms of the GPL Version 2.

0+1 records in
0+1 records out

D:\>fc sample.txt test.txt
Comparing files sample.txt and TEST.TXT
FC: no differences encountered

Archiving the text file with zpaqfranz does not have any effect, I get only the options presenting how to use zpaqfranz.exe : 

D:\>dd if=sample.txt | zpaqfranz.exe a output.zpaq -stdin
rawwrite dd for windows version 0.6beta3.
Written by John Newbigin <jn@it.swin.edu.au>
This program is covered by terms of the GPL Version 2.

zpa0+1 records in
q0+1 records out
franz v56.1j-JIT-L (HW BLAKE3), SFX64 v55.1, (15 Nov 2022)
franz:-stdin
Usage: zpaqfranz command archive[.zpaq] files|directory... -switches...
             h: **** Help on Help ****|  With great power comes great  ...  help!
             a: Append files          |          t: Test (integrity)
             x: Extract versions      |          l: List files
             v: Verify on filesystem  |          i: Info (show versions)
           sfx: Create SFX (Windows)  |         rd: Remove  'highlander'  folders
 c d0 d1 d2...: Compare d0 to d1,d2.. | s d0 d1 d2: Cumulative size of d0, d1, d2
 r d0 d1 d2...: Mirror  d0 in d1...   |       d d0: Deduplicate d0  WITHOUT MERCY
 z d0 d1 d2...: Delete empty dirs     |        m X: Merge multipart archive
          f d0: Fill /wipe free space |     utf d0: Detox filenames  in  d0
  sum d0 d1...: Hashing/deduplication |     dir d0: Win dir (/s /a /os /od)
     n d0 -n X: Keep X files in d0    |          b: CPU benchmarking (-all)
  find d0 what: Search file(s)        | cp X -to Y: Copy files (w/wildcards) to Y
          trim: Trim incomplete add() |    dirsize: Get size of archive's folders
      password: Add/remove/change pwd |   q z:\foo: VSS-archive C: in z:\foo.zpaq
      autotest: Check internals       |      pause: Pause script
                                 Main | switches
      -all [N]: All versions N digit  |     -key X: Use/set archive password to X
 -mN -method N: 0..5= faster..better  |     -force: Always overwrite (extraction)
         -test: Test (extract/add)    |      -kill: Allow destructive (NO dryrun)
    -to out...: Prefix files to out   |   -until N: Roll back to N'th version

output.zpaq is not generated. The statements :
zpa0+1 records in q0+1 records out

are not very meaningful since there is no any output.

Thanks for your support.

fcorbelli commented 1 year ago

The stdin is rewritted into 56.2 You can find the pre release here www.francocorbelli.it/zpaqfranz.cpp

This version include a full size dd (!) for Windows to make images of drives (with admin privileges of course) creating a batch file It seems to work Almost no error handling etc

zpaqfranz a d:\thebackup.zpaq c: -dd

The very last version (not released yet) image drive without dd and stdin, reading directly disk sectors

fcorbelli commented 1 year ago

Ps even with version 56.1 should work (slow) BUT you must give a name for the stdin stream

Something | zpaqfranz a thearchive.zpaq NAMEOFSTREAM -stdin

Then into thearchive.zpaq a NAMEOFSTREAM (or whatever) file will be stored As said 56.1 use a full on byte at time (aka should work everytime,but slow, about 8 MB/s) 56.2 (more complex) runs up to 270MB/s if you use something like -minsize 1MB with -dd (the bs parameter)

Erol-2022 commented 1 year ago

Hello Mr. Corbelli,

Thanks for your hard work. Much appreciated. Sorry if I missed the parameter specifiying the stream name.

With your help, I managed to get it work : 

D:\>dd if=sample.txt | zpaqfranz.exe a output.zpaq stream -stdin
zpaqfranz v56.1j-JIT-L (HW BLAKE3), SFX64 v55.1, (15 Nov 2022)
franz:-stdin
40587: REBUILDING STDIN
40828: IMPORT FROM STDIN
Creating output.zpaq at offset 0 + 0
Adding stream from stdin with 16 threads @ 2022-11-29 12:47:45
rawwrite dd for windows version 0.6beta3.
Written by John Newbigin <jn@it.swin.edu.au>
This program is covered by terms of the GPL Version 2.

2378+1 records in
2378+1 records out
1 +added, 0 -removed.

0 + (1.217.704 -> 1.217.700 -> 3.336) = 3.336 @ 9.29 MB/s

0.125 seconds (00:00:00) (all OK)

D:\>dir output.zpaq

 Directory of D:\

29.11.2022  12:47             3.336 output.zpaq
               1 File(s)          3.336 bytes

I will do other testings on the dd capability of zpaqfranz.

Thanks again,

Erol

fcorbelli commented 1 year ago

As said 56.2 is MUCH faster You can download from my italian homesite (not sure if even an updated exe is here, just try, it is safe, my own site afterall :)

Erol-2022 commented 1 year ago

Hello Mr. Corbelli,

Thanks, I can download the source code :

D:>wget http://www.francocorbelli.it/zpaqfranz.cpp

Any link to your website to receive the latest executable?

fcorbelli commented 1 year ago

I am not sure, but you can try www.francocorbelli.it/zpaqfranz.exe

I really do not remember if build script upload the exe too (I am now on smartphone)

Erol-2022 commented 1 year ago

Hello Mr. Corbelli,

Thanks, I downloaded the latest executable from your website but upon running, it reports :

“This version of this file is not compatible with the version of Windows you’re running”

OS : Windows 10 64-bit Version 10.0.19044.2251

I decided to build the executable with Msys2 : 

wget https://github.com/fcorbelli/zpaqfranz/archive/refs/tags/55.16.tar.gz
tar xvf 55.16.tar.gz zpaqfranz-55.16/
cd zpaqfranz-55.16/
mv zpaqfranz.cpp zpaqfranz.cpp.original
wget http://www.francocorbelli.it/zpaqfranz.cpp
cd WINDOWS/
cp -p ../*.cpp .
make

After some warning messages, Msys2 managed to compile the source code : 

C:/msys64/mingw64/include/winnt.h:6151: note: this is the location of the previous definition
 6151 | #define IO_REPARSE_TAG_CLOUD (__MSABI_LONG(0x9000001A))
      |
zpaqfranz.cpp:30932: warning: "IO_REPARSE_TAG_CLOUD_1" redefined
30932 | #define IO_REPARSE_TAG_CLOUD_1 0x9000101A
      |
C:/msys64/mingw64/include/winnt.h:6152: note: this is the location of the previous definition
 6152 | #define IO_REPARSE_TAG_CLOUD_1 (__MSABI_LONG(0x9000101A))
      |
zpaqfranz.cpp:30933: warning: "IO_REPARSE_TAG_CLOUD_2" redefined
30933 | #define IO_REPARSE_TAG_CLOUD_2 0x9000201A
      |
C:/msys64/mingw64/include/winnt.h:6153: note: this is the location of the previous definition
 6153 | #define IO_REPARSE_TAG_CLOUD_2 (__MSABI_LONG(0x9000201A))
      |
zpaqfranz.cpp:30934: warning: "IO_REPARSE_TAG_CLOUD_3" redefined
30934 | #define IO_REPARSE_TAG_CLOUD_3 0x9000301A
      |
C:/msys64/mingw64/include/winnt.h:6154: note: this is the location of the previous definition
 6154 | #define IO_REPARSE_TAG_CLOUD_3 (__MSABI_LONG(0x9000301A))
      |
zpaqfranz.cpp:30935: warning: "IO_REPARSE_TAG_CLOUD_4" redefined
30935 | #define IO_REPARSE_TAG_CLOUD_4 0x9000401A
      |
C:/msys64/mingw64/include/winnt.h:6155: note: this is the location of the previous definition
 6155 | #define IO_REPARSE_TAG_CLOUD_4 (__MSABI_LONG(0x9000401A))
      |
zpaqfranz.cpp:30936: warning: "IO_REPARSE_TAG_CLOUD_5" redefined
30936 | #define IO_REPARSE_TAG_CLOUD_5 0x9000501A
      |
C:/msys64/mingw64/include/winnt.h:6156: note: this is the location of the previous definition
 6156 | #define IO_REPARSE_TAG_CLOUD_5 (__MSABI_LONG(0x9000501A))
      |
zpaqfranz.cpp:30937: warning: "IO_REPARSE_TAG_CLOUD_6" redefined
30937 | #define IO_REPARSE_TAG_CLOUD_6 0x9000601A
      |
C:/msys64/mingw64/include/winnt.h:6157: note: this is the location of the previous definition
 6157 | #define IO_REPARSE_TAG_CLOUD_6 (__MSABI_LONG(0x9000601A))
      |
zpaqfranz.cpp:30938: warning: "IO_REPARSE_TAG_CLOUD_7" redefined
30938 | #define IO_REPARSE_TAG_CLOUD_7 0x9000701A
      |
C:/msys64/mingw64/include/winnt.h:6158: note: this is the location of the previous definition
 6158 | #define IO_REPARSE_TAG_CLOUD_7 (__MSABI_LONG(0x9000701A))
      |
zpaqfranz.cpp:30939: warning: "IO_REPARSE_TAG_CLOUD_8" redefined
30939 | #define IO_REPARSE_TAG_CLOUD_8 0x9000801A
      |
C:/msys64/mingw64/include/winnt.h:6159: note: this is the location of the previous definition
 6159 | #define IO_REPARSE_TAG_CLOUD_8 (__MSABI_LONG(0x9000801A))
      |
zpaqfranz.cpp:30940: warning: "IO_REPARSE_TAG_CLOUD_9" redefined
30940 | #define IO_REPARSE_TAG_CLOUD_9 0x9000901A
      |
C:/msys64/mingw64/include/winnt.h:6160: note: this is the location of the previous definition
 6160 | #define IO_REPARSE_TAG_CLOUD_9 (__MSABI_LONG(0x9000901A))
      |
zpaqfranz.cpp:30941: warning: "IO_REPARSE_TAG_CLOUD_A" redefined
30941 | #define IO_REPARSE_TAG_CLOUD_A 0x9000A01A
      |
C:/msys64/mingw64/include/winnt.h:6161: note: this is the location of the previous definition
 6161 | #define IO_REPARSE_TAG_CLOUD_A (__MSABI_LONG(0x9000A01A))
      |
zpaqfranz.cpp:30942: warning: "IO_REPARSE_TAG_CLOUD_B" redefined
30942 | #define IO_REPARSE_TAG_CLOUD_B 0x9000B01A
      |
C:/msys64/mingw64/include/winnt.h:6162: note: this is the location of the previous definition
 6162 | #define IO_REPARSE_TAG_CLOUD_B (__MSABI_LONG(0x9000B01A))
      |
zpaqfranz.cpp:30943: warning: "IO_REPARSE_TAG_CLOUD_C" redefined
30943 | #define IO_REPARSE_TAG_CLOUD_C 0x9000C01A
      |
C:/msys64/mingw64/include/winnt.h:6163: note: this is the location of the previous definition
 6163 | #define IO_REPARSE_TAG_CLOUD_C (__MSABI_LONG(0x9000C01A))
      |
zpaqfranz.cpp:30944: warning: "IO_REPARSE_TAG_CLOUD_D" redefined
30944 | #define IO_REPARSE_TAG_CLOUD_D 0x9000D01A
      |
C:/msys64/mingw64/include/winnt.h:6164: note: this is the location of the previous definition
 6164 | #define IO_REPARSE_TAG_CLOUD_D (__MSABI_LONG(0x9000D01A))
      |
zpaqfranz.cpp:30945: warning: "IO_REPARSE_TAG_CLOUD_E" redefined
30945 | #define IO_REPARSE_TAG_CLOUD_E 0x9000E01A
      |
C:/msys64/mingw64/include/winnt.h:6165: note: this is the location of the previous definition
 6165 | #define IO_REPARSE_TAG_CLOUD_E (__MSABI_LONG(0x9000E01A))
      |
zpaqfranz.cpp:30946: warning: "IO_REPARSE_TAG_CLOUD_F" redefined
30946 | #define IO_REPARSE_TAG_CLOUD_F 0x9000F01A
      |
C:/msys64/mingw64/include/winnt.h:6166: note: this is the location of the previous definition
 6166 | #define IO_REPARSE_TAG_CLOUD_F (__MSABI_LONG(0x9000F01A))
      |

The executable : 

# ls -l zpaqfranz.exe
-rwxr-xr-x 1 Erol Domain Users 3979776 Nov 29 14:10 zpaqfranz.exe

# ./zpaqfranz.exe | grep v56
zpaqfranz v56.2d-JIT-L (HW BLAKE3), SFX64 v55.1, (28 Nov 2022)

I will test the new exe built with Msys2.

fcorbelli commented 1 year ago

I confirm that the .EXE is not good As mentioned I did not remember the script

You need to compile from the .cpp (this IS good) with something like

g++ -O3  zpaqfranz.cpp -o zpaqfranz -pthread -static

Or even

g++ -O3  zpaqfranz.cpp -o zpaqfranz
Erol-2022 commented 1 year ago

Hello Mr.Corbelli,

g++ -O3  zpaqfranz.cpp -o zpaqfranz : zpaqfranz.exe

g++ -O3  zpaqfranz.cpp -o zpaqfranz -pthread -static : zpaqfranz.exe.second-build

make : zpaqfranz.exe.first-build

The make file is producing the smallest executable. 

# ls -l z*.exe*
-rwxr-xr-x 1 Erol Domain Users 4267521 Nov 29 15:08 zpaqfranz.exe
-rwxr-xr-x 1 Erol Domain Users 3979776 Nov 29 14:10 zpaqfranz.exe.first-build
-rwxr-xr-x 1 Erol Domain Users 4767652 Nov 29 15:07 zpaqfranz.exe.second-build

Thanks for your help.

fcorbelli commented 1 year ago

In the comment (of source code) you will find the suggested way to compile zpaqfranz

TARGET EXAMPLES

Windows 64 (g++ 7.3.0)
g++ -O3  zpaqfranz.cpp -o zpaqfranz 

Windows 64 (g++ 10.3.0) MSYS2
g++ -O3  zpaqfranz.cpp -o zpaqfranz -pthread -static

Windows 64 (g++, Hardware Blake3 implementation)
In this case, of course, linking the .S file is mandatory
g++ -O3 -DHWBLAKE3 blake3_windows_gnu.S zpaqfranz.cpp -o zpaqfranz -pthread -static

Windows 64 (g++, Hardware Blake3 implementation PLUS HW SHA1)
g++ -O3 -DHWBLAKE3 -DHWSHA1 blake3_windows_gnu.s zpaqfranz.cpp sha1ugo.obj -o zpaqfranzhw -pthread -static

Windows 32 (g++ 7.3.0 64 bit)
c:\mingw32\bin\g++ -m32 -O3 zpaqfranz.cpp -o zpaqfranz32 -pthread -static

Windows 64 (g++ 7.3.0), WITH cloud paq
g++ -O3 -DSERVER zpaqfranz.cpp -o zpaqfranz -lwsock32 -lws2_32

FreeBSD (11.x) gcc 7
gcc7 -O3 -march=native -Dunix zpaqfranz.cpp -lstdc++ -pthread -o zpaqfranz -static -lm

FreeBSD (12.1) gcc 9.3.0
g++ -O3 -march=native -Dunix zpaqfranz.cpp  -pthread -o zpaqfranz -static-libstdc++ -static-libgcc

FreeBSD (11.4) gcc 10.2.0
g++ -O3 -march=native -Dunix zpaqfranz.cpp  -pthread -o zpaqfranz -static-libstdc++ -static-libgcc -Wno-stringop-overflow

FreeBSD (11.3) clang 6.0.0
clang++ -march=native -Dunix zpaqfranz.cpp  -pthread -o zpaqfranz -static

OpenBSD 6.6 clang++ 8.0.1
OpenBSD 7.1 clang++ 13.0.0
clang++ -Dunix -O3 -march=native zpaqfranz.cpp -o zpaqfranz -pthread -static

Arch Linux
I strongly advise against using zpaqfranz on this Linux distro(s).
There is a bizarre policy on compiling executables.
Obviously no one forbids it (runs fine), but just don't ask for help.

Debian Linux (10/11) gcc 8.3.0
ubuntu 21.04 desktop-amd64 gcc  10.3.0
manjaro 21.07 gcc 11.1.0
g++ -O3 -Dunix zpaqfranz.cpp  -pthread -o zpaqfranz -static

QNAP NAS TS-431P3 (Annapurna AL314) gcc 7.4.0
g++ -Dunix zpaqfranz.cpp  -pthread -o zpaqfranz -Wno-psabi

Fedora 34 gcc 11.2.1
Typically you will need some library (out of a fresh Fedora box)
sudo dnf install glibc-static libstdc++-static -y;
Then you can compile, via Makefile or "by hand"
(do not forget... sudo!)

CentoOS
Please note:
"Red Hat discourages the use of static linking for security reasons. 
Use static linking only when necessary, especially against libraries provided by Red Hat. "
Therefore a -static linking is often a nightmare on CentOS => change the Makefile
g++ -O3 -Dunix zpaqfranz.cpp  -pthread -o zpaqfranz

Solaris 11.4 gcc 7.3.0
OmniOS r151042 gcc 7.5.0
Beware: -DSOLARIS and some different linking options
g++ -O3 -march=native -DSOLARIS zpaqfranz.cpp -o zpaqfranz  -pthread -static-libgcc -lkstat

MacOS 11.0 gcc (clang) 12.0.5, INTEL
MacOS 12.6 gcc (clang) 13.1.6, INTEL
Please note:
The -std=c++11 is required, otherwise you have to change half a dozen lines (or -DANCIENT). 
No -static here
"Apple does not support statically linked binaries on Mac OS X. 
(...) Rather, we strive to ensure binary 
compatibility in each dynamically linked system library and framework
(AHAHAHAHAHAH, note by me)
Warning: Shipping a statically linked binary entails a significant compatibility risk. 
We strongly recommend that you not do this..."
Short version: Apple does not like -static, so compile with
g++ -Dunix  -O3 -march=native zpaqfranz.cpp -o zpaqfranz -pthread  -std=c++11

Mac PowerPC with gcc4.x
Look at -DBIG (for BIG ENDIAN) and -DANCIENT (old-compiler)
g++ -O3 -DBIG -DANCIENT -Dunix -DNOJIT zpaqfranz.cpp -o zpaqfranz -pthread

Apple Macintosh (M1/M2)
Untested (yet), should be
g++ -Dunix  -O3 -DNOJIT zpaqfranz.cpp -o zpaqfranz -pthread  -std=c++11

ESXi (gcc 3.4.6)
Note: not fully developed ( extract() with minimum RAM need to be implemented )
g++ -O3 -DESX zpaqfranz.cpp -o zpaqfranz6  -pthread -static -s
fcorbelli commented 1 year ago

If you want please leave a review on
https://sourceforge.net/projects/zpaqfranz/
and put a star on github (if you haven't already). Any comment or suggestion is welcome
Thanks for collaboration

Erol-2022 commented 1 year ago

Hello Mr. Corbelli,

Thanks, You have a new star in your GitHub account. I will post a review to the SourceForge after creating an account.

Here is my latest report. Compiling the source code with Msys2 :

 wget https://github.com/fcorbelli/zpaqfranz/archive/refs/tags/55.16.tar.gz
 tar xvf 55.16.tar.gz
 cd WINDOWS/
 wget http://www.francocorbelli.it/zpaqfranz.cpp
 gcc --version
 gcc.exe (Rev6, Built by MSYS2 project) 12.2.0

 g++ -O3  zpaqfranz.cpp -o zpaqfranz -pthread -static

 # ls -l zpaqfranz.exe
 -rwxr-xr-x 1 Erol None 4769446 Nov 29 20:36 zpaqfranz.exe

 # ldd zpaqfranz.exe
        ntdll.dll => /c/Windows/SYSTEM32/ntdll.dll (0x776f0000)
        kernel32.dll => /c/Windows/system32/kernel32.dll (0x775d0000)
        KERNELBASE.dll => /c/Windows/system32/KERNELBASE.dll (0x7fefd260000)
        ADVAPI32.dll => /c/Windows/system32/ADVAPI32.dll (0x7feff900000)
        msvcrt.dll => /c/Windows/system32/msvcrt.dll (0x7feff750000)
        sechost.dll => /c/Windows/SYSTEM32/sechost.dll (0x7fefd6a0000)
        RPCRT4.dll => /c/Windows/system32/RPCRT4.dll (0x7fefee60000)
        SHELL32.dll => /c/Windows/system32/SHELL32.dll (0x7fefd6c0000)
        SHLWAPI.dll => /c/Windows/system32/SHLWAPI.dll (0x7fefedd0000)
        GDI32.dll => /c/Windows/system32/GDI32.dll (0x7feff170000)
        USER32.dll => /c/Windows/system32/USER32.dll (0x774d0000)
        LPK.dll => /c/Windows/system32/LPK.dll (0x7fefee50000)
        USP10.dll => /c/Windows/system32/USP10.dll (0x7fefea60000)

The static linking eliminates the extra dependencies like the new universal C run-time library.

After booting off from a Windows PE environment designed for Windows 10, I shrinked the root partition hosting the operating system to reduce the archiving time.

Operating system backup to Network drive ( WinPE system ) :

D:\zpaqfranz>zpaqfranz.exe a Z:\Win10.zpaq C: -image
zpaqfranz v56.2e-JIT-L archiver, SFX64 v55.1, (29 Nov 2022)
franz:-image
42165: REBUILDING IMAGE filename to image_c.img
Creating Z:/Win10.zpaq at offset 0 + 0
Adding drive image with 8 threads @ 2022-11-29 21:26:02
42743: opening drive c:
42751: image size 24.941.253.632 (23.23 GB)
38385: WARN expected                   4 getted      24.941.252.608 for image_c.img
1 +added, 0 -removed.

0 + (24.941.253.632 -> 18.471.624.704 -> 8.671.225.708) = 8.671.225.708 @ 28.14 MB/s

38271: HOUSTON expected 24.941.253.632, done 49.882.505.216, diff 24.941.251.584
38271: Corrupted source files? Lost connection? Cannot access? Media full?
38271: =>The updated .zpaq archive is almost certainly incompleted

845.344 seconds (000:14:05) (with errors)

Operating system backup to Ramdisk ( WinPE system ) :

D:\zpaqfranz>zpaqfranz.exe a R:\Win10.zpaq C: -image
zpaqfranz v56.2e-JIT-L archiver, SFX64 v55.1, (29 Nov 2022)
franz:-image
42165: REBUILDING IMAGE filename to image_c.img
Creating R:/Win10.zpaq at offset 0 + 0
Adding drive image with 8 threads @ 2022-11-29 22:09:32
42743: opening drive c:
42751: image size 24.941.253.632 (23.23 GB)
38385: WARN expected                   4 getted      24.941.252.608 for image_c.img
1 +added, 0 -removed.

0 + (24.941.253.632 -> 18.471.624.704 -> 8.671.219.249) = 8.671.219.249 @ 67.98 MB/s

38271: HOUSTON expected 24.941.253.632, done 49.882.505.216, diff 24.941.251.584
38271: Corrupted source files? Lost connection? Cannot access? Media full?
38271: =>The updated .zpaq archive is almost certainly incompleted

349.906 seconds (000:05:49) (with errors)

The final zpaq archive :

29.11.2022 22:15 8.671.219.249 Win10.zpaq

How to extract the zpaq archive after formatting the C partition?

D:\zpaqfranz>zpaqfranz.exe x R:\Win10.zpaq -to C:\ -space
zpaqfranz v56.2e-JIT-L archiver, SFX64 v55.1, (29 Nov 2022)
franz:Do not check free space/writeability
R:/Win10.zpaq:
1 versions, 1 files, 35.543 fragments, 1.113 blocks, 8.671.219.249 bytes (8.08 GB)
Extracting 24.941.252.608 bytes (23.23 GB) in 1 files (0 folders) with 8 threads

17519: access denied             C:\
17620: percorso does not exists <<C:\>>
        99.66% 00:00:00 (  23.15 GB) of (  23.23 GB)  171.77 MB/sec
********************************************************************************************************************
Something STRANGE happened. Archive seems corrupt. Media full?
WRITTEN BYTES       18.529.820.672
EXPECTED            18.529.886.208
********************************************************************************************************************
139.734 seconds (000:02:19) (with errors)

The image file cannot be opened with 7-Zip :

30.11.2022 01:09 24.047.497.216 image_c.img

I guess you need to add an option to extract directly the operating system backup to the drive C: ( or any other one. )

fcorbelli commented 1 year ago

I am getting a lot of problems with windows,unable to report correctly the size of a partition (!) Work in progress for -image
In this release I think I can fix
In the next a double buffering to speed up
Thanks for your help

fcorbelli commented 1 year ago

How to extract the zpaq archive after formatting the C partition?

Simply, you can't :) Imaging C partition is NOT (yet) something like Acronis or Macrium It is (or should be) a full-backup of C: that, in case of emergency, you need to

  1. Restore (extract from .zpaq the .img)
  2. Mount with something else (es. OSFMount https://www.osforensics.com/tools/mount-disk-images.html), then copy-and-paste your files
  3. OR open the .img with 7zip (supposing NTFS format), then extract with 7zip
  4. OR write back the image with "something" (example dd) into a virtual machine, or even the "real" HW (booting from USB key, for example)

Into the future (aka: no way :) a restore of C: 56_2e.zip The attached pre-release now have a -buffer parameter too (zpaq's default 4KB is not so good for high speed NVMes, usually 32-64KB are faster)

zpaqfranz a k:\copyofc.zpaq c: -image -buffer 1MB -key pippo
Erol-2022 commented 1 year ago

Hello Mr. Corbelli,

Thanks for your efforts, much appreciated. I will try your new release.

In order to avoid the additional step to restore a complete raw image, kindly could please bring back the -stdin option? A separate dd tool could do the job with the condition that you add an extra -stdout option to your tool.

fcorbelli commented 1 year ago

The -stdin is present, as the -dd (the embedded dd)
It is unfortunately NOT possible to make a "-stdout" option, due deduplication
Or, at least, I do not know how to do :) It is possible (the w command) only if the available RAM is bigger than the uncompressed image

fcorbelli commented 1 year ago

BTW it is impossible because the output of zpaq is NOT streamed, but random (seeked)

Basically the archives (.zpaq files) are read sequentially, from beginning to end The output instead is not sequential, a stream, but the individual files are gradually filled with data. This requires seeks and ultimately prevents stdout

Also, incidentally, explains why the extraction of complex archives on systems with magnetic disks is not as fast as SSDs (where seeks are much faster)

It is possible, in theory, to do the opposite That is to create an ordered list of blocks, then randomly read (with seek) the .zpaq archive, then write output sequentially

Very complex, very slow and, actually, I don't find it particularly interesting at the moment (actually yes, but to test the archives, rather than to extract them)

Erol-2022 commented 1 year ago

Hello Mr. Corbelli,

Thanks for your explanation, I understand that data stored in the zpaq archives is not sequential.

Here is my new report :

Booting off from WinPE and creating a new virtual drive with Microsoft's VssSampleProvider :

https://learn.microsoft.com/en-us/windows/win32/vss/vsssampleprovider-tool-and-sample

X:\Windows\System32\Tools64\vstor64>vstorcontrol.exe install
Installing Virtual Bus Driver... Succeeded

X:\Windows\System32\Tools64\vstor64>vstorcontrol create fixeddisk -newimage c:\new.img -size 524288000
Virtual Storage v10.0.0.0

Creating backing file c:\new.img ... Succeeded
Creating drive backed by c:\new.img ...
Waiting for drive initialization...............
Erasing Drive ...Succeeded
Drive     : \\.\PhysicalDrive1
Drive ID  : {B7B739C2-73EB-11ED-B1AF-9107B3883313}
Type      : Fixed Disk
Size      : 500MB
No volumes on the drive

Formatting the new drive and copying some files and directories. The virtual drive is identified with the symbol H:

Testing the new release of zpaqfranz : 

D:\zpaqfranz>zpaqfranz.exe a C:\test1.zpaq H: -image
zpaqfranz v56.2f-JIT-L (HW BLAKE3), SFX64 v55.1, (30 Nov 2022)
franz:-image
42165: REBUILDING IMAGE filename to image_h.img
42743: opening drive h:
42751: image size 521.142.272 (497.00 MB)
Creating C:/test1.zpaq at offset 0 + 0
Adding drive image with 4 threads @ 2022-12-04 19:05:39
1 +added, 0 -removed.

0 + (521.142.272 -> 56.706.598 -> 23.660.343) = 23.660.343 @ 23.56 MB/s

21.109 seconds (000:00:21) (all OK)

04.12.2022  19:06        23.660.343 test1.zpaq

Extracting the raw image file from the zpaq archive : 

D:\zpaqfranz>md C:\backup

D:\zpaqfranz>zpaqfranz.exe x C:\test1.zpaq -to C:\backup
zpaqfranz v56.2f-JIT-L (HW BLAKE3), SFX64 v55.1, (30 Nov 2022)
C:/test1.zpaq:
1 versions, 1 files, 798 fragments, 6 blocks, 23.660.343 bytes (22.56 MB)
Extracting 521.138.176 bytes (497.00 MB) in 1 files (0 folders) with 4 threads
         1.08% 00:01:00 (   5.35 MB) of ( 497.00 MB)    5.35 MB/sec

2.985 seconds (000:00:02) (all OK)

04.12.2022  22:05       521.138.176 image_h.img

With 7-Zip, I can extract all the files and directories from the image file image_h.img

Removing the virtual drive : 

X:\Windows\System32\Tools64\vstor64>vstorcontrol.exe remove \\.\PhysicalDrive1
Virtual Storage v10.0.0.0

Removing...
Drive     : \\.\PhysicalDrive1
Drive ID  : {B7B739C2-73EB-11ED-B1AF-9107B3883313}
Type      : Fixed Disk
Size      : 500MB

Thanks for the new release.