search

fcorbelli / zpaqfranz

Deduplicating archiver with encryption and paranoid-level tests. Swiss army knife for the serious backup and disaster recovery manager. Ransomware neutralizer. Win/Linux/Unix
MIT License
275 stars 25 forks source link

error with r mode (robocopy) #62

Closed ruptotus closed 1 year ago

ruptotus commented 1 year ago

Hi,

after newest version there maybe something broken with r switch? when running robocopy mode with -kill switch it is not working, only gives lot of errors on every file and make files with 0 lenght:

zpaqfranz v58.5o-JIT-GUI-L,HW BLAKE3,SHA1/2,SFX64 v55.1,(2023-07-12)
franz:-kill
*** ROBOCOPY MODE *** *** ignoring .zfs and :$DATA ***
18/07/2023 20:35:45 Scan dir <<D:/test/>>
18/07/2023 20:35:45 Scan dir <<Z:/test/>>

Master  1.059.049.893 (1009.99 MB files 9) <<D:/test/>>
--------------------------------------------------------------------------------------------------------------------

35107: HOUSTON something seems wrong: expected 685.129.712, done 0
35107: Corrupted source files? Lost connection? Cannot access? Media full?
31170: error robocoping data  <35109: COPY CORRUPTED> to Z:/test/536.40-desktop-win10-win11-64bit-international-dch-whql
35107: HOUSTON something seems wrong: expected 20.891.980, done 0
35107: Corrupted source files? Lost connection? Cannot access? Media full?
31170: error robocoping data  <35109: COPY CORRUPTED> to Z:/test/Aegisub-3.2.2-64.exe

35107: HOUSTON something seems wrong: expected 109.074.047, done 0
35107: Corrupted source files? Lost connection? Cannot access? Media full?
31170: error robocoping data  <35109: COPY CORRUPTED> to Z:/test/KopiaUI-0.13.0-win.zip

35107: HOUSTON something seems wrong: expected 79.615.280, done 0
35107: Corrupted source files? Lost connection? Cannot access? Media full?
31170: error robocoping data  <35109: COPY CORRUPTED> to Z:/test/KopiaUI-Setup-0.13.0.exe

35107: HOUSTON something seems wrong: expected 1.632.874, done 0
35107: Corrupted source files? Lost connection? Cannot access? Media full?
31170: error robocoping data  <35109: COPY CORRUPTED> to Z:/test/NVIDIA_Power_Management_v2.6.8.30_Setup_20230701.exe

35107: HOUSTON something seems wrong: expected 110.628.872, done 0
35107: Corrupted source files? Lost connection? Cannot access? Media full?
31170: error robocoping data  <35109: COPY CORRUPTED> to Z:/test/VirtualBox-7.0.8-156879-Win.exe

35107: HOUSTON something seems wrong: expected 49.495.374, done 0
35107: Corrupted source files? Lost connection? Cannot access? Media full?
31170: error robocoping data  <35109: COPY CORRUPTED> to Z:/test/ffmpeg-2023-06-19-git-1617d1a752-full_build.7z

35107: HOUSTON something seems wrong: expected 2.528.029, done 0
35107: Corrupted source files? Lost connection? Cannot access? Media full?
31170: error robocoping data  <35109: COPY CORRUPTED> to Z:/test/ipscan-win64-3.9.1.exe

35107: HOUSTON something seems wrong: expected 53.725, done 0
35107: Corrupted source files? Lost connection? Cannot access? Media full?
31170: error robocoping data  <35109: COPY CORRUPTED> to Z:/test/week+03+binary+numbering.pdf

=              0                    0 B
+              0                    0 B in      0.00s               0/sec
-              0                    0 B in      0.00s               0/sec

Robocopy time       6.06 s
Slaves getinfo      0.08 s
Written bytes 0 (0.00 B)

********************************************************************************************************************
Something STRANGE happened. Archive seems corrupt. Media full?
WRITTEN BYTES                    0
EXPECTED             1.059.049.893
********************************************************************************************************************

6.109 seconds (000:00:06) (with errors)

I tried on 2 different pc (windows 10 and 11) with local -> networkshare and local -> local (same drive)

and with -debug switch every file looks like:

From:   D:/test/ipscan-win64-3.9.1.exe
To:     Z:/test/ipscan-win64-3.9.1.exe
Sorgente     esiste  1
Sorgente     size    2.528.029
Sorgente     data    20.230.624.221.950
Destinazione esiste  0
Destinazione size    0
Destinazione data    0

NON ESISTE APPEND Z:/test/ipscan-win64-3.9.1.exe

Apro con WRITE
20277: expected bytes != written (media full?)  65.536 vs  0
20277: expected bytes != written (media full?)  65.536 vs  0
20277: expected bytes != written (media full?)  65.536 vs  0
20277: expected bytes != written (media full?)  65.536 vs  0
20277: expected bytes != written (media full?)  65.536 vs  0
20277: expected bytes != written (media full?)  65.536 vs  0
20277: expected bytes != written (media full?)  65.536 vs  0
20277: expected bytes != written (media full?)  65.536 vs  0
20277: expected bytes != written (media full?)  65.536 vs  0
20277: expected bytes != written (media full?)  65.536 vs  0
20277: expected bytes != written (media full?)  65.536 vs  0
20277: expected bytes != written (media full?)  65.536 vs  0
20277: expected bytes != written (media full?)  65.536 vs  0
20277: expected bytes != written (media full?)  65.536 vs  0
20277: expected bytes != written (media full?)  65.536 vs  0
20277: expected bytes != written (media full?)  65.536 vs  0
20277: expected bytes != written (media full?)  65.536 vs  0
20277: expected bytes != written (media full?)  65.536 vs  0
20277: expected bytes != written (media full?)  65.536 vs  0
20277: expected bytes != written (media full?)  65.536 vs  0
20277: expected bytes != written (media full?)  65.536 vs  0
20277: expected bytes != written (media full?)  65.536 vs  0
20277: expected bytes != written (media full?)  65.536 vs  0
20277: expected bytes != written (media full?)  65.536 vs  0
20277: expected bytes != written (media full?)  65.536 vs  0
20277: expected bytes != written (media full?)  65.536 vs  0
20277: expected bytes != written (media full?)  65.536 vs  0
20277: expected bytes != written (media full?)  65.536 vs  0
20277: expected bytes != written (media full?)  65.536 vs  0
20277: expected bytes != written (media full?)  65.536 vs  0
20277: expected bytes != written (media full?)  65.536 vs  0
20277: expected bytes != written (media full?)  65.536 vs  0
20277: expected bytes != written (media full?)  65.536 vs  0
20277: expected bytes != written (media full?)  65.536 vs  0
20277: expected bytes != written (media full?)  65.536 vs  0
20277: expected bytes != written (media full?)  65.536 vs  0
20277: expected bytes != written (media full?)  65.536 vs  0
20277: expected bytes != written (media full?)  65.536 vs  0
20277: expected bytes != written (media full?)  37.661 vs  0

35107: HOUSTON something seems wrong: expected 2.528.029, done 0
35107: Corrupted source files? Lost connection? Cannot access? Media full?
31170: error robocoping data  <35109: COPY CORRUPTED> to Z:/test/ipscan-win64-3.9.1.exe

PS. Also when running dry run against empty directory zpaqfranz creates all empty directory structure in destination.

fcorbelli commented 1 year ago

Yes 58.5 does have a different fwrite (myfwrite) for on the fly CRC computation I'll check tomorrow

Heavy developement underway for zpaqfranz over TCP

fcorbelli commented 1 year ago

I confirm that it is a bug. More precisely, it is due to the use of a "hidden" overload function by the original zpaq developer. Essentially there is an fwrite() function that is sometimes called as a special version, sometimes from the standard library

The attached pre-release shoud fix (note: a LOT of "strange things" inside, not a release) the behavior for robocopy

58_6a.zip

Can I add you in the thanks inside the source?

fcorbelli commented 1 year ago

Please check this release

ruptotus commented 1 year ago

Thank You for quick fix. Now it works fine (albeit slower then ex. copying through explorer but I don't know where I can ask questions about performance).

Only issue if it is issue ;-) remains that dry run is modifying "target" dir and I guess it should not modify anything in dry run:

PS D:\test> tree /F
Folder PATH listing
Volume serial number is 7CB0-E412
D:.
No subfolders exist

PS D:\test> zpaqfranz r 'C:\Program Files (x86)\Steam\steamapps\common\Last Epoch' D:\test\
zpaqfranz v58.6c-JIT-GUI-L,HW BLAKE3,SHA1/2,SFX64 v55.1,(2023-07-19)
*** ROBOCOPY MODE *** *** -kill missing: dry run *** *** ignoring .zfs and :$DATA ***
19/07/2023 21:47:52 Scan dir <<C:/Program Files (x86)/Steam/steamapps/common/Last Epoch/>>
19/07/2023 21:47:52 Scan dir <<D:/test/>>

Master  24.348.955.126 (22.68 GB files 945) <<C:/Program Files (x86)/Steam/steamapps/common/Last Epoch/>>
--------------------------------------------------------------------------------------------------------------------

FAKE: dry run!
=              0                    0 B
+            945       24.348.955.126 B in      0.05s 507.269.898.458/sec
-              0                    0 B in      0.00s               0/sec

Robocopy time       0.08 s
Slaves getinfo      0.05 s
Written bytes 0 (0.00 B)

0.109 seconds (00:00:00) (all OK)
PS D:\test> tree /F
Folder PATH listing
Volume serial number is 7CB0-E412
D:.
└───Last Epoch_Data
    ├───il2cpp_data
    │   ├───etc
    │   │   └───mono
    │   │       ├───2.0
    │   │       │   └───Browsers
    │   │       ├───4.0
    │   │       │   └───Browsers
    │   │       ├───4.5
    │   │       │   └───Browsers
    │   │       └───mconfig
    │   ├───Metadata
    │   └───Resources
    ├───Logs
    ├───Plugins
    │   └───x86_64
    ├───Resources
    └───StreamingAssets
        └───aa
            ├───AddressablesLink
            └───StandaloneWindows64
PS D:\test>

Also I found something weird about last "\" in dir name.

On my Windows 11 works both 'C:\Program Files (x86)\Western Digital\' and 'C:\Program Files (x86)\Western Digital':

PS D:\test> zpaqfranz r 'C:\Program Files (x86)\Western Digital\' D:\test\
zpaqfranz v58.6c-JIT-GUI-L,HW BLAKE3,SHA1/2,SFX64 v55.1,(2023-07-19)
*** ROBOCOPY MODE *** *** -kill missing: dry run *** *** ignoring .zfs and :$DATA ***
19/07/2023 21:51:25 Scan dir <<C:/Program Files (x86)/Western Digital/>>
19/07/2023 21:51:25 Scan dir <<D:/test/>>

Master  255.051.908 (243.24 MB files 413) <<C:/Program Files (x86)/Western Digital/>>
--------------------------------------------------------------------------------------------------------------------

FAKE: dry run!
=              0                    0 B
+            413          255.051.908 B in      0.02s  15.940.744.250/sec
-              0                    0 B in      0.00s               0/sec

Robocopy time       0.03 s
Slaves getinfo      0.01 s
Written bytes 0 (0.00 B)

0.047 seconds (00:00:00) (all OK)
PS D:\test> zpaqfranz r 'C:\Program Files (x86)\Western Digital' D:\test\
zpaqfranz v58.6c-JIT-GUI-L,HW BLAKE3,SHA1/2,SFX64 v55.1,(2023-07-19)
*** ROBOCOPY MODE *** *** -kill missing: dry run *** *** ignoring .zfs and :$DATA ***
19/07/2023 21:51:35 Scan dir <<C:/Program Files (x86)/Western Digital/>>
19/07/2023 21:51:35 Scan dir <<D:/test/>>

Master  255.051.908 (243.24 MB files 413) <<C:/Program Files (x86)/Western Digital/>>
--------------------------------------------------------------------------------------------------------------------

FAKE: dry run!
=              0                    0 B
+            413          255.051.908 B in      0.02s  15.003.053.411/sec
-              0                    0 B in      0.00s               0/sec

Robocopy time       0.03 s
Slaves getinfo      0.02 s
Written bytes 0 (0.00 B)

0.047 seconds (00:00:00) (all OK)
PS D:\test>

But on My Windows 10 only 'C:\Program Files (x86)\Steam\steamapps\common\Last Epoch' works and with last "\" not :

PS D:\test> zpaqfranz r 'C:\Program Files (x86)\Steam\steamapps\common\Last Epoch\' D:\test\
*** ROBOCOPY MODE *** *** -kill missing: dry run *** *** ignoring .zfs and :$DATA ***
40188: at least two folders needed

0.016 seconds (00:00:00) (with warnings)
PS D:\test> zpaqfranz r 'C:\Program Files (x86)\Steam\steamapps\common\Last Epoch' D:\test\
zpaqfranz v58.6c-JIT-GUI-L,HW BLAKE3,SHA1/2,SFX64 v55.1,(2023-07-19)
*** ROBOCOPY MODE *** *** -kill missing: dry run *** *** ignoring .zfs and :$DATA ***
19/07/2023 21:27:38 Scan dir <<C:/Program Files (x86)/Steam/steamapps/common/Last Epoch/>>
19/07/2023 21:27:38 Scan dir <<D:/test/>>

Master  24.348.955.126 (22.68 GB files 945) <<C:/Program Files (x86)/Steam/steamapps/common/Last Epoch/>>
--------------------------------------------------------------------------------------------------------------------

FAKE: dry run!
=              0                    0 B
+            945       24.348.955.126 B in      0.05s 507.269.898.458/sec
-              0                    0 B in      0.00s               0/sec

Robocopy time       0.08 s
Slaves getinfo      0.05 s
Written bytes 0 (0.00 B)

0.109 seconds (00:00:00) (all OK)

As for putting me in source file... please do if this is what You think is right... but I merely report things that I use to be broken ^_^

fcorbelli commented 1 year ago

(1) The dry run

Cannot reproduce Can you please attach a run with -debug ? Ad a run NOT inside the steam folder, but (for example) on Adobe one? (not everything is a file, on Windows. Or, better, there a are a lot of "things" on NTFS drives)

C:\zpaqfranz>zpaqfranz r "c:\Program Files (x86)\Adobe" z:\test
zpaqfranz v58.6b-JIT-GUI-L,HW BLAKE3,SHA1/2,SFX64 v55.1,(2023-07-19)
franz:-hw
*** ROBOCOPY MODE *** *** -kill missing: dry run *** *** ignoring .zfs and :$DATA ***
20/07/2023 12:53:25 Scan dir <<c:/Program Files (x86)/Adobe/>>
20/07/2023 12:53:26 Scan dir <<z:/test/>>

Master  2.236.395.259 (2.08 GB files 5.560) <<c:/Program Files (x86)/Adobe/>>
--------------------------------------------------------------------------------------
Dir 1 (slavez) DOES NOT EXISTS z:/test/

FAKE: dry run!
=              0                    0 B
+              0                    0 B in      0.00s               0/sec
-              0                    0 B in      0.00s               0/sec

Robocopy time       0.02 s
Slaves getinfo      0.00 s
Written bytes 0 (0.00 B)

0.141 seconds (00:00:00) (with warnings)

C:\zpaqfranz>dir z:\test
 Il volume nell'unità Z è RamDisk
 Numero di serie del volume: 66C8-3DCC

 Directory di z:\

File non trovato

C:\zpaqfranz>

The strange \ as last parameters (on Windows)

It is a Windows "strangeness" (in zpaqfranz there is a quick-and-VERY-dirty-hack for Windows console)

You cannot use \", you must use \" (escaping ) Therefore "d:\pippo\" is NOT ok This should be on Windows "d:\pippo\" barre

ruptotus commented 1 year ago

OK, I will be wary with "\"

debug with dry run:

PS D:\> zpaqfranz r 'C:\Program Files (x86)\Adobe\' D:\test\ -debug
48149: array franz flag size 65
48150: -715                   0  <<Runs just about like zpaq 7.15>>
48150: -append                0  <<Append-only (antiransomware, slow)>>
48150: -backupxxh3            0  <<Use XXH3 in backup instead of MD5>>
48150: -big                   0  <<Big>>
48150: -checksum              0  <<Do checksums>>
48150: -checktxt              0  <<Checktxt (MD5)>>
48150: -comment               0  <<Comment version>>
48150: -debug                 1  <<Activate debug mode>>
48150: -debug -zero           0  <<Add files but zero-filled (debugging)>>
48150: -debug -zero -kill     0  <<Add 0-byte long file (debugging)>>
48150: -desc                  0  <<Orderby desc>>
48150: -fasttxt               0  <<Create test .txt with CRC-32 and QUICK>>
48150: -filelist              0  <<Create a filelist .txt>>
48150: -findzpaq              0  <<Search .zpaq in every drive letter (USB device)>>
48150: -fix255                0  <<Fix 255>>
48150: -fixcase               0  <<Fix CAse>>
48150: -fixeml                0  <<Fix eml filenames>>
48150: -fixreserverd          0  <<fixreserverd>>
48150: -flat                  0  <<Flat filenames>>
48150: -force                 0  <<Force>>
48150: -forcewindows          0  <<Store ADS stuff                (default: NO)>>
48150: -forcezfs              0  <<Enforce using .zfs>>
48150: -frugal                0  <<Frugal backup>>
48150: -hashdeep              0  <<Hashdeep>>
48150: -hw                    0  <<Use HW SHA1>>
48150: -image                 0  <<Drive image>>
48150: -kill                  0  <<Kill>>
48150: -longpath              0  <<Longpath>>
48150: -mm                    0  <<Memory mapped>>
48150: -noattributes          0  <<Nessun attributo>>
48150: -nodedup               0  <<Turn off deduplicator>>
48150: -noeta                 0  <<Do not show ETA>>
48150: -nopath                0  <<Do not store path>>
48150: -noqnap                0  <<Exclude QNAP snap & trash>>
48150: -norecursion           0  <<Do not recurse into folders (default: YES)>>
48150: -nosort                0  <<Do not sort file>>
48150: -open                  0  <<open>>
48150: -pakka                 0  <<New output>>
48150: -paranoid              0  <<Paranoid>>
48150: -quiet                 0  <<Do not show filesystem errors>>
48150: -ramdisk               0  <<ramdisk>>
48150: -rename                0  <<Rename>>
48150: -sfxall                0  <<Sfx all>>
48150: -sfxforce              0  <<Sfx force>>
48150: -silent                0  <<Silent>>
48150: -space                 0  <<Do not check space/writeability>>
48150: -ssd                   0  <<SSD>>
48150: -stat                  0  <<Statistics>>
48150: -stdin                 0  <<stdin>>
48150: -stdout                0  <<stdout>>
48150: -store                 0  <<Store mode: no deduplication, no compression>>
48150: -tar                   0  <<TAR mode (store posix)>>
48150: -test                  0  <<Only do test>>
48150: -touch                 0  <<Force 'touch' on date (7.15 to zpaqfranz)>>
48150: -utc                   0  <<Use UTC time>>
48150: -utf                   0  <<UTF-8>>
48150: -verbose               0  <<Verbose output>>
48150: -verify                0  <<Verify>>
48150: -vss                   0  <<Enable Volume Shadow Copies>>
48150: -xls                   0  <<Do NOT force adding of XLS/PPT (default: NO)>>
48150: -zero                  0  <<Flag zero>>
48150: -zfs                   0  <<Do NOT ignore .zfs             (default: YES)>>
48150: /od                    0  <<Order by date>>
48150: /on                    0  <<Order by name>>
48150: /os                    0  <<Order by size>>
zpaqfranz v58.6c-JIT-GUI-L,HW BLAKE3,SHA1/2,SFX64 v55.1,(2023-07-19)
FULL exename <<C:/Users/user/programy/zpaqfranz/zpaqfranz.exe>>
42993: The chosen algo 3 SHA-1
1838: new ecx -330865
1843: new ebx 10250157
SSSE3 :OK
SSE41 :OK
SHA   :NO
franz:-debug -verbose
*** ROBOCOPY MODE *** *** -kill missing: dry run *** *** ignoring .zfs and :$DATA ***
Size checkdup 2
40228: 000 files[i] C:/Program Files (x86)/Adobe/  checkdupe [i] |c:/program files (x86)/adobe/|
40228: 001 files[i] D:/test/  checkdupe [i] |d:/test/|
40245: after 000 files[i] C:/Program Files (x86)/Adobe/
40245: after 001 files[i] D:/test/
20/07/2023 14:39:33 Scan dir <<C:/Program Files (x86)/Adobe/>>
00000010 MY new t2 .
DIRECTORY;

00000011 MY new t2 ..
DIRECTORY;READONY;

00000010 MY new t2 Acrobat Reader DC
DIRECTORY;

00000010 MY new t2 .
DIRECTORY;

00000010 MY new t2 ..
DIRECTORY;

00000010 MY new t2 Reader
DIRECTORY;

00000010 MY new t2 .
DIRECTORY;

00000010 MY new t2 ..
DIRECTORY;

00000010 MY new t2 acrocef_1
DIRECTORY;

00000010 MY new t2 .
DIRECTORY;

00000010 MY new t2 ..
DIRECTORY;

00000020 MY new t2 cef.pak
ARCHIVE;

00000020 MY new t2 cef_100_percent.pak
ARCHIVE;

00000020 MY new t2 cef_200_percent.pak
ARCHIVE;

00000020 MY new t2 cef_extensions.pak
ARCHIVE;

00000020 MY new t2 chrome_elf.dll
ARCHIVE;

00000020 MY new t2 COPYING.LGPLv2.1.txt
ARCHIVE;

00000020 MY new t2 d3dcompiler_43.dll
ARCHIVE;

00000020 MY new t2 d3dcompiler_47.dll
ARCHIVE;

00000020 MY new t2 icudt.dll
ARCHIVE;

00000020 MY new t2 icudtl.dat
ARCHIVE;

00000020 MY new t2 libcef.dll
ARCHIVE;

00000020 MY new t2 libEGL.dll
ARCHIVE;

00000020 MY new t2 libGLESv2.dll
ARCHIVE;

00000020 MY new t2 LICENSE.txt
ARCHIVE;

00000010 MY new t2 locales
DIRECTORY;

00000010 MY new t2 .
DIRECTORY;

00000010 MY new t2 ..
DIRECTORY;

00000020 MY new t2 en-US.pak
ARCHIVE;

00000020 MY new t2 natives_blob.bin
ARCHIVE;

00000020 MY new t2 RdrCEF.exe
ARCHIVE;

00000020 MY new t2 RdrServicesUpdater.exe
ARCHIVE;

00000020 MY new t2 snapshot_blob.bin
ARCHIVE;

00000010 MY new t2 swiftshader
DIRECTORY;

00000010 MY new t2 .
DIRECTORY;

00000010 MY new t2 ..
DIRECTORY;

00000020 MY new t2 libEGL.dll
ARCHIVE;

00000020 MY new t2 libGLESv2.dll
ARCHIVE;

00000020 MY new t2 v8_context_snapshot.bin
ARCHIVE;

00000020 MY new t2 widevinecdmadapter.dll
ARCHIVE;

C:/Program Files (x86)/Adobe/Acrobat Reader DC/
C:/Program Files (x86)/Adobe/Acrobat Reader DC/Reader/
C:/Program Files (x86)/Adobe/Acrobat Reader DC/Reader/acrocef_1/
C:/Program Files (x86)/Adobe/Acrobat Reader DC/Reader/acrocef_1/COPYING.LGPLv2.1.txt
C:/Program Files (x86)/Adobe/Acrobat Reader DC/Reader/acrocef_1/LICENSE.txt
C:/Program Files (x86)/Adobe/Acrobat Reader DC/Reader/acrocef_1/RdrCEF.exe
C:/Program Files (x86)/Adobe/Acrobat Reader DC/Reader/acrocef_1/RdrServicesUpdater.exe
C:/Program Files (x86)/Adobe/Acrobat Reader DC/Reader/acrocef_1/cef.pak
C:/Program Files (x86)/Adobe/Acrobat Reader DC/Reader/acrocef_1/cef_100_percent.pak
C:/Program Files (x86)/Adobe/Acrobat Reader DC/Reader/acrocef_1/cef_200_percent.pak
C:/Program Files (x86)/Adobe/Acrobat Reader DC/Reader/acrocef_1/cef_extensions.pak
C:/Program Files (x86)/Adobe/Acrobat Reader DC/Reader/acrocef_1/chrome_elf.dll
C:/Program Files (x86)/Adobe/Acrobat Reader DC/Reader/acrocef_1/d3dcompiler_43.dll
C:/Program Files (x86)/Adobe/Acrobat Reader DC/Reader/acrocef_1/d3dcompiler_47.dll
C:/Program Files (x86)/Adobe/Acrobat Reader DC/Reader/acrocef_1/icudt.dll
C:/Program Files (x86)/Adobe/Acrobat Reader DC/Reader/acrocef_1/icudtl.dat
C:/Program Files (x86)/Adobe/Acrobat Reader DC/Reader/acrocef_1/libEGL.dll
C:/Program Files (x86)/Adobe/Acrobat Reader DC/Reader/acrocef_1/libGLESv2.dll
C:/Program Files (x86)/Adobe/Acrobat Reader DC/Reader/acrocef_1/libcef.dll
C:/Program Files (x86)/Adobe/Acrobat Reader DC/Reader/acrocef_1/locales/
C:/Program Files (x86)/Adobe/Acrobat Reader DC/Reader/acrocef_1/locales/en-US.pak
C:/Program Files (x86)/Adobe/Acrobat Reader DC/Reader/acrocef_1/natives_blob.bin
C:/Program Files (x86)/Adobe/Acrobat Reader DC/Reader/acrocef_1/snapshot_blob.bin
C:/Program Files (x86)/Adobe/Acrobat Reader DC/Reader/acrocef_1/swiftshader/
C:/Program Files (x86)/Adobe/Acrobat Reader DC/Reader/acrocef_1/swiftshader/libEGL.dll
C:/Program Files (x86)/Adobe/Acrobat Reader DC/Reader/acrocef_1/swiftshader/libGLESv2.dll
C:/Program Files (x86)/Adobe/Acrobat Reader DC/Reader/acrocef_1/v8_context_snapshot.bin
C:/Program Files (x86)/Adobe/Acrobat Reader DC/Reader/acrocef_1/widevinecdmadapter.dll
20/07/2023 14:39:33 Scan dir <<D:/test/>>
00000010 MY new t2 .
DIRECTORY;

00000016 MY new t2 ..
DIRECTORY;HIDDEN;SYSTEM;

Master  185.509.910 (176.92 MB files 23) <<C:/Program Files (x86)/Adobe/>>
--------------------------------------------------------------------------------------------------------------------

From:   C:/Program Files (x86)/Adobe/Acrobat Reader DC/
To:     D:/test/Acrobat Reader DC/

From:   C:/Program Files (x86)/Adobe/Acrobat Reader DC/Reader/
To:     D:/test/Acrobat Reader DC/Reader/

From:   C:/Program Files (x86)/Adobe/Acrobat Reader DC/Reader/acrocef_1/
To:     D:/test/Acrobat Reader DC/Reader/acrocef_1/

From:   C:/Program Files (x86)/Adobe/Acrobat Reader DC/Reader/acrocef_1/COPYING.LGPLv2.1.txt
To:     D:/test/Acrobat Reader DC/Reader/acrocef_1/COPYING.LGPLv2.1.txt
Sorgente     esiste  1
Sorgente     size    27.028
Sorgente     data    20.180.919.223.222
Destinazione esiste  0
Destinazione size    0
Destinazione data    0

From:   C:/Program Files (x86)/Adobe/Acrobat Reader DC/Reader/acrocef_1/LICENSE.txt
To:     D:/test/Acrobat Reader DC/Reader/acrocef_1/LICENSE.txt
Sorgente     esiste  1
Sorgente     size    1.691
Sorgente     data    20.211.005.004.444
Destinazione esiste  0
Destinazione size    0
Destinazione data    0

From:   C:/Program Files (x86)/Adobe/Acrobat Reader DC/Reader/acrocef_1/RdrCEF.exe
To:     D:/test/Acrobat Reader DC/Reader/acrocef_1/RdrCEF.exe
Sorgente     esiste  1
Sorgente     size    6.421.216
Sorgente     data    20.220.302.205.622
Destinazione esiste  0
Destinazione size    0
Destinazione data    0

From:   C:/Program Files (x86)/Adobe/Acrobat Reader DC/Reader/acrocef_1/RdrServicesUpdater.exe
To:     D:/test/Acrobat Reader DC/Reader/acrocef_1/RdrServicesUpdater.exe
Sorgente     esiste  1
Sorgente     size    28.402.400
Sorgente     data    20.220.302.205.622
Destinazione esiste  0
Destinazione size    0
Destinazione data    0

From:   C:/Program Files (x86)/Adobe/Acrobat Reader DC/Reader/acrocef_1/cef.pak
To:     D:/test/Acrobat Reader DC/Reader/acrocef_1/cef.pak
Sorgente     esiste  1
Sorgente     size    1.970.714
Sorgente     data    20.211.224.100.322
Destinazione esiste  0
Destinazione size    0
Destinazione data    0

From:   C:/Program Files (x86)/Adobe/Acrobat Reader DC/Reader/acrocef_1/cef_100_percent.pak
To:     D:/test/Acrobat Reader DC/Reader/acrocef_1/cef_100_percent.pak
Sorgente     esiste  1
Sorgente     size    214.198
Sorgente     data    20.211.224.100.322
Destinazione esiste  0
Destinazione size    0
Destinazione data    0

From:   C:/Program Files (x86)/Adobe/Acrobat Reader DC/Reader/acrocef_1/cef_200_percent.pak
To:     D:/test/Acrobat Reader DC/Reader/acrocef_1/cef_200_percent.pak
Sorgente     esiste  1
Sorgente     size    290.080
Sorgente     data    20.211.224.100.322
Destinazione esiste  0
Destinazione size    0
Destinazione data    0

From:   C:/Program Files (x86)/Adobe/Acrobat Reader DC/Reader/acrocef_1/cef_extensions.pak
To:     D:/test/Acrobat Reader DC/Reader/acrocef_1/cef_extensions.pak
Sorgente     esiste  1
Sorgente     size    1.313.312
Sorgente     data    20.211.224.100.322
Destinazione esiste  0
Destinazione size    0
Destinazione data    0

From:   C:/Program Files (x86)/Adobe/Acrobat Reader DC/Reader/acrocef_1/chrome_elf.dll
To:     D:/test/Acrobat Reader DC/Reader/acrocef_1/chrome_elf.dll
Sorgente     esiste  1
Sorgente     size    858.848
Sorgente     data    20.220.302.205.622
Destinazione esiste  0
Destinazione size    0
Destinazione data    0

From:   C:/Program Files (x86)/Adobe/Acrobat Reader DC/Reader/acrocef_1/d3dcompiler_43.dll
To:     D:/test/Acrobat Reader DC/Reader/acrocef_1/d3dcompiler_43.dll
Sorgente     esiste  1
Sorgente     size    12.288
Sorgente     data    20.211.005.004.444
Destinazione esiste  0
Destinazione size    0
Destinazione data    0

From:   C:/Program Files (x86)/Adobe/Acrobat Reader DC/Reader/acrocef_1/d3dcompiler_47.dll
To:     D:/test/Acrobat Reader DC/Reader/acrocef_1/d3dcompiler_47.dll
Sorgente     esiste  1
Sorgente     size    3.657.992
Sorgente     data    20.220.302.205.622
Destinazione esiste  0
Destinazione size    0
Destinazione data    0

From:   C:/Program Files (x86)/Adobe/Acrobat Reader DC/Reader/acrocef_1/icudt.dll
To:     D:/test/Acrobat Reader DC/Reader/acrocef_1/icudt.dll
Sorgente     esiste  1
Sorgente     size    11.776
Sorgente     data    20.220.302.205.622
Destinazione esiste  0
Destinazione size    0
Destinazione data    0

From:   C:/Program Files (x86)/Adobe/Acrobat Reader DC/Reader/acrocef_1/icudtl.dat
To:     D:/test/Acrobat Reader DC/Reader/acrocef_1/icudtl.dat
Sorgente     esiste  1
Sorgente     size    10.413.552
Sorgente     data    20.211.005.084.444
Destinazione esiste  0
Destinazione size    0
Destinazione data    0

From:   C:/Program Files (x86)/Adobe/Acrobat Reader DC/Reader/acrocef_1/libEGL.dll
To:     D:/test/Acrobat Reader DC/Reader/acrocef_1/libEGL.dll
Sorgente     esiste  1
Sorgente     size    336.096
Sorgente     data    20.220.302.205.622
Destinazione esiste  0
Destinazione size    0
Destinazione data    0

From:   C:/Program Files (x86)/Adobe/Acrobat Reader DC/Reader/acrocef_1/libGLESv2.dll
To:     D:/test/Acrobat Reader DC/Reader/acrocef_1/libGLESv2.dll
Sorgente     esiste  1
Sorgente     size    5.694.176
Sorgente     data    20.220.302.205.622
Destinazione esiste  0
Destinazione size    0
Destinazione data    0

From:   C:/Program Files (x86)/Adobe/Acrobat Reader DC/Reader/acrocef_1/libcef.dll
To:     D:/test/Acrobat Reader DC/Reader/acrocef_1/libcef.dll
Sorgente     esiste  1
Sorgente     size    122.612.960
Sorgente     data    20.220.302.205.622
Destinazione esiste  0
Destinazione size    0
Destinazione data    0

From:   C:/Program Files (x86)/Adobe/Acrobat Reader DC/Reader/acrocef_1/locales/
To:     D:/test/Acrobat Reader DC/Reader/acrocef_1/locales/

From:   C:/Program Files (x86)/Adobe/Acrobat Reader DC/Reader/acrocef_1/locales/en-US.pak
To:     D:/test/Acrobat Reader DC/Reader/acrocef_1/locales/en-US.pak
Sorgente     esiste  1
Sorgente     size    251.246
Sorgente     data    20.211.224.100.324
Destinazione esiste  0
Destinazione size    0
Destinazione data    0

From:   C:/Program Files (x86)/Adobe/Acrobat Reader DC/Reader/acrocef_1/natives_blob.bin
To:     D:/test/Acrobat Reader DC/Reader/acrocef_1/natives_blob.bin
Sorgente     esiste  1
Sorgente     size    0
Sorgente     data    20.180.919.223.222
Destinazione esiste  0
Destinazione size    0
Destinazione data    0

From:   C:/Program Files (x86)/Adobe/Acrobat Reader DC/Reader/acrocef_1/snapshot_blob.bin
To:     D:/test/Acrobat Reader DC/Reader/acrocef_1/snapshot_blob.bin
Sorgente     esiste  1
Sorgente     size    47.375
Sorgente     data    20.211.005.004.444
Destinazione esiste  0
Destinazione size    0
Destinazione data    0

From:   C:/Program Files (x86)/Adobe/Acrobat Reader DC/Reader/acrocef_1/swiftshader/
To:     D:/test/Acrobat Reader DC/Reader/acrocef_1/swiftshader/

From:   C:/Program Files (x86)/Adobe/Acrobat Reader DC/Reader/acrocef_1/swiftshader/libEGL.dll
To:     D:/test/Acrobat Reader DC/Reader/acrocef_1/swiftshader/libEGL.dll
Sorgente     esiste  1
Sorgente     size    353.504
Sorgente     data    20.211.224.100.324
Destinazione esiste  0
Destinazione size    0
Destinazione data    0

From:   C:/Program Files (x86)/Adobe/Acrobat Reader DC/Reader/acrocef_1/swiftshader/libGLESv2.dll
To:     D:/test/Acrobat Reader DC/Reader/acrocef_1/swiftshader/libGLESv2.dll
Sorgente     esiste  1
Sorgente     size    2.444.000
Sorgente     data    20.211.224.100.324
Destinazione esiste  0
Destinazione size    0
Destinazione data    0

From:   C:/Program Files (x86)/Adobe/Acrobat Reader DC/Reader/acrocef_1/v8_context_snapshot.bin
To:     D:/test/Acrobat Reader DC/Reader/acrocef_1/v8_context_snapshot.bin
Sorgente     esiste  1
Sorgente     size    163.682
Sorgente     data    20.211.005.004.444
Destinazione esiste  0
Destinazione size    0
Destinazione data    0

From:   C:/Program Files (x86)/Adobe/Acrobat Reader DC/Reader/acrocef_1/widevinecdmadapter.dll
To:     D:/test/Acrobat Reader DC/Reader/acrocef_1/widevinecdmadapter.dll
Sorgente     esiste  1
Sorgente     size    11.776
Sorgente     data    20.220.302.205.622
Destinazione esiste  0
Destinazione size    0
Destinazione data    0

FAKE: dry run!
=              0                    0 B
+             23          185.509.910 B in      0.28s     657.836.560/sec
-              0                    0 B in      0.00s               0/sec

Robocopy time       0.34 s
Slaves getinfo      0.00 s
Written bytes 0 (0.00 B)

0.594 seconds (00:00:00) (all OK)
33040: call xcommand with a different errorcode (not 1, not 2) 0

Maybe points where are only "From" and "To" without "Sorgente" and "Destinazione" is the point where folders are created?

ruptotus commented 1 year ago

And this is how it looks in process monitor:

Procmon64_rrJxccbRXV

Procmon64_a2jomO3ok7

Procmon64_OeeyvPYvbJ

fcorbelli commented 1 year ago

OK, I think I get the point

does the output folder exists, before your dry-run?

fcorbelli commented 1 year ago

Definitely a bug, due to the assumption that a global variable would change the behavior of a function, but it does not.

Please try this attached pre-release

The problem happened if the target folder exists. If it did not exist, the procedure would not operate, but it is a case to be considered.

Should be fixed now Added 2 hits in "thanks"

fcorbelli commented 1 year ago

Regarding the performance issue, it is good to clarify what the robocopy command is for (since Microsoft's robocopy already exists) It has 6 uses for 6 different scenarios (!)

  1. You work with ancient .XLS (and .PPT) files. Robocopy does not handle this kind of files, because internal metadata can be changed without difference in external one. Therefore, if you are unlucky, a post-binary-based compare will fail. Yep, ancient fileservers with .XLS does exists even in 2023
  2. You want to double-check everything, with -verify -checksum -xxh3 and maybe even -ssd too (if source and destination folders runs on different filesystems, ex. local drive/NAS, or local drive/USB). Robocopy does not run hash-based comparing
  3. One source folder, multiple destinations. In this case you want to syncronize a folder with TWO (or more) NASes. Robocopy works 1:1, not 1:many
  4. You aren't on Windows, but on Linux or BSD or whatever. Robocopy does not exists here
  5. You are dealing with .zpaq archives, and the -append switch. When syncronizing huge files, from one position to another, robocopy will start again every time, running from 0. On .zpaq files, with robocopy -append only the changed portion will be sended, with a HUGE speed improvement. Of course you must EXACTLY know what you are doing. You want to send a local 501GB zpaq to a NAS, keeping the older 500GB version. zpaqfranz will copy only 1GB
  6. Source and destinations are on different media, like different NASes. In this case using -ssd you will create as many thread as possible, splitting the overload (aka: enumerating files) in parallel. Only newer robocopy versions allow something like that (but complex and fragile)

I think you can guess how it's a tool usually specific to the non-Windows server world (the one I mainly deal with) I am practically never concerned with performance, meaning read and write speed, at the first point there is verification and at the second exploitation of inherently parallel scenarios and finally sending the minimum amount of data Then again robocopy already exists, and it definitely works well, I am not going to reinvent the wheel, but make a better wheel Or at least... try to :)

ruptotus commented 1 year ago

Yes the target dir d:\test exist. After fix is all OK. Thanks.

And about Your "performance" comment I absolutely 100% agree with You ^_^. Option 2 I used last week to check when I had to copy production to test server and it was tremendous help. And I did not known about option 5 so will be testing soon as right now I connect with VPN to remote server start FTP there and use filezilla to resume download to copy only that last "1GB" and using zpaqfranz over network share might be simpler.

My comment about performance was about scenario when I use Windows Robocopy or explorer copy the speed is around 115MB/s... And with zpaqfranz r is around 25MB/s (and CPU is at 10%, RAM at 50% and disk read at 25MB/s and utilization at 8%... so I was curious what is bottleneck there... is it some internal things in zpaqfranz. Or maybe zpaqfranz use some CPU function that is not reflected in task manager CPU measurement and in reality this component in CPU is at 100%?).

Simply put, sometimes I just want to put some movie episodes on NAS and the slower speed the longer wait. But perhaps I should use other tool if it work better for that job and not force my self to use zpaqfranz in every scenario ^_^

fcorbelli commented 1 year ago

On speed issue This is strange, indeed, I'll try some test Usually real world speed is about 90% of robocopy, due to not optimized at all use of internal buffer I'll make a profiler-enabled build, just to check with smaller granularity

fcorbelli commented 1 year ago

Please test the attached pre-release against "real" robocopy on your machine

58_7d.zip

Thanks

ruptotus commented 1 year ago

Thank You... now it is A LOT better ^_^:

Zrzut ekranu (11)

For comparison Windows Robocopy:

Zrzut ekranu (10)

fcorbelli commented 1 year ago

Please put the output of a -verbose run

ruptotus commented 1 year ago

Here:

PS C:\Users\akhkharu\Desktop> .\zpaqfranz.exe r 'C:\Program Files (x86)\Diablo IV' Z:\test\ -kill -verbose
zpaqfranz v58.7d-JIT-GUI-L,HW BLAKE3,SHA1/2,SFX64 v55.1,(2023-07-21)
franz:-kill -verbose
*** ROBOCOPY MODE *** *** ignoring .zfs and :$DATA ***
22/07/2023 21:12:20 Scan dir <<C:/Program Files (x86)/Diablo IV/>>
22/07/2023 21:12:20 Scan dir <<Z:/test/>>

Master  48.334.767.672 (45.01 GB files 608) <<C:/Program Files (x86)/Diablo IV/>>
--------------------------------------------------------------------------------------------------------------------
005% 00:11:39 (   2.26 GB) of (  45.01 GB)           67.307.322/sEc
015% 00:09:23 (   7.14 GB) of (  45.01 GB)           72.285.279/sEc
020% 00:08:49 (   9.03 GB) of (  45.01 GB)           73.419.719/sEc
030% 00:07:31 (  13.88 GB) of (  45.01 GB)           74.160.648/sEc
035% 00:07:01 (  15.84 GB) of (  45.01 GB)           74.622.278/sEc
045% 00:05:56 (  20.32 GB) of (  45.01 GB)           74.450.424/sEc
055% 00:04:50 (  24.88 GB) of (  45.01 GB)           74.610.463/sEc
060% 00:04:13 (  27.43 GB) of (  45.01 GB)           74.371.645/sEc
070% 00:03:13 (  31.63 GB) of (  45.01 GB)           74.308.875/sEc
075% 00:02:38 (  34.08 GB) of (  45.01 GB)           74.082.739/sEc
085% 00:01:33 (  38.60 GB) of (  45.01 GB)           73.871.699/sEc
090% 00:01:04 (  40.58 GB) of (  45.01 GB)           73.844.625/sEc

=              0                    0 B
+            608       48.334.767.672 B in    683.83s      70.682.432/sec
-              0                    0 B in      0.00s               0/sec

Robocopy time     684.33  - Slaves getinfo     57.83 s
Written bytes 48.334.767.672 (45.01 GB) @ 70.630.891 B/sec
====================================================================================================================
buffersize                1.048.576 scandir                       32
check_source                      0 check_dest                57.827
makepath                          0 makepath2                    358
isequal                           0 close                          0
close2                            0 touch                      3.406
delete                            0 readopen                      94
openoutfile                   3.827 fclose                     2.989
copy (total)                615.126 @ 78.577.019 (write 599.566)
====================================================================================================================

684.406 seconds (000:11:24) (all OK)
fcorbelli commented 1 year ago

I'll try to make it faster, with a specific-Windows-only function Work in progress...

fcorbelli commented 1 year ago

Please post the output (timing) of 1) robocopy ... /mir /r:0 /w:0 /nfl /ndl 2) zpaqfranz r ... -verbose -noeta 3) zpaqfranz r ... -verbose -noeta -buffer 512KB 4) zpaqfranz r ... -verbose -noeta -buffer 64KB 5) zpaqfranz r ... -verbose -noeta -big

Possibly with loaded and unloaded cache

58_7f.zip

The -big (on Windows) runs a new method (not tested at all, just to see how quick in your scenario)

It seems that the slowdown is more related to access management on network share (aka: Windows running slow on standard file I/O), and the mixing of files (few giant files vs many small files)

I have rather fast hardware at home, so I struggle to simulate an environment with a gigabit network and what appears to be a NAS

Thank you!

ruptotus commented 1 year ago

Well, yes, sorry my hardware is rather old, cheap :P

My PC is from 2015 (only system disk new) and my "NAS" is cheap BEELINK minipc with void linux with attached 4TB HDD (all ZFS) through USB 3.0 and shared with SAMBA to Windows.

Here test from PC to "NAS"

robocopy 'C:\Program Files (x86)\Diablo IV\' Z:\test\ /mir /r:0 /w:0 /nfl /ndl

robocopy 'C:\Program Files (x86)\Diablo IV\' Z:\test\ /mir /r:0 /w:0 /nfl /ndl

-------------------------------------------------------------------------------
   ROBOCOPY     ::     Robust File Copy for Windows
-------------------------------------------------------------------------------

  Started : niedziela, 23 lipca 2023 10:56:53
   Source : C:\Program Files (x86)\Diablo IV\
     Dest : Z:\test\

    Files : *.*

  Options : *.* /NDL /NFL /S /E /DCOPY:DA /COPY:DAT /PURGE /MIR /R:0 /W:0

------------------------------------------------------------------------------

------------------------------------------------------------------------------

               Total    Copied   Skipped  Mismatch    FAILED    Extras
    Dirs :        46        45         1         0         0         0
   Files :       608       608         0         0         0         0
   Bytes :  45.025 g  45.025 g         0         0         0         0
   Times :   0:07:25   0:07:22                       0:00:00   0:00:02

   Speed :           109177989 Bytes/sec.
   Speed :            6247.214 MegaBytes/min.
   Ended : niedziela, 23 lipca 2023 11:04:19

.\zpaqfranz.exe r 'C:\Program Files (x86)\Diablo IV' Z:\test\ -kill -verbose -noeta

.\zpaqfranz.exe r 'C:\Program Files (x86)\Diablo IV' Z:\test\ -kill -verbose -noeta
zpaqfranz v58.7f-JIT-GUI-L,HW BLAKE3,SHA1/2,SFX64 v55.1,(2023-07-23)
franz:-kill -noeta -verbose
*** ROBOCOPY MODE *** *** ignoring .zfs and :$DATA ***
23/07/2023 11:06:34 Scan dir <<C:/Program Files (x86)/Diablo IV/>>
23/07/2023 11:06:34 Scan dir <<Z:/test/>>

Master  48.345.760.542 (45.03 GB files 608) <<C:/Program Files (x86)/Diablo IV/>>
--------------------------------------------------------------------------------------------------------------------

+            608       48.345.760.542 B in    683.88s      70.693.855/sec

Robocopy time     684.56  - Slaves getinfo     53.19 s
Written bytes 48.345.760.542 (45.03 GB) @ 70.622.703 B/sec
====================================================================================================================
check_source          0 check_dest     53.194 makepath          0
makepath2           202 isequal             0 close             0
close2                0 touch           3.753 delete            0
readopen            342 openout         3.342 fclose        3.403
buffersize      1.00 MB scandir            31 copy        618.989 @ 78.104.393/s

684.640 seconds (000:11:24) (all OK)

.\zpaqfranz.exe r 'C:\Program Files (x86)\Diablo IV' Z:\test\ -kill -verbose -noeta -buffer 512KB

.\zpaqfranz.exe r 'C:\Program Files (x86)\Diablo IV' Z:\test\ -kill -verbose -noeta -buffer 512KB
zpaqfranz v58.7f-JIT-GUI-L,HW BLAKE3,SHA1/2,SFX64 v55.1,(2023-07-23)
franz:-buffer                             524.288
franz:-kill -noeta -verbose
*** ROBOCOPY MODE *** *** ignoring .zfs and :$DATA ***
23/07/2023 11:19:17 Scan dir <<C:/Program Files (x86)/Diablo IV/>>
23/07/2023 11:19:17 Scan dir <<Z:/test/>>

Master  48.345.760.542 (45.03 GB files 608) <<C:/Program Files (x86)/Diablo IV/>>
--------------------------------------------------------------------------------------------------------------------

+            608       48.345.760.542 B in    747.78s      64.652.298/sec

Robocopy time     748.38  - Slaves getinfo     60.80 s
Written bytes 48.345.760.542 (45.03 GB) @ 64.600.896 B/sec
====================================================================================================================
check_source          0 check_dest     60.798 makepath          0
makepath2           248 isequal             0 close             0
close2                0 touch           3.537 delete            0
readopen             46 openout         4.446 fclose        2.953
buffersize    512.00 KB scandir            31 copy        675.722 @ 71.546.820/s

748.453 seconds (000:12:28) (all OK)

.\zpaqfranz.exe r 'C:\Program Files (x86)\Diablo IV' Z:\test\ -kill -verbose -noeta -buffer 64KB

.\zpaqfranz.exe r 'C:\Program Files (x86)\Diablo IV' Z:\test\ -kill -verbose -noeta -buffer 64KB
zpaqfranz v58.7f-JIT-GUI-L,HW BLAKE3,SHA1/2,SFX64 v55.1,(2023-07-23)
franz:-buffer                              65.536
franz:-kill -noeta -verbose
*** ROBOCOPY MODE *** *** ignoring .zfs and :$DATA ***
23/07/2023 11:33:19 Scan dir <<C:/Program Files (x86)/Diablo IV/>>
23/07/2023 11:33:19 Scan dir <<Z:/test/>>

Master  48.345.760.542 (45.03 GB files 608) <<C:/Program Files (x86)/Diablo IV/>>
--------------------------------------------------------------------------------------------------------------------

+            608       48.345.760.542 B in   1818.95s      26.578.938/sec

Robocopy time    1819.58  - Slaves getinfo     50.38 s
Written bytes 48.345.760.542 (45.03 GB) @ 26.569.736 B/sec
====================================================================================================================
check_source          0 check_dest     50.385 makepath          0
makepath2           252 isequal             0 close             0
close2                0 touch           3.813 delete            0
readopen             62 openout         4.362 fclose        2.922
buffersize     64.00 KB scandir            31 copy      1.757.155 @ 27.513.657/s

1819.656 seconds (000:30:19) (all OK)

.\zpaqfranz.exe r 'C:\Program Files (x86)\Diablo IV' Z:\test\ -kill -verbose -noeta -big

.\zpaqfranz.exe r 'C:\Program Files (x86)\Diablo IV' Z:\test\ -kill -verbose -noeta -big
zpaqfranz v58.7f-JIT-GUI-L,HW BLAKE3,SHA1/2,SFX64 v55.1,(2023-07-23)
franz:-big -kill -noeta -verbose
*** ROBOCOPY MODE *** *** ignoring .zfs and :$DATA ***
23/07/2023 12:05:02 Scan dir <<C:/Program Files (x86)/Diablo IV/>>
23/07/2023 12:05:02 Scan dir <<Z:/test/>>

Master  48.345.760.542 (45.03 GB files 608) <<C:/Program Files (x86)/Diablo IV/>>
--------------------------------------------------------------------------------------------------------------------

+            608       48.345.760.542 B in    492.12s      98.239.380/sec

Robocopy time     492.64  - Slaves getinfo     61.37 s
Written bytes 48.345.760.542 (45.03 GB) @ 98.135.685 B/sec
====================================================================================================================
check_source          0 check_dest     61.371 makepath          0
makepath2             0 isequal             0 close             0
close2                0 touch           3.291 delete            0
readopen              0 openout             0 fclose            0
buffersize      1.00 MB scandir            31 copy        427.460 @ 113.100.080/s

492.750 seconds (000:08:12) (all OK)

####### #    #
#     # #   #
#     # #  #
#     # ###!
#     # #  #
#     # #   #
####### #    #

"loaded and unloaded cache"... Sorry, don't know what to do to load/unload cache :P

ruptotus commented 1 year ago

And here is test from internal SSD to other SSD.

Unfortunately second SSD is old and slow ;-( Zrzut ekranu (14)

But all test where faster then Windows robocopy:

robocopy 'C:\Program Files (x86)\Diablo IV\' K:\test\ /mir /r:0 /w:0 /nfl /ndl

-------------------------------------------------------------------------------
   ROBOCOPY     ::     Robust File Copy for Windows
-------------------------------------------------------------------------------

  Started : niedziela, 23 lipca 2023 13:04:42
   Source : C:\Program Files (x86)\Diablo IV\
     Dest : K:\test\

    Files : *.*

  Options : *.* /NDL /NFL /S /E /DCOPY:DA /COPY:DAT /PURGE /MIR /R:0 /W:0

------------------------------------------------------------------------------

------------------------------------------------------------------------------

               Total    Copied   Skipped  Mismatch    FAILED    Extras
    Dirs :        46        45         1         0         0         0
   Files :       608       608         0         0         0         0
   Bytes :  45.025 g  45.025 g         0         0         0         0
   Times :   0:05:06   0:05:06                       0:00:00   0:00:00

   Speed :           157766858 Bytes/sec.
   Speed :            9027.492 MegaBytes/min.
   Ended : niedziela, 23 lipca 2023 13:09:49

.\zpaqfranz.exe r 'C:\Program Files (x86)\Diablo IV' K:\test\ -kill -verbose -noeta
zpaqfranz v58.7f-JIT-GUI-L,HW BLAKE3,SHA1/2,SFX64 v55.1,(2023-07-23)
franz:-kill -noeta -verbose
*** ROBOCOPY MODE *** *** ignoring .zfs and :$DATA ***
23/07/2023 13:13:13 Scan dir <<C:/Program Files (x86)/Diablo IV/>>
23/07/2023 13:13:13 Scan dir <<K:/test/>>

Master  48.345.760.542 (45.03 GB files 608) <<C:/Program Files (x86)/Diablo IV/>>
--------------------------------------------------------------------------------------------------------------------

+            608       48.345.760.542 B in    299.62s     161.354.227/sec

Robocopy time     299.67  - Slaves getinfo      0.01 s
Written bytes 48.345.760.542 (45.03 GB) @ 161.328.921 B/sec
====================================================================================================================
check_source          0 check_dest         15 makepath          0
makepath2            16 isequal             0 close             0
close2                0 touch           5.560 delete            0
readopen             96 openout           496 fclose       13.200
buffersize      1.00 MB scandir            16 copy        280.270 @ 172.497.094/s

299.719 seconds (000:04:59) (all OK)

.\zpaqfranz.exe r 'C:\Program Files (x86)\Diablo IV' K:\test\ -kill -verbose -noeta -buffer 512KB
zpaqfranz v58.7f-JIT-GUI-L,HW BLAKE3,SHA1/2,SFX64 v55.1,(2023-07-23)
franz:-buffer                             524.288
franz:-kill -noeta -verbose
*** ROBOCOPY MODE *** *** ignoring .zfs and :$DATA ***
23/07/2023 13:21:08 Scan dir <<C:/Program Files (x86)/Diablo IV/>>
23/07/2023 13:21:08 Scan dir <<K:/test/>>

Master  48.345.760.542 (45.03 GB files 608) <<C:/Program Files (x86)/Diablo IV/>>
--------------------------------------------------------------------------------------------------------------------

+            608       48.345.760.542 B in    294.10s     164.387.684/sec

Robocopy time     294.17  - Slaves getinfo      0.06 s
Written bytes 48.345.760.542 (45.03 GB) @ 164.344.656 B/sec
====================================================================================================================
check_source          0 check_dest         62 makepath          0
makepath2            31 isequal             0 close             0
close2                0 touch           5.474 delete            0
readopen             94 openout           394 fclose       11.032
buffersize    512.00 KB scandir            16 copy        277.040 @ 174.508.231/s

294.250 seconds (000:04:54) (all OK)

.\zpaqfranz.exe r 'C:\Program Files (x86)\Diablo IV' K:\test\ -kill -verbose -noeta -buffer 64KB
zpaqfranz v58.7f-JIT-GUI-L,HW BLAKE3,SHA1/2,SFX64 v55.1,(2023-07-23)
franz:-buffer                              65.536
franz:-kill -noeta -verbose
*** ROBOCOPY MODE *** *** ignoring .zfs and :$DATA ***
23/07/2023 13:26:48 Scan dir <<C:/Program Files (x86)/Diablo IV/>>
23/07/2023 13:26:48 Scan dir <<K:/test/>>

Master  48.345.760.542 (45.03 GB files 608) <<C:/Program Files (x86)/Diablo IV/>>
--------------------------------------------------------------------------------------------------------------------

+            608       48.345.760.542 B in    293.05s     164.976.131/sec

Robocopy time     293.10  - Slaves getinfo      0.06 s
Written bytes 48.345.760.542 (45.03 GB) @ 164.949.113 B/sec
====================================================================================================================
check_source          0 check_dest         61 makepath          0
makepath2            30 isequal             0 close             0
close2                0 touch           5.535 delete            0
readopen             78 openout           478 fclose       11.006
buffersize     64.00 KB scandir            16 copy        275.914 @ 175.220.396/s

293.172 seconds (000:04:53) (all OK)

.\zpaqfranz.exe r 'C:\Program Files (x86)\Diablo IV' K:\test\ -kill -verbose -noeta -big
zpaqfranz v58.7f-JIT-GUI-L,HW BLAKE3,SHA1/2,SFX64 v55.1,(2023-07-23)
franz:-big -kill -noeta -verbose
*** ROBOCOPY MODE *** *** ignoring .zfs and :$DATA ***
23/07/2023 13:32:32 Scan dir <<C:/Program Files (x86)/Diablo IV/>>
23/07/2023 13:32:32 Scan dir <<K:/test/>>

Master  48.345.760.542 (45.03 GB files 608) <<C:/Program Files (x86)/Diablo IV/>>
--------------------------------------------------------------------------------------------------------------------

+            608       48.345.760.542 B in    295.27s     163.735.739/sec

Robocopy time     296.70  - Slaves getinfo      0.08 s
Written bytes 48.345.760.542 (45.03 GB) @ 162.942.183 B/sec
====================================================================================================================
check_source          0 check_dest         79 makepath          0
makepath2             0 isequal             0 close             0
close2                0 touch           5.532 delete            0
readopen              0 openout             0 fclose            0
buffersize      1.00 MB scandir            15 copy        289.656 @ 166.907.505/s

296.781 seconds (000:04:56) (all OK)

####### #    #
#     # #   #
#     # #  #
#     # ###!
#     # #  #
#     # #   #
####### #    #

But maybe it is this "cache" thing?

If this is it: Zrzut ekranu (15)

I can disable it and do tests again

ruptotus commented 1 year ago

After disabling write cache and reboot (just in case it is it)

test results are quite similar:

 robocopy 'C:\Program Files (x86)\Diablo IV\' K:\test\ /mir /r:0 /w:0 /nfl /ndl

-------------------------------------------------------------------------------
   ROBOCOPY     ::     Robust File Copy for Windows
-------------------------------------------------------------------------------

  Started : niedziela, 23 lipca 2023 13:57:05
   Source : C:\Program Files (x86)\Diablo IV\
     Dest : K:\test\

    Files : *.*

  Options : *.* /NDL /NFL /S /E /DCOPY:DA /COPY:DAT /PURGE /MIR /R:0 /W:0

------------------------------------------------------------------------------

------------------------------------------------------------------------------

               Total    Copied   Skipped  Mismatch    FAILED    Extras
    Dirs :        46        45         1         0         0         0
   Files :       608       608         0         0         0         0
   Bytes :  45.025 g  45.025 g         0         0         0         0
   Times :   0:04:51   0:04:51                       0:00:00   0:00:00

   Speed :           165870443 Bytes/sec.
   Speed :            9491.182 MegaBytes/min.
   Ended : niedziela, 23 lipca 2023 14:01:57

 .\zpaqfranz.exe r 'C:\Program Files (x86)\Diablo IV' K:\test\ -kill -verbose -noeta
zpaqfranz v58.7f-JIT-GUI-L,HW BLAKE3,SHA1/2,SFX64 v55.1,(2023-07-23)
franz:-kill -noeta -verbose
*** ROBOCOPY MODE *** *** ignoring .zfs and :$DATA ***
23/07/2023 14:03:09 Scan dir <<C:/Program Files (x86)/Diablo IV/>>
23/07/2023 14:03:09 Scan dir <<K:/test/>>

Master  48.345.760.542 (45.03 GB files 608) <<C:/Program Files (x86)/Diablo IV/>>
--------------------------------------------------------------------------------------------------------------------

+            608       48.345.760.542 B in    302.49s     159.827.564/sec

Robocopy time     303.35  - Slaves getinfo      0.10 s
Written bytes 48.345.760.542 (45.03 GB) @ 159.375.498 B/sec
====================================================================================================================
check_source          0 check_dest         96 makepath          0
makepath2            79 isequal             0 close             0
close2                0 touch           5.533 delete            0
readopen            202 openout         2.202 fclose       12.011
buffersize      1.00 MB scandir            31 copy        282.294 @ 171.260.319/s

303.421 seconds (000:05:03) (all OK)

 .\zpaqfranz.exe r 'C:\Program Files (x86)\Diablo IV' K:\test\ -kill -verbose -noeta -buffer 512KB
zpaqfranz v58.7f-JIT-GUI-L,HW BLAKE3,SHA1/2,SFX64 v55.1,(2023-07-23)
franz:-buffer                             524.288
franz:-kill -noeta -verbose
*** ROBOCOPY MODE *** *** ignoring .zfs and :$DATA ***
23/07/2023 14:08:38 Scan dir <<C:/Program Files (x86)/Diablo IV/>>
23/07/2023 14:08:38 Scan dir <<K:/test/>>

Master  48.345.760.542 (45.03 GB files 608) <<C:/Program Files (x86)/Diablo IV/>>
--------------------------------------------------------------------------------------------------------------------

+            608       48.345.760.542 B in    286.60s     168.689.585/sec

Robocopy time     286.66  - Slaves getinfo      0.01 s
Written bytes 48.345.760.542 (45.03 GB) @ 168.653.689 B/sec
====================================================================================================================
check_source          0 check_dest         15 makepath          0
makepath2            61 isequal             0 close             0
close2                0 touch           5.691 delete            0
readopen             77 openout           277 fclose       10.753
buffersize    512.00 KB scandir            16 copy        269.749 @ 179.224.985/s

286.718 seconds (000:04:46) (all OK)

 .\zpaqfranz.exe r 'C:\Program Files (x86)\Diablo IV' K:\test\ -kill -verbose -noeta -buffer 64KB
zpaqfranz v58.7f-JIT-GUI-L,HW BLAKE3,SHA1/2,SFX64 v55.1,(2023-07-23)
franz:-buffer                              65.536
franz:-kill -noeta -verbose
*** ROBOCOPY MODE *** *** ignoring .zfs and :$DATA ***
23/07/2023 14:14:16 Scan dir <<C:/Program Files (x86)/Diablo IV/>>
23/07/2023 14:14:16 Scan dir <<K:/test/>>

Master  48.345.760.542 (45.03 GB files 608) <<C:/Program Files (x86)/Diablo IV/>>
--------------------------------------------------------------------------------------------------------------------

+            608       48.345.760.542 B in    287.11s     168.387.002/sec

Robocopy time     287.16  - Slaves getinfo      0.08 s
Written bytes 48.345.760.542 (45.03 GB) @ 168.360.027 B/sec
====================================================================================================================
check_source          0 check_dest         78 makepath          0
makepath2            32 isequal             0 close             0
close2                0 touch           5.548 delete            0
readopen             90 openout           390 fclose       11.233
buffersize     64.00 KB scandir            15 copy        269.767 @ 179.213.026/s

287.234 seconds (000:04:47) (all OK)

 .\zpaqfranz.exe r 'C:\Program Files (x86)\Diablo IV' K:\test\ -kill -verbose -noeta -big
zpaqfranz v58.7f-JIT-GUI-L,HW BLAKE3,SHA1/2,SFX64 v55.1,(2023-07-23)
franz:-big -kill -noeta -verbose
*** ROBOCOPY MODE *** *** ignoring .zfs and :$DATA ***
23/07/2023 14:19:23 Scan dir <<C:/Program Files (x86)/Diablo IV/>>
23/07/2023 14:19:23 Scan dir <<K:/test/>>

Master  48.345.760.542 (45.03 GB files 608) <<C:/Program Files (x86)/Diablo IV/>>
--------------------------------------------------------------------------------------------------------------------

+            608       48.345.760.542 B in    287.49s     168.167.941/sec

Robocopy time     287.56  - Slaves getinfo      0.06 s
Written bytes 48.345.760.542 (45.03 GB) @ 168.121.741 B/sec
====================================================================================================================
check_source          0 check_dest         63 makepath          0
makepath2             0 isequal             0 close             0
close2                0 touch           5.637 delete            0
readopen              0 openout             0 fclose            0
buffersize      1.00 MB scandir            15 copy        281.769 @ 171.579.416/s

287.625 seconds (000:04:47) (all OK)

####### #    #
#     # #   #
#     # #  #
#     # ###!
#     # #  #
#     # #   #
####### #    #
fcorbelli commented 1 year ago

Recap to your NAS

robocopy 07:25 zpaqfranz 11:24 zpaqfranz -big 12:28

Internal SSD to SSD Robocopy 05:06 zpaqfranz 04:59 zpaqfranz -big 04:54

I made my own test on Qnap NAS over 1Gbit network (big files)

robocopy 03:36 58.7c -big 03:48 58.7c 04:33 58.6c 06:29

Same file from PC to PC (over 1Gb), two powerful PC (Nvme, 16core-CPUs)

58.7c -big 01:52 @ 117.310.244 B/s asamma

The -big switch use (on Windows) the specific API function (WinCopy) that SHOULD use about "everything" (aka: SMB acceleration, even SMB compression if available)

With robocopy the speed is just about the same (1:52) robo

I suppose that since NAS are actually 'fake' SMB servers (not Windows), performance is highly dependent on the level of compatibility provided.

zpaqfranz 58.7 without -big (aka: using classic fread/fwrite), just like the Windows 2:04 @ 105.940.070 B/s

std

zpaqfranz 58.6 03:00 @ 73.089.250 B/s 6c

I could look into it further but for now I'd say the speed is now decent

PS what kind of Linux do you run on the NAS? zpaqfranz does run on NASes... too, Annapurna-based for example

ruptotus commented 1 year ago

Yes, speed is now decent... thank You for looking into that.

My "NAS" is beelink bt3 pro with Void linux (https://voidlinux.org/):

                __.;=====;.__                   root@void-live
            _.=+==++=++=+=+===;.                --------------
             -=+++=+===+=+=+++++=_              OS: Void Linux x86_64
        .     -=:``     `--==+=++==.            Host: BT3 PRO BT3 PRO
       _vi,    `            --+=++++:           Kernel: 6.3.12_1
      .uvnvi.       _._       -==+==+.          Uptime: 7 hours, 33 mins
     .vvnvnI`    .;==|==;.     :|=||=|.         Packages: 204 (xbps-query)
+QmQQmpvvnv; _yYsyQQWUUQQQm #QmQ#:QQQWUV$QQm.   Shell: sh
 -QQWQWpvvowZ?.wQQQE==<QWWQ/QWQW.QQWW(: jQWQE   CPU: Intel Atom x5-Z8350 (4) @ 1.920GHz
  -$QQQQmmU'  jQQQ@+=<QWQQ)mQQQ.mQQQC+;jWQQ@'   GPU: Intel Atom/Celeron/Pentium Processor x5-E8000/J3xxx/N3xxx
   -$WQ8YnI:   QWQQwgQQWV`mWQQ.jQWQQgyyWW@!     Memory: 2686MiB / 3855MiB (69%)
     -1vvnvv.     `~+++`        ++|+++          GPU Driver: Intel Corporation Device [8086:7270]
      +vnvnnv,                 `-|===           CPU Usage: 110%
       +vnvnvns.           .      :=-           Disk (/): 3.0G / 56G (6%)
        -Invnvvnsi..___..=sv=.     `
          +Invnvnvnnnnnnnnvvnn;.
            ~|Invnvnvvnvvvnnv}+`
               -~|{*l}*|~

with USB attached disk

bash-5.2# zpool list
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
dysk   3.62T  2.84T   808G        -         -     2%    78%  1.00x    ONLINE  -
zroot  57.5G  3.19G  54.3G        -         -     0%     5%  1.00x    ONLINE  -
fcorbelli commented 1 year ago

Please install a C++ compiler and... run zpaqfranz inside the NAS! :) As you know, or maybe not, there are zfs-specific functions, to list, purge snapshot and even doing zfs-stream-captured backups

Yes, zpaqfranz does zfs too :) :)

root@aserver:/zroot/ssd/copie/interna # zpaqfranz
zpaqfranz v58.7i-JIT-L,HW SHA1/2,(2023-07-23)
Usage: zpaqfranz command archive.zpaq files|directories -switches
       double quote for multipart archive name => "test_???.zpaq"
UNIX   zfs commands available      (zpaqfranz h zfs   for details)

  a: Append files     | x: Extract            |   t: Test
  l: List             | v: Verify on FS       |   i: Info
  w: Chunked extract  | p: Paranoid verify    | sum: File hashing

   -to out...: Prefix  files to out   -until N: Rollback to ver.N
     -all [N]: All versions N digit     -key X: Archive  password
        -test: Run additional tests     -force: Always  overwrite
          -mN: 0..5= faster..better      -kill: Destructive    ON

    More help: zpaqfranz h       Extended help: zpaqfranz h h
zpaqfranz v58.7i-JIT-L,HW SHA1/2,(2023-07-23)
On UNIX you get some zfs-specific commands

--- Enumerate / delete snapshots
zfslist
zfspurge
--- Freezing snapshot inside archive
zfsadd
--- Make / restore zfs backups inside zpaq
zfsbackup
zfsreceive
zfsrestore
--- Make / restore zfs backups of proxmox VM
zfsproxbackup
zfsproxrestore
Ask extensive help, example zpaqfranz h zfsadd       zpaqfranz h zfslist
--------------------------------------------------------------------------------------
ruptotus commented 1 year ago

I have this version there:

zpaqfranz v58.5o-JIT-L(2023-07-12)

I tried to use it but as I am ZFS newbie I was not able to make it work :P

I have:

bash-5.2# zfs list
NAME              USED  AVAIL     REFER  MOUNTPOINT
dysk             2.84T   693G     3.20G  /dysk
dysk/dataset     2.83T   693G     2.83T  /mnt/dysk
dysk/kopia       2.62G   693G     2.60G  /dysk/kopia
zroot            3.19G  52.5G       96K  none
zroot/ROOT       3.18G  52.5G       96K  none
zroot/ROOT/void  3.18G  52.5G     2.94G  /
zroot/home       2.84M  52.5G     2.84M  /home

When i try backup home all is fine:

bash-5.2# ./zpaqfranz zfsbackup /mnt/dysk/test/home.zpaq zroot/home -snapshot homecopy
zpaqfranz v58.5o-JIT-L(2023-07-12)
franz:-snapshot             <<homecopy>>
53549: Archive        /mnt/dysk/test/home.zpaq
53550: Pool           zroot/home
53551: Purged pool    zroot_home
53552: Mark           homecopy
53015: The archive does not exist, checking for snapshots
53586: The snap |homecopy00000001|
53587: Fullsnap |zroot/home@homecopy00000001|
53030: Exename  |./zpaqfranz|
53031: Now preparing a brand-new archive and snapshot (starting from 00000001)
53033: CREATING script /tmp/newbackup_zfs_zroot_home_00001.sh
53049: RUNNING  script /tmp/newbackup_zfs_zroot_home_00001.sh
00000001.zfs: No such file or directory
53054: The run seems OK, filesize 5.014.643

3.059 seconds (000:00:03) (all OK)

bash-5.2# ./zpaqfranz l /mnt/dysk/test/home.zpaq
zpaqfranz v58.5o-JIT-L(2023-07-12)
/mnt/dysk/test/home.zpaq:
1 versions, 1 files, 5.014.643 bytes (4.78 MB)

- 2023-07-23 19:15:00          18.141.672  0644 00000001.zfs

           18.141.672 (17.30 MB) of 18.141.672 (17.30 MB) in 1 files shown
            5.014.643 compressed

0.004 seconds (00:00:00) (all OK)

But when I try backup zroot/ROOT/void I get:

bash-5.2# ./zpaqfranz zfsbackup /mnt/dysk/test/root.zpaq zroot/ROOT/void -snapshot rootcopy
zpaqfranz v58.5o-JIT-L(2023-07-12)
franz:-snapshot             <<rootcopy>>
53549: Archive        /mnt/dysk/test/root.zpaq
53550: Pool           zroot/ROOT/void
53551: Purged pool    zroot_ROOT_void
53552: Mark           rootcopy
53549: A pool-dataset |zroot/ROOT/void| must have exactly 0 or 1 / instead of 2

0.002 seconds (00:00:00) (with errors)

Also command zfspurge not working:

bash-5.2# ./zpaqfranz zfslist "*"
zpaqfranz v58.5o-JIT-L(2023-07-12)
franz:zfslist                                   B - command
dysk/kopia@31052023
zroot/ROOT/void@18062023
zroot/home@homecopy00000001

0.062 seconds (00:00:00) (all OK)
bash-5.2# ./zpaqfranz zfspurge "zroot/home@homecopy00000001" -debug
48149: array franz flag size 59
48150: -715                   0  <<Runs just about like zpaq 7.15>>
48150: -append                0  <<Append-only (antiransomware, slow)>>
48150: -backupxxh3            0  <<Use XXH3 in backup instead of MD5>>
48150: -big                   0  <<Big>>
48150: -checksum              0  <<Do checksums>>
48150: -checktxt              0  <<Checktxt (MD5)>>
48150: -comment               0  <<Comment version>>
48150: -debug                 1  <<Activate debug mode>>
48150: -debug -zero           0  <<Add files but zero-filled (debugging)>>
48150: -debug -zero -kill     0  <<Add 0-byte long file (debugging)>>
48150: -desc                  0  <<Orderby desc>>
48150: -fasttxt               0  <<Create test .txt with CRC-32 and QUICK>>
48150: -filelist              0  <<Create a filelist .txt>>
48150: -fix255                0  <<Fix 255>>
48150: -fixeml                0  <<Fix eml filenames>>
48150: -flat                  0  <<Flat filenames>>
48150: -force                 0  <<Force>>
48150: -forcewindows          0  <<Store ADS stuff                (default: NO)>>
48150: -forcezfs              0  <<Enforce using .zfs>>
48150: -frugal                0  <<Frugal backup>>
48150: -hashdeep              0  <<Hashdeep>>
48150: -hw                    0  <<Use HW SHA1>>
48150: -kill                  0  <<Kill>>
48150: -mm                    0  <<Memory mapped>>
48150: -noattributes          0  <<Nessun attributo>>
48150: -nodedup               0  <<Turn off deduplicator>>
48150: -noeta                 0  <<Do not show ETA>>
48150: -nopath                0  <<Do not store path>>
48150: -noqnap                0  <<Exclude QNAP snap & trash>>
48150: -norecursion           0  <<Do not recurse into folders (default: YES)>>
48150: -nosort                0  <<Do not sort file>>
48150: -pakka                 0  <<New output>>
48150: -paranoid              0  <<Paranoid>>
48150: -quiet                 0  <<Do not show filesystem errors>>
48150: -ramdisk               0  <<ramdisk>>
48150: -rename                0  <<Rename>>
48150: -sfxall                0  <<Sfx all>>
48150: -sfxforce              0  <<Sfx force>>
48150: -silent                0  <<Silent>>
48150: -space                 0  <<Do not check space/writeability>>
48150: -ssd                   0  <<SSD>>
48150: -stat                  0  <<Statistics>>
48150: -stdin                 0  <<stdin>>
48150: -stdout                0  <<stdout>>
48150: -store                 0  <<Store mode: no deduplication, no compression>>
48150: -tar                   0  <<TAR mode (store posix)>>
48150: -test                  0  <<Only do test>>
48150: -touch                 0  <<Force 'touch' on date (7.15 to zpaqfranz)>>
48150: -utc                   0  <<Use UTC time>>
48150: -utf                   0  <<UTF-8>>
48150: -verbose               0  <<Verbose output>>
48150: -verify                0  <<Verify>>
48150: -vss                   0  <<Enable Volume Shadow Copies>>
48150: -xls                   0  <<Do NOT force adding of XLS/PPT (default: NO)>>
48150: -zero                  0  <<Flag zero>>
48150: -zfs                   0  <<Do NOT ignore .zfs             (default: YES)>>
48150: /od                    0  <<Order by date>>
48150: /on                    0  <<Order by name>>
48150: /os                    0  <<Order by size>>
41575: Pretend to be Linux
zpaqfranz v58.5o-JIT-L(2023-07-12)
FULL exename <</root/tmp/zpaqfranz>>
42993: The chosen algo 3 SHA-1
franz:zfspurge                                  C - command
franz:-debug -verbose
55311: script EXECUTING /tmp/exec_zfs.sh
55444: NAME                          USED  AVAIL     REFER  MOUNTPOINT
dysk/kopia@31052023          11.9M      -     2.30G  -
zroot/ROOT/void@18062023      248M      -     1.58G  -
zroot/home@homecopy00000001     0B      -     7.43M  -

55452: LINE   00000000 |NAME                          USED  AVAIL     REFER  MOUNTPOINT|
55452: LINE   00000001 |dysk/kopia@31052023          11.9M      -     2.30G  -|
55452: LINE   00000002 |zroot/ROOT/void@18062023      248M      -     1.58G  -|
55452: LINE   00000003 |zroot/home@homecopy00000001     0B      -     7.43M  -|
zfs destroy zroot/home@homecopy00000001

0.053 seconds (00:00:00) (all OK)
33040: call xcommand with a different errorcode (not 1, not 2) 0

bash-5.2# ./zpaqfranz zfslist "*"
zpaqfranz v58.5o-JIT-L(2023-07-12)
franz:zfslist                                   B - command
dysk/kopia@31052023
zroot/ROOT/void@18062023
zroot/home@homecopy00000001

0.054 seconds (00:00:00) (all OK)

When executed manually it works:

bash-5.2# zfs destroy zroot/home@homecopy00000001

bash-5.2# ./zpaqfranz zfslist "*"
zpaqfranz v58.5o-JIT-L(2023-07-12)
franz:zfslist                                   B - command
dysk/kopia@31052023
zroot/ROOT/void@18062023

0.052 seconds (00:00:00) (all OK)
fcorbelli commented 1 year ago

1) zfs purge does NOT delete anything. You have to copy-and-paste (or make a script and run the script) 2) zfsbackup store zfs-streams inside a zpaq. AKA. You cannot restore a file, you have to restore EVERYTHING. The idea is to create copies of very large systems, so large that scanning the file system is slow. By saving every piece of information there are no problems with ACLs, various permissions, virtual machine or whatever is running etc. It is a disaster recovery measure, to be ADDED to normal file-level copies. When all fails, go for zfs recovery. 3) But when I try backup zroot/ROOT/void I get (...) |zroot/ROOT/void| must have exactly 0 or 1 / instead of 2 Try zroot/ROOT

ruptotus commented 1 year ago 
bash-5.2# ./zpaqfranz zfsbackup /mnt/dysk/test/root.zpaq zroot/ROOT
zpaqfranz v58.7k-JIT-L(2023-07-24)
53549: Archive        /mnt/dysk/test/root.zpaq
53550: Pool           zroot/ROOT
53551: Purged pool    zroot_ROOT
53552: Mark           zpaqfranz
53015: The archive does not exist, checking for snapshots
53586: The snap |zpaqfranz00000001|
53587: Fullsnap |zroot/ROOT@zpaqfranz00000001|
53030: Exename  |./zpaqfranz|
53031: Now preparing a brand-new archive and snapshot (starting from 00000001)
53033: CREATING script /tmp/newbackup_zfs_zroot_ROOT.sh
53049: RUNNING  script /tmp/newbackup_zfs_zroot_ROOT.sh
00000001.zfs: No such file or directory
53054: The run seems OK, filesize 5.485

0.230 seconds (00:00:00) (all OK)

It backup zroot/ROOT = 96K and not zroot/ROOT/void = 2.94G

bash-5.2# ./zpaqfranz l /mnt/dysk/test/root.zpaq
zpaqfranz v58.7k-JIT-L(2023-07-24)
/mnt/dysk/test/root.zpaq:
1 versions, 1 files, 5.485 bytes (5.36 KB)

- 2023-07-30 19:34:52              45.264  0644 00000001.zfs

               45.264 (44.20 KB) of 45.264 (44.20 KB) in 1 files shown
                5.485 compressed

0.004 seconds (00:00:00) (all OK)

zfs list

NAME              USED  AVAIL     REFER  MOUNTPOINT
zroot            3.20G  52.5G       96K  none
zroot/ROOT       3.18G  52.5G       96K  none
zroot/ROOT/void  3.18G  52.5G     2.94G  /
zroot/home       7.43M  52.5G     7.43M  /home

but I installed Void linux with custom tutorial, and not official installer as it do not support installing on ZFS... so maybe there is not good practice to make nested datasets like that?