Windows -longpath switch, inconsistently reports number of files

Noticed an inconsistency, not sure if this is documented?

I have a folder with 187,372 files.

1466 folders

I cannot seem to find the right combination of flags to pick up all of these files, because:

Some of the paths are longer than 255 characters
UNC or //?/, -longpath, doesn't seem to pick up all the files either.

I am attaching output of various commands I've tried.

PS C:\Users\Alex\_tempworkdir> zpaqfranz.exe a EmailProcessingBucket.zpaq .\EmailProcessing\ -test -longpath
zpaqfranz v55.14b-experimental-JIT-L (HW BLAKE3), SFX64 v55.1, (Sep  5 2022)
franz:Long path (on Windows)
38992: INFO: getting Windows' long filenames
EmailProcessingBucket.zpaq:
3 versions, 352.056 files, 299.464 fragments, 6.983 blocks, 9.940.897.057 bytes (9.26 GB)

15367: Windows error # 53
       //?/UNC/EmailProcessing/EMLProcessing/*

MAYBE OUT OF FREE SPACE OR INVALID PATH? 15367

QUIT: total size,file/folder count == zero. Already archived/wrong/inaccessible source?

5.922 seconds (000:00:05)  (with warnings)
PS C:\Users\Alex\_tempworkdir>

zpaqfranz.exe a EmailProcessingBucket.zpaq \\TAICHI\C$\Users\Alex\_tempworkdir\EmailProcessing\ -test

16672: path not found : maybe length 00000273 >255? //TAICHI/C$/Users/Alex/_tempworkdir/EmailProcessing/unzip/FastmailEmls3/FastmailEmls 3/ProcessToPDF-Nov92019/2018/12/2018-12-10 17.49.51Z  UPDATE  Door issue @ Civic Center cleared. Resuming #subwaysvc. Residual congestion slow svc possible while we work to balance svc.pdf

16672: path not found : maybe length 00000277 >255? //TAICHI/C$/Users/Alex/_tempworkdir/EmailProcessing/unzip/FastmailEmls3/FastmailEmls 3/ProcessToPDF-Nov92019/2018/12/2018-12-10 21.31.21Z  HEAL Initiative  Preventing Opioid Use Disorder in Older Adolescents and Young Adults (ages 16–30) (UG3 UH3 Clinical Trial Required).pdf

[...thousands more line...s]

PS C:\Users\Alex\_tempworkdir> zpaqfranz.exe a EmailProcessingBucket.zpaq \\TAICHI\C$\Users\Alex\_tempworkdir\EmailProcessing\ -test -longpath
zpaqfranz v55.14b-experimental-JIT-L (HW BLAKE3), SFX64 v55.1, (Sep  5 2022)
franz:Long path (on Windows)
38992: INFO: getting Windows' long filenames
EmailProcessingBucket.zpaq:
3 versions, 352.056 files, 299.464 fragments, 6.983 blocks, 9.940.897.057 bytes (9.26 GB)
Updating EmailProcessingBucket.zpaq at offset 9.940.897.057 + 0
Adding 212.584 (207.60 KB) in 2 files (50 dirs), 16 threads @ 2022-09-07 10:09:52
Long filenames (>255)       991 *** WARNING *** (-fix255)
2 +added, 1 -removed.

9.940.897.057 + (212.584 -> 0 -> 1.101) = 9.940.898.158 @ 154.58 KB/s
=============================================================================================================================================================================================================
Compare archive content of:EmailProcessingBucket.zpaq:
4 versions, 352.058 files, 299.464 fragments, 6.985 blocks, 9.940.898.158 bytes (9.26 GB)
Scanning     10.000 2.69s      3.720 file/s (        1.160.305.159)
   10.641 in <<//?/UNC/TAICHI/C$/Users/Alex/_tempworkdir/EmailProcessing/EMLProcessing/>>
Total files found: 10.641

PS C:\Users\Alex\_tempworkdir> zpaqfranz.exe a EmailProcessingBucket.zpaq \\TAICHI\C$\Users\Alex\_tempworkdir\EmailProcessing\ -test -longpath -fix255
zpaqfranz v55.14b-experimental-JIT-L (HW BLAKE3), SFX64 v55.1, (Sep  5 2022)
franz:-fix255 Long path (on Windows)
38992: INFO: getting Windows' long filenames
EmailProcessingBucket.zpaq:
4 versions, 352.058 files, 299.464 fragments, 6.985 blocks, 9.940.898.158 bytes (9.26 GB)
Updating EmailProcessingBucket.zpaq at offset 9.940.898.158 + 0
Adding 0 (0.00 B) in 0 files (50 dirs), 16 threads @ 2022-09-07 10:11:31
Long filenames (>255)       991 *** WARNING *** (-fix255)
0 +added, 0 -removed.

9.940.898.158 + (0 -> 0 -> 0) = 9.940.898.158 @ 0.00 B/s
=============================================================================================================================================================================================================
Compare archive content of:EmailProcessingBucket.zpaq:
4 versions, 352.058 files, 299.464 fragments, 6.985 blocks, 9.940.898.158 bytes (9.26 GB)
Scanning     10.000 2.69s      3.720 file/s (        1.160.305.159)
   10.641 in <<//?/UNC/TAICHI/C$/Users/Alex/_tempworkdir/EmailProcessing/EMLProcessing/>>
Total files found: 10.641

Thank you very much. I'll look ASAP

Thanks for letting me know!

Honestly, I deeply enjoy this program and how it works. Thank you so much for implementing this and putting it on github.

But I also understand it's free, so, no rush to solve this.

On my end, I've been digging through the code to see if it's some kind of Windows API call quirk? Not sure, not super familiar with Windows internals / C++.

Very first thing: use a fullpath instead of a relative one (file selection to be added)
Something like

C:\Users\Alex\_tempworkdir\EmailProcessing\  or whatever

Let me know, thank you

Thank you for the suggestion, the issue seems to persist though.

Here's the output:

PowerShell 7.2.6
Copyright (c) Microsoft Corporation.

https://aka.ms/powershell
Type 'help' to get help.

PS C:\Users\Alex\_tempworkdir> zpaqfranz.exe a EmailProcessingBucket.zpaq C:\Users\Alex\_tempworkdir\EmailProcessing\ -test -verify -longpath
zpaqfranz v55.14b-experimental-JIT-L (HW BLAKE3), SFX64 v55.1, (Sep  5 2022)
franz:-verify Long path (on Windows)
38992: INFO: getting Windows' long filenames
Creating EmailProcessingBucket.zpaq at offset 0 + 0
Adding 1.425.019.051 (1.33 GB) in 2 files (0 dirs), 16 threads @ 2022-09-08 10:23:45
2 +added, 0 -removed.

0 + (1.425.019.051 -> 1.424.804.534 -> 1.425.567.750) = 1.425.567.750 @ 177.86 MB/s
====================================================================================================================
Compare archive content of:EmailProcessingBucket.zpaq:
1 versions, 2 files, 20.507 fragments, 89 blocks, 1.425.567.750 bytes (1.33 GB)

        2 in <<//?/C:/Users/Alex/_tempworkdir/EmailProcessing/InboxAndEmlComparison/>>
Total files found: 2

Done 00%     0.00 B of    1.33 GB, diff 0 bytes so far
Done 50%  678.27 MB of    1.33 GB, diff 0 bytes so far
SHA1 100 % (   674.04 MB) @    463.58 MB/s

00000002 = same
Total different file size: 0 bytes
====================================================================================================================
EmailProcessingBucket.zpaq:
1 versions, 2 files, 20.507 fragments, 89 blocks, 1.425.567.750 bytes (1.33 GB)

Verify hashes of one version vs filesystem (1 thread, -ssd for multithread)
Total files 2 -> in 001 threads -> 2 to be checked
--------------------------------------------------------------------------------------------------------------------
OK   XXHASH64 : 00000002 of 00000002 (     1.33 GB hash check against file on disk)
--------------------------------------------------------------------------------------------------------------------

10.860 seconds (000:00:10)  (all OK)
PS C:\Users\Alex\_tempworkdir> zpaqfranz.exe a EmailProcessingBucket2_test.zpaq C:\Users\Alex\_tempworkdir\EmailProcessing\ -test -verify
zpaqfranz v55.14b-experimental-JIT-L (HW BLAKE3), SFX64 v55.1, (Sep  5 2022)
franz:-verify
Creating EmailProcessingBucket2_test.zpaq at offset 0 + 0
Adding 15.665.935.642 (14.59 GB) in 12.915 files (17 dirs), 16 threads @ 2022-09-08 10:25:34
Long filenames (>255)         7 *** WARNING *** (-fix255)
        19.26% 00:01:13 (   2.81 GB)->( 292.47 MB) of (  14.59 GB)  169.29 MB/sec
16672: path not found : maybe length 00000280 >255? C:/Users/Alex/_tempworkdir/EmailProcessing/ProcessThese/AppleMailExport/Emls/OutlookDL/New 2019 QPP Resources Available on QPP.CMS.GOV; Reminder_ MIPS Virtual Group Elections for 2019 are Due on December 31; Reminder_ Submit Your 2018 QPP Exception Applications by December 31.eml
        19.72% 00:01:22 (   2.88 GB)->( 305.42 MB) of (  14.59 GB)  147.32 MB/sec
16672: path not found : maybe length 00000317 >255? C:/Users/Alex/_tempworkdir/EmailProcessing/ProcessThese/AppleMailExport/Emls/OutlookDL/Save the Date for Budget Work Sessions and Public Hearings! - Reserve la fecha para las sesiones de trabajo y las audiencias públicas para el presupuesto! - Réservez la date des séances de travail et des audiences publiques.eml
        20.63% 00:01:46 (   3.01 GB)->( 335.46 MB) of (  14.59 GB)  114.18 MB/sec
16672: path not found : maybe length 00000267 >255? C:/Users/Alex/_tempworkdir/EmailProcessing/ProcessThese/AppleMailExport/Emls/OutlookDL/Statement at Open Meeting on Consideration of Whether to Adopt Rule of Practice 194 and to Propose Rules Regarding Risk Mitigation Techniques for Uncleared Security-Based Swaps.eml
        20.99% 00:01:56 (   3.06 GB)->( 346.47 MB) of (  14.59 GB)  104.54 MB/sec
16672: path not found : maybe length 00000319 >255? C:/Users/Alex/_tempworkdir/EmailProcessing/ProcessThese/AppleMailExport/Emls/OutlookDL/Examining protective and buffering associations between sociocultural factors and adverse childhood experiences among American Indian adults with type 2 diabetes_ a quantitative, community-based participatory research approach 2.eml

16672: path not found : maybe length 00000317 >255? C:/Users/Alex/_tempworkdir/EmailProcessing/ProcessThese/AppleMailExport/Emls/OutlookDL/Examining protective and buffering associations between sociocultural factors and adverse childhood experiences among American Indian adults with type 2 diabetes_ a quantitative, community-based participatory research approach.eml
        21.02% 00:01:57 (   3.07 GB)->( 346.47 MB) of (  14.59 GB)  101.29 MB/sec
16672: path not found : maybe length 00000265 >255? C:/Users/Alex/_tempworkdir/EmailProcessing/ProcessThese/AppleMailExport/Emls/OutlookDL/Health Care, Family, and Community Factors Associated with Mental, Behavioral, and Developmental Disorders and Poverty Among Children Aged 2–8 Years — United States, 2016.eml
        21.04% 00:01:58 (   3.07 GB)->( 346.47 MB) of (  14.59 GB)  101.41 MB/sec
16672: path not found : maybe length 00000269 >255? C:/Users/Alex/_tempworkdir/EmailProcessing/ProcessThese/AppleMailExport/Emls/OutlookDL/Notice of NLM's Participation in PAR-19-093 _Leveraging Health Information Technology (Health IT) to Address Minority Health and Health Disparities (R01 Clinical Trial Optional)_.eml
12.925 +added, 0 -removed.

0 + (15.665.606.695 -> 13.410.102.066 -> 12.615.151.879) = 12.615.151.879 @ 142.94 MB/s

*** WINDOWS WARNING *** found file length >255. Suggestion: use -longpath switch
====================================================================================================================
Compare archive content of:EmailProcessingBucket2_test.zpaq:
1 versions, 12.925 files, 207.692 fragments, 995 blocks, 12.615.151.879 bytes (11.75 GB)
Scanning     12.000 104.66s        114 file/s (        2.216.056.100)
   12.932 in <<C:/Users/Alex/_tempworkdir/EmailProcessing/>>
Total files found: 12.932

Done 00%     0.00 B of   14.59 GB, diff 0 bytes so far
SHA1 100 % (   672.34 MB) @    462.41 MB/s
Done 99%    2.10 GB of   14.59 GB, diff 328.947 bytes so far
SHA1 100 % (     2.00 GB) @    482.81 MB/s
SHA1 100 % (   318.16 MB) @    441.89 MB/s
SHA1 100 % (    10.03 GB) @    488.81 MB/s

+ 2022-09-08 04:07:07              31.026 C:/Users/Alex/_tempworkdir/EmailProcessing/ProcessThese/AppleMailExport/Emls/OutlookDL/Examining protective and buffering associations between sociocultural factors and adverse childhood experiences among American Indian adults with type 2 diabetes_ a quantitative, community-based participatory research approach 2.eml
+ 2022-09-08 04:18:47              31.243 C:/Users/Alex/_tempworkdir/EmailProcessing/ProcessThese/AppleMailExport/Emls/OutlookDL/Examining protective and buffering associations between sociocultural factors and adverse childhood experiences among American Indian adults with type 2 diabetes_ a quantitative, community-based participatory research approach.eml
+ 2022-09-08 04:15:13              31.099 C:/Users/Alex/_tempworkdir/EmailProcessing/ProcessThese/AppleMailExport/Emls/OutlookDL/Health Care, Family, and Community Factors Associated with Mental, Behavioral, and Developmental Disorders and Poverty Among Children Aged 2–8 Years — United States, 2016.eml
+ 2022-09-08 04:08:42             109.947 C:/Users/Alex/_tempworkdir/EmailProcessing/ProcessThese/AppleMailExport/Emls/OutlookDL/New 2019 QPP Resources Available on QPP.CMS.GOV; Reminder_ MIPS Virtual Group Elections for 2019 are Due on December 31; Reminder_ Submit Your 2018 QPP Exception Applications by December 31.eml
+ 2022-09-08 04:53:53              30.771 C:/Users/Alex/_tempworkdir/EmailProcessing/ProcessThese/AppleMailExport/Emls/OutlookDL/Notice of NLM's Participation in PAR-19-093 _Leveraging Health Information Technology (Health IT) to Address Minority Health and Health Disparities (R01 Clinical Trial Optional)_.eml
+ 2022-09-08 04:53:47              61.337 C:/Users/Alex/_tempworkdir/EmailProcessing/ProcessThese/AppleMailExport/Emls/OutlookDL/Save the Date for Budget Work Sessions and Public Hearings! - Reserve la fecha para las sesiones de trabajo y las audiencias públicas para el presupuesto! - Réservez la date des séances de travail et des audiences publiques.eml
+ 2022-09-08 04:53:42              33.524 C:/Users/Alex/_tempworkdir/EmailProcessing/ProcessThese/AppleMailExport/Emls/OutlookDL/Statement at Open Meeting on Consideration of Whether to Adopt Rule of Practice 194 and to Propose Rules Regarding Risk Mitigation Techniques for Uncleared Security-Based Swaps.eml
00012925 = same
00000007 +external (file missing in ZPAQ)
Total different file size: 328.947 bytes
====================================================================================================================
EmailProcessingBucket2_test.zpaq:
1 versions, 12.925 files, 207.692 fragments, 995 blocks, 12.615.151.879 bytes (11.75 GB)

Verify hashes of one version vs filesystem (1 thread, -ssd for multithread)
Total files 12.908 -> in 001 threads -> 12.908 to be checked
--------------------------------------------------------------------------------------------------------------------
OK   XXHASH64 : 00012908 of 00012908 (    14.59 GB hash check against file on disk)
--------------------------------------------------------------------------------------------------------------------

139.250 seconds (000:02:19)  (all OK)

Seems 00000007 errors by path/filename too long (>255)
PS C:\Users\Alex\_tempworkdir> zpaqfranz.exe a EmailProcessingBucket3_test.zpaq C:\Users\Alex\_tempworkdir\EmailProcessing\ -test -verify -fix255
zpaqfranz v55.14b-experimental-JIT-L (HW BLAKE3), SFX64 v55.1, (Sep  5 2022)
franz:-verify -fix255
Creating EmailProcessingBucket3_test.zpaq at offset 0 + 0
Adding 15.665.935.642 (14.59 GB) in 12.915 files (17 dirs), 16 threads @ 2022-09-08 10:28:06
Long filenames (>255)         7 *** WARNING *** (-fix255)
        18.16% 00:01:07 (   2.65 GB)->( 283.25 MB) of (  14.59 GB)  180.86 MB/sec
16672: path not found : maybe length 00000280 >255? C:/Users/Alex/_tempworkdir/EmailProcessing/ProcessThese/AppleMailExport/Emls/OutlookDL/New 2019 QPP Resources Available on QPP.CMS.GOV; Reminder_ MIPS Virtual Group Elections for 2019 are Due on December 31; Reminder_ Submit Your 2018 QPP Exception Applications by December 31.eml
        19.29% 00:01:07 (   2.81 GB)->( 283.25 MB) of (  14.59 GB)  180.16 MB/sec
16672: path not found : maybe length 00000317 >255? C:/Users/Alex/_tempworkdir/EmailProcessing/ProcessThese/AppleMailExport/Emls/OutlookDL/Save the Date for Budget Work Sessions and Public Hearings! - Reserve la fecha para las sesiones de trabajo y las audiencias públicas para el presupuesto! - Réservez la date des séances de travail et des audiences publiques.eml

16672: path not found : maybe length 00000267 >255? C:/Users/Alex/_tempworkdir/EmailProcessing/ProcessThese/AppleMailExport/Emls/OutlookDL/Statement at Open Meeting on Consideration of Whether to Adopt Rule of Practice 194 and to Propose Rules Regarding Risk Mitigation Techniques for Uncleared Security-Based Swaps.eml

16672: path not found : maybe length 00000319 >255? C:/Users/Alex/_tempworkdir/EmailProcessing/ProcessThese/AppleMailExport/Emls/OutlookDL/Examining protective and buffering associations between sociocultural factors and adverse childhood experiences among American Indian adults with type 2 diabetes_ a quantitative, community-based participatory research approach 2.eml

16672: path not found : maybe length 00000317 >255? C:/Users/Alex/_tempworkdir/EmailProcessing/ProcessThese/AppleMailExport/Emls/OutlookDL/Examining protective and buffering associations between sociocultural factors and adverse childhood experiences among American Indian adults with type 2 diabetes_ a quantitative, community-based participatory research approach.eml

16672: path not found : maybe length 00000265 >255? C:/Users/Alex/_tempworkdir/EmailProcessing/ProcessThese/AppleMailExport/Emls/OutlookDL/Health Care, Family, and Community Factors Associated with Mental, Behavioral, and Developmental Disorders and Poverty Among Children Aged 2–8 Years — United States, 2016.eml

16672: path not found : maybe length 00000269 >255? C:/Users/Alex/_tempworkdir/EmailProcessing/ProcessThese/AppleMailExport/Emls/OutlookDL/Notice of NLM's Participation in PAR-19-093 _Leveraging Health Information Technology (Health IT) to Address Minority Health and Health Disparities (R01 Clinical Trial Optional)_.eml
12.925 +added, 0 -removed.

0 + (15.665.606.695 -> 13.410.102.066 -> 12.615.151.879) = 12.615.151.879 @ 171.01 MB/s

*** WINDOWS WARNING *** found file length >255. Suggestion: use -longpath switch
====================================================================================================================
Compare archive content of:EmailProcessingBucket3_test.zpaq:
1 versions, 12.925 files, 207.692 fragments, 995 blocks, 12.615.151.879 bytes (11.75 GB)
Scanning     12.000 87.50s        137 file/s (        2.216.056.100)
   12.932 in <<C:/Users/Alex/_tempworkdir/EmailProcessing/>>
Total files found: 12.932

Done 00%     0.00 B of   14.59 GB, diff 0 bytes so far
SHA1 100 % (   672.34 MB) @    457.37 MB/s
Done 99%    2.10 GB of   14.59 GB, diff 328.947 bytes so far
SHA1 100 % (     2.00 GB) @    484.64 MB/s
SHA1 100 % (   318.16 MB) @    451.93 MB/s
SHA1 100 % (    10.03 GB) @    492.84 MB/s

+ 2022-09-08 04:07:07              31.026 C:/Users/Alex/_tempworkdir/EmailProcessing/ProcessThese/AppleMailExport/Emls/OutlookDL/Examining protective and buffering associations between sociocultural factors and adverse childhood experiences among American Indian adults with type 2 diabetes_ a quantitative, community-based participatory research approach 2.eml
+ 2022-09-08 04:18:47              31.243 C:/Users/Alex/_tempworkdir/EmailProcessing/ProcessThese/AppleMailExport/Emls/OutlookDL/Examining protective and buffering associations between sociocultural factors and adverse childhood experiences among American Indian adults with type 2 diabetes_ a quantitative, community-based participatory research approach.eml
+ 2022-09-08 04:15:13              31.099 C:/Users/Alex/_tempworkdir/EmailProcessing/ProcessThese/AppleMailExport/Emls/OutlookDL/Health Care, Family, and Community Factors Associated with Mental, Behavioral, and Developmental Disorders and Poverty Among Children Aged 2–8 Years — United States, 2016.eml
+ 2022-09-08 04:08:42             109.947 C:/Users/Alex/_tempworkdir/EmailProcessing/ProcessThese/AppleMailExport/Emls/OutlookDL/New 2019 QPP Resources Available on QPP.CMS.GOV; Reminder_ MIPS Virtual Group Elections for 2019 are Due on December 31; Reminder_ Submit Your 2018 QPP Exception Applications by December 31.eml
+ 2022-09-08 04:53:53              30.771 C:/Users/Alex/_tempworkdir/EmailProcessing/ProcessThese/AppleMailExport/Emls/OutlookDL/Notice of NLM's Participation in PAR-19-093 _Leveraging Health Information Technology (Health IT) to Address Minority Health and Health Disparities (R01 Clinical Trial Optional)_.eml
+ 2022-09-08 04:53:47              61.337 C:/Users/Alex/_tempworkdir/EmailProcessing/ProcessThese/AppleMailExport/Emls/OutlookDL/Save the Date for Budget Work Sessions and Public Hearings! - Reserve la fecha para las sesiones de trabajo y las audiencias públicas para el presupuesto! - Réservez la date des séances de travail et des audiences publiques.eml
+ 2022-09-08 04:53:42              33.524 C:/Users/Alex/_tempworkdir/EmailProcessing/ProcessThese/AppleMailExport/Emls/OutlookDL/Statement at Open Meeting on Consideration of Whether to Adopt Rule of Practice 194 and to Propose Rules Regarding Risk Mitigation Techniques for Uncleared Security-Based Swaps.eml
00012925 = same
00000007 +external (file missing in ZPAQ)
Total different file size: 328.947 bytes
====================================================================================================================
EmailProcessingBucket3_test.zpaq:
1 versions, 12.925 files, 207.692 fragments, 995 blocks, 12.615.151.879 bytes (11.75 GB)

Verify hashes of one version vs filesystem (1 thread, -ssd for multithread)
Total files 12.908 -> in 001 threads -> 12.908 to be checked
--------------------------------------------------------------------------------------------------------------------
OK   XXHASH64 : 00012908 of 00012908 (    14.59 GB hash check against file on disk)
--------------------------------------------------------------------------------------------------------------------

121.891 seconds (000:02:01)  (all OK)

Seems 00000007 errors by path/filename too long (>255)
PS C:\Users\Alex\_tempworkdir>

Please run outside powershell and use -debug

I ran this inside the CMD prompt.

C:\Users\Alex\_tempworkdir>zpaqfranz.exe a EmailProcessingBucket.zpaq C:\Users\Alex\_tempworkdir\EmailProcessing\ -test -verify -longpath -debug 
zpaqfranz v55.14b-experimental-JIT-L (HW BLAKE3), SFX64 v55.1, (Sep  5 2022)
franz:-verify Long path (on Windows) DEBUG very verbose (-debug)
38992: INFO: getting Windows' long filenames
38994: 000 From C:/Users/Alex/_tempworkdir/EmailProcessing/
FATTR 00080010 RES0 00000000 C:/Users/Alex/_tempworkdir/EmailProcessing/.
DIRECTORY;
FATTR 00080010 RES0 00000000 C:/Users/Alex/_tempworkdir/EmailProcessing/..
DIRECTORY;
FATTR 00080010 RES0 00000000 C:/Users/Alex/_tempworkdir/EmailProcessing/InboxAndEmlComparison
DIRECTORY;
FATTR 00080020 RES0 00000000 C:/Users/Alex/_tempworkdir/EmailProcessing/mboxemlcount.py
ARCHIVE;
FATTR 00080010 RES0 00000000 C:/Users/Alex/_tempworkdir/EmailProcessing/PDFs
DIRECTORY;
FATTR 00080010 RES0 00000000 C:/Users/Alex/_tempworkdir/EmailProcessing/ProcessThese
DIRECTORY;
FATTR 00080010 RES0 00000000 C:/Users/Alex/_tempworkdir/EmailProcessing/Sources
DIRECTORY;
--------------------------------------------------------------------------------------------------------------------
39027: 000 From //?/C:/Users/Alex/_tempworkdir/EmailProcessing/InboxAndEmlComparison/
39028: 000 To   C:/Users/Alex/_tempworkdir/EmailProcessing/InboxAndEmlComparison/
21026: check_if_password of EmailProcessingBucket.zpaq
EmailProcessingBucket.zpaq:
1 versions, 2 files, 20.507 fragments, 89 blocks, 1.425.567.750 bytes (1.33 GB)
FATTR 00080010 RES0 0305514E //?/C:/Users/Alex/_tempworkdir/EmailProcessing/InboxAndEmlComparison/.
DIRECTORY;
FATTR 00080010 RES0 0305514E //?/C:/Users/Alex/_tempworkdir/EmailProcessing/InboxAndEmlComparison/..
DIRECTORY;
FATTR 00080020 RES0 0305514E //?/C:/Users/Alex/_tempworkdir/EmailProcessing/InboxAndEmlComparison/aeml.zip
ARCHIVE;
FATTR 00080020 RES0 0305514E //?/C:/Users/Alex/_tempworkdir/EmailProcessing/InboxAndEmlComparison/INBOX-1.mbox.zip
ARCHIVE;
Default sort

QUIT: total size,file/folder count == zero. Already archived/wrong/inaccessible source?

0.031 seconds (00:00:00)  (with warnings)
33033: call xcommand on errorcode==1
Entering xcommand()
i_command   ||
i_parameter ||
Exit because empty i_command

C:\Users\Alex\_tempworkdir>

I've been able to work around this by putting the files into a zip container, uncompressed. zpaqfranz is doing a great job with it.

It reduces the size due to compression
If I add similar files in a different container, it deduplicates just fine

Please copy the folder with long names using something like

robocopy c:\mylongdir c:\myemptydir /mir /create /nfl /ndl

this will create (for privacy reason) a zero-bytes-long c: \ myemptydir, but with filenames

Then 7-zip and send to me (if you want, of course) franco@francocorbelli.com

I normally use zpaqfranz, with -longpath, with millions of EML files (there is even a specific function for renaming).
I am very curious to understand exactly what happens
The command to use is simply

zpaqfranz a d:\thecopy.zpaq c:\mylongdir -longpath -verbose

Nothing more

Thank you

Fast test of extracting 3.000+ long filenames on Windows

263 versions, 626.041 files, 7.236.137 fragments, 45.777 blocks, 383.229.089.963 bytes (356.91 GB)
32152: -longpath and ONE to: deal with rename()
Extracting 617.865.959.426 bytes (575.43 GB) in 480.601 files (35.056 folders) with 32 threads
**** CREATING FILES AND FOLDER STRUCTURE  *****
**** EXTRACTIBILITY TEST (ON FILESYSTEM)  *****
29247: Scanning dir <<//?/z:/pippo3>>

29260: files 0 dirs 0 size 0 long 0
Start counting... done in 0.23s
Folders        38.534 (long            5)
Files         552.466 (long        3.991)
Time            51.77s
Total         591.000
Folders        38.534
Files         552.466
Errors              0

72.110 seconds (000:01:12)  (all OK)

This is a fresh eml-only test

J:\>c:\zpaqfranz\release\55_14\zpaqfranzhw a j:\emlbackup.zpaq j:\emlarchive -verbose -longpath
zpaqfranz v55.14b-experimental-JIT-L (HW BLAKE3,SHA1), SFX64 v55.1, (Sep  5 2022)
franz:-verbose Long path (on Windows)
38992: INFO: getting Windows' long filenames
Integrity check type: XXHASH64+CRC-32
Creating j:/emlbackup.zpaq at offset 0 + 0
Adding 138.048.360.109 (128.57 GB) in 264.269 files (19.402 dirs), 32 threads -method 14 @ 2022-09-10 10:59:27
Long filenames (>255)     2.842 *** WARNING *** (-fix255)
Non-latin (UTF-8)        16.116
283.671 +added, 0 -removed.

                     0 starting size
       138.048.360.109 data to be added
        78.125.398.170 after deduplication
+       58.315.370.986 after compression
        58.315.370.986 total size
Total speed 161.65 MB/s
20650: no file errors tracked

814.718 seconds (000:13:34)  (all OK)

So I'm really curious to see what kind of filenames Maybe the problem is in non-Latin characters?
I have tested with Russian, Chinese, Indian and Arabic, but maybe something unexpected could be there

55_15k.zip This pre-release (should) works with either

relative
full
UNC longpaths

Relative path
PLEASE NOTE: relative "long" path (AFAIK) does not exists. Therefore relatives will be converted in full

C:\zpaqfranz>zpaqfranz a z:\relative .\ok -longpath
zpaqfranz v55.15k-experimental-JIT-L (HW BLAKE3), SFX64 v55.1, (Sep 12 2022)
franz:Long path (on Windows)
38992: INFO: getting Windows' long filenames
Creating z:/relative.zpaq at offset 0 + 0
Adding 6.792.854 (6.48 MB) in 15 files (0 dirs), 32 threads @ 2022-09-12 18:42:45
15 +added, 0 -removed.

0 + (6.792.854 -> 4.462.528 -> 939.494) = 939.494 @ 45.94 MB/s

0.156 seconds (00:00:00)  (all OK)

Full path

C:\zpaqfranz>zpaqfranz a z:\full c:\zpaqfranz\ok -longpath
zpaqfranz v55.15k-experimental-JIT-L (HW BLAKE3), SFX64 v55.1, (Sep 12 2022)
franz:Long path (on Windows)
38992: INFO: getting Windows' long filenames
Creating z:/full.zpaq at offset 0 + 0
Adding 6.792.854 (6.48 MB) in 15 files (0 dirs), 32 threads @ 2022-09-12 18:43:55
15 +added, 0 -removed.

0 + (6.792.854 -> 4.462.528 -> 939.494) = 939.494 @ 41.26 MB/s

0.157 seconds (00:00:00)  (all OK)

UNC (not very tested)

C:\zpaqfranz>zpaqfranz a z:\fromnetwork \\franzk\c$\zpaqfranz\ok -longpath
zpaqfranz v55.15k-experimental-JIT-L (HW BLAKE3), SFX64 v55.1, (Sep 12 2022)
franz:Long path (on Windows)
38992: INFO: getting Windows' long filenames
38748: WARNING: VERY LIMITED SUPPORT FOR UNC AND -LONGPATH
Creating z:/fromnetwork.zpaq at offset 0 + 0
Adding 6.792.854 (6.48 MB) in 15 files (0 dirs), 32 threads @ 2022-09-12 18:44:34
15 +added, 0 -removed.

0 + (6.792.854 -> 4.462.528 -> 939.502) = 939.502 @ 5.53 MB/s

1.172 seconds (000:00:01)  (all OK)

Please test and let me know.
Thanks

Thank you for all the detailed comments. I'll do testing today.

Wanted to drop an update about this: I think the issue might be related to the path being inside C:\Users\Alex.

I am running the same version on F:\EmailProcessing with -longpath switch, and it picks up all the files.

Spoke too soon, it looks like extraction from linux to NTFS throws a lot of filepath too long errors. Or, filepath cannot be found if I use the long-switch.

Still testing your suggestions.

Extracting from *nix to NTFS does work (with long file names), "straight" (x) and even with w (the "ramdisk")

I regularly use zpaqfranz for these tasks (actually files created on FreeBSD, but the paths are == Linux ones), just like the previous example

I suggest something like

zpaqfranz x fserver.zpaq -to j:\myeml -only *.eml -longpath -verbose

This will work fine with path like this (/tank/condivisioni...)

 2022-05-06 11:07:21                   0  0666 /tank/condivisioni/.cestino/administrator/Utenti/Ufficio Tecnico/PROGETTI/012 - CHISSA MULTICONFIGURATO/DOCUMENTI 123-456/Corrispondenza/My Very Own System - A great Project indeed - 'testme@thegreattest.com' (zpaqfranz@thegreatthest.com) - 2022-09-04 1106.eml

will become (j:\myeml\tank\condivisioni...)

J:\>dir myeml /s|more
 Il volume nell'unità J è J-WORK
 Numero di serie del volume: 3A09-40D4

 Directory di J:\myeml

14/09/2022  11:17    <DIR>          .
14/09/2022  11:17    <DIR>          ..
14/09/2022  11:17    <DIR>          tank
               0 File              0 byte

 Directory di J:\myeml\tank

14/09/2022  11:17    <DIR>          .
14/09/2022  11:17    <DIR>          ..
14/09/2022  11:17    <DIR>          condivisioni
               0 File              0 byte

 Directory di J:\myeml\tank\condivisioni

14/09/2022  11:17    <DIR>          .
14/09/2022  11:17    <DIR>          ..
14/09/2022  11:29    <DIR>          .cestino
14/09/2022  11:29    <DIR>          Utenti
               0 File              0 byte

 Directory di J:\myeml\tank\condivisioni\.cestino

14/09/2022  11:29    <DIR>          .
14/09/2022  11:29    <DIR>          ..
14/09/2022  11:17    <DIR>          administrator

Here a successful extraction of ~2.000 long eml (with utf command you can quick-and-dirty check for long filenames)

J:\>zpaqfranz utf myeml
zpaqfranz v55.15m-experimental-JIT-L archiver, SFX64 v55.1, (Sep 13 2022)
Fix UTF-8 dry run (because no -kill)

UTF-8 dirs:         1.875
UTF-8 files:       57.528
LONG  files:        2.052

*** WINDOWS WARNING *** found long path. Suggestion: use -longpath switch

3.031 seconds (000:00:03)  (all OK)

Please test the attached long255.zpaq

C:\zpaqfranz>zpaqfranz x long255.zpaq -to z:\test-if-extracting-ok -longpath -verbose
zpaqfranz v55.15m-experimental-JIT-L archiver, SFX64 v55.1, (Sep 13 2022)
franz:-verbose Long path (on Windows)
31876: INFO: setting Windows' long filenames
long255.zpaq:
1 versions, 12 files, 3 fragments, 3 blocks, 1.676 bytes (1.64 KB)
Extracting 27 bytes (27.00 B) in 4 files (8 folders) with 32 threads

0.016 seconds (00:00:00)  (all OK)

You should get a longpath without any problem

C:\zpaqfranz>rd z:\test-if-extracting-ok /s
z:\test-if-extracting-ok, Procedere con l'operazione (S/N)? s
Il percorso z:\test-if-extracting-ok\temporaneo\alongpath\withalongsubfolder\makinganothersubfolder\this-time-with-another-subfolder\another-sub-folder-just-to-make-longer\and-now-more-and-more-just-to-test-zpaqfranz\this-is-a-single-long-chars-01234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789.eml è troppo lungo.

long255.zip

You can delete the testfolder with the rd command (on Windows)

C:\zpaqfranz>zpaqfranz rd z:\test-if-extracting-ok -longpath -kill -force
zpaqfranz v55.15m-experimental-JIT-L archiver, SFX64 v55.1, (Sep 13 2022)
franz:do a wet run! (-kill) Long path (on Windows)
Remove Directory                                                                                      
==================================================================================================
Files 1 for 7 bytes (7.00 B) longpath 1

To confirm a dangerous command
>>> Remove folder WITH subfolders?
enter EXACTLY the capcha, then press CR (return)
Entering anything else will quit.

Captcha to continue:     y
y
Captcha OK

OK: folder 0 does not seems to exist anymore //?/z:/test-if-extracting-ok

1.859 seconds (000:00:01)  (all OK)

C:\zpaqfranz>zpaqfranz utf z:\test-if-extracting-ok
zpaqfranz v55.15m-experimental-JIT-L archiver, SFX64 v55.1, (Sep 13 2022)
Fix UTF-8 dry run (because no -kill)

UTF-8 dirs:             0
UTF-8 files:            0
LONG  files:            2

*** WINDOWS WARNING *** found long path. Suggestion: use -longpath switch

0.000 seconds (00:00:00)  (all OK)

Please let me know if you have any difficulties. Maybe you use some kind of "strange" Linux path? A symlink, a relative path?

If you get something like this

17451: path not found            //?/z:/long/./alongpath/withalongsubfolder/makinganothersubfolder/this-time-with-another-subfolder/thelast-folder-now-we-start-with-files/file-name-very-long-to-get-a-path-but-NOT-longer-than-255-chars-lets-see-if-it-is-handled-ok.eml

it is by relative paths (.)
Fixing...

55_15n.zip This pre-release build sould take care of relative paths in *nix extracted the -verbose switch (in x, extracting) will show some infos on filenames (slows down a bit)

Cannot handle (yet) extracting on Windows UNC (\franzk\something) -longpath

Closing without further reporting

fcorbelli / zpaqfranz

Windows -longpath switch, inconsistently reports number of files #30