search

mcmilk / 7-Zip-zstd

7-Zip with support for Brotli, Fast-LZMA2, Lizard, LZ4, LZ5 and Zstandard
https://mcmilk.de/projects/7-Zip-zstd/
Other
5.22k stars 303 forks source link

update performance analysis ? #65

Closed stonebig closed 5 years ago

stonebig commented 5 years ago

hi,

I would be interested in a refresh of performance analysis.

Indeed, it's not clear what zstd-1.3.7+ compression level needs to be to match 7zip default (lzma2 level 5 ?) compression within 3%.

stonebig commented 5 years ago

Hum, apparently level 20 is needed.

mcmilk commented 5 years ago

I will update the graphs in 2019 ;-)

stonebig commented 5 years ago

Thanks !

stonebig commented 5 years ago

When not de-compressing one big file, the bottleneck is no more the de-compressing speed. it's a 10 seconds (the branded speed of zstandard) to 10 minutes difference.

mcmilk commented 5 years ago

Yes, I wanted also faster compression methods for backup of computers via USB3 HDD's ... this was to slow with Standard 7-Zip, even the deflate encoder was not okay for this scenario ... this was the main reason for 7-Zip ZS.

mcmilk commented 5 years ago

Just this:

At environment this:
Build worker image: VS2015
Clone directory: c:\build

At Build this:
Type: "Script"
Content:
cd %APPVEYOR_BUILD_FOLDER%\CPP
call appveyor.cmd
stonebig commented 5 years ago

Thanks ! now I have a "7z.sfx" of:

==> do I guess right that I can:

maybe it's un-needed trick ?

mcmilk commented 5 years ago

The 32-bit "7z.sfx" should also work nicely for 64-bit systems - so this should be not problem

In general - the size for the dot.sfx files is a bit big, but in some 1.x.x version of zstd there was a need for compression functions in the de-compression code... (this may not needed anymore, Yann wanted to clean up that...)

Maybe you have the time for checking that, just remove all /uncompressed/*.obj functions of zstd in the makefile of the two sfx files.... and when you are sure you will only use the zstd stuff, remove also the lzma object files from the makefile... than the size will be even smaller... but mostly, some 100kb more size isn't really a problem for current setup files I think.. ;-)

stonebig commented 5 years ago

Thanks for the 32 bit confirmation.

For size, I try to reduce the global size of 7z.exe + 7z.dll + 7.sfx, by not incorporating brotli+lz5+lizard. I'll try your next trick after.

Would it be easy to patch 7zip to allow the "graphic" 7z.sfx auto-installer to work optionnaly in silent mode (no screen nor progress display), via a command line, like a /S or other -silent option ? It would make it equivalent to "nsis" or "inno" installers, for my usage.

stonebig commented 5 years ago

hum, it's a big number of files to remove a codec from final inclusion.

CPP/7zip/Archive/7z/7zHandler.cpp
CPP/7zip/UI/GUI/CompressDialog.cpp
CPP/7zip/Bundles/Alone/makefile
CPP/7zip/Bundles/Format7z/makefile
CPP/7zip/Bundles/Format7zExtract/makefile
CPP/7zip/Bundles/Format7zF/makefile
CPP/7zip/Bundles/Format7zUSB/makefile
CPP/7zip/Bundles/SFXCon/makefile
CPP/7zip/Bundles/SFXWin/makefile
CPP/7zip/7zip.mak

Apparently, maybe just touching 7zip.mak shall do the trick, but not fully:

brotli-mt_compress.c
../../../../C/zstdmt\brotli-mt_common.c(12): fatal error C1083: Cannot open include file: 'encode.h': No such file or directory
...
   Creating library o\7zxa.lib and object o\7zxa.exp
brotli-mt_decompress.obj : error LNK2001: unresolved external symbol @BrotliDecoderDecompress@16
stonebig commented 5 years ago

7zip ZS with just flzma2 and zstd is much smaller, now only 33% bigger than original, but 7z.sfx didn't shrink.

removing some 'unused" compressing zstd obj from the 7z.sfx makefile doesn't change the size of 7z.sfx... maybe the compiler may have done the clean-up automatically (or I mistaken again with makefiles) ?

https://github.com/stonebig/7-Zip-zstd/commit/08685ce627a1dcc955f9c546669c970a80518ae7

stonebig commented 5 years ago

I don't catch why:

==> why is 7z ZS version of 7z.sfx 60K bigger than expected (or let say, ideal) ?

mcmilk commented 5 years ago

Yes, I also don't know why. I also created an Makefile without Ppmd, LZMA and so on, this 7z.sfx is also about 300KB ... no sure why ;)

stonebig commented 5 years ago

when I try on the 32 bit version to compact in level 21 a directory that is '308 folders, 3689 files, 67500356 bytes (65 MiB)', I get immediately a 'System ERROR: Error not specified'

it doesn't happen with level 21. Could it be a bug of zstd-1.3.8 ?

mcmilk commented 5 years ago

No, it's some error while compressing. Igor told me to remove the messagebox which exactly shows what error there is.... and I didn't find a why to have a nice error message :/ (ENOMEM is catched)

You can catch the error code (and text) by enabling DEBUG in this file: CPP/7zip/Compress/ZstdEncoder.cpp

There you will find these lines:

      err = ZSTD_compressStream2(_ctx, &outBuff, &inBuff, ZSTD_todo);
      if (ZSTD_isError(err)) return E_FAIL;

Change them to this:

      err = ZSTD_compressStream2(_ctx, &outBuff, &inBuff, ZSTD_todo);
      if (ZSTD_isError(err)) {
        printf("zstd compression error: %s\n", ERR_getErrorString(err));
        fflush(stdout);
        return E_FAIL;
      }

You will have to use 7z.exe instead of the GUI... or you have to use some MessageBox() ...

stonebig commented 5 years ago

as it's close to 64 Mo (67108864), I would suspect something like a too small thing for level 22.

Hermholtz commented 5 years ago

I tried co compress ASP.NET Core MVC repo (22 MB) using ZStandard level 22, 7-Zip 18.06 ZS v 1.3.8 R1 (x64), also got error immediately when the compression started.

image

stonebig commented 5 years ago

experimenting with facebook provided zstd command line:

D:\zstd-v1.3.8-win32\zstd.exe -19 D:\mcstonebig\zstd-v1.3.8-win32\README.md -o test.zst
D:\mcstonebig\zstd-v1.3.8-win32\README.md : 37.05%   (  3088 =>   1144 bytes, test.zst)

D:\zstd-v1.3.8-win32\zstd.exe -22 D:\mcstonebig\zstd-v1.3.8-win32\README.md -o test22.zst
Warning : compression level higher than max, reduced to 19
D:\mcstonebig\zstd-v1.3.8-win32\README.md : 37.05%   (  3088 =>   1144 bytes, test22.zst)

but with 7Z zstd integration, compressing only the README.md, with 7z.Sfx or not, the error message rises. ==> I think I can trigger it also with long:30 ,whatever the official level

Hermholtz commented 5 years ago

You need to pass --ultra to enable levels 20-22.

stonebig commented 5 years ago

A yes, I didn't notice the warning.

With '--ultra' it works without warning message, but I need to compress zstd.exe itself to verify it makes a difference in output size.

Hermholtz commented 5 years ago

What's funny, error from the screenshot above stopped showing. It now only occurs when I select 12 or more CPU threads (out of 8 that I actually have). However CPU usage suggests that only 1 thread is doing work. Don't know why it happens and how to interpret this.

stonebig commented 5 years ago

Do you have 16 Go ram ? it may be related to a bad check of Ram needed

Hermholtz commented 5 years ago

Yes, it's 16 GiB of RAM, but only 7.3 available at the moment. I suspected it may be related to memory, will try to observe it further.

stonebig commented 5 years ago

So I fail with 8go / 2+2 cpus, you fail with 16go / 4+4 cpus. someone may have done a bad test in checking memory needs,.

mcmilk commented 5 years ago

I will try to get the error messages of zstd lib to the failure message of 7-zip ...

stonebig commented 5 years ago

I notice that Igor has noticed Zstd. https://sourceforge.net/p/sevenzip/discussion/45798/thread/068e2f00b7/#c38b

2) zstandard - not now. = Yes later (in Mathematical term) It would be so nice if at least the decompressing bit could make it in default 7zip