search

inikep / lizard

Lizard (formerly LZ5) is an efficient compressor with very fast decompression. It achieves compression ratio that is comparable to zip/zlib and zstd/brotli (at low and medium compression levels) at decompression speed of 1000 MB/s and faster.
Other
644 stars 40 forks source link

Enlarging the Window #13

Closed Sanmayce closed 3 years ago

Sanmayce commented 6 years ago

Hi inikep, as a future enhancement, could you consider adding bigger windows than 256MB to your superb Lizard?! I just did it, despite the general trend to work with smaller windows (and more threads) and the lack of computational power for gigantic windows. The toy is called 'Ryuugan+' - a strong and fully functional one, it inherits all the good stuff from 'Washigan+' - my mainstay. Haven't been happy with the limited Eagleeye's 512MB sliding window, so the successor Dragoneye adds to all previous windows and the 37bit one - 128GB. The underlying notion is to have (in the future) one compressor able to see deeply in the whole English Wikipedia XML dump from start to the end - it steadily is growing 2GB per year, now it is 60GB.

The idea is, when the right time comes, the compile&source of such performer to be available - not to rush and write when the machine nomenclature matches the requirements for 15GB+60GB=75GB RAM. Looking back, say, 10 years, many tools are blind to the rich resources that machines of today offer. It is good, for a change, to have the software ready beforehand, no?

NakamichiRyuugan+(32bit_64bit_executables_source).zip 5.87 MB (6,160,495 bytes): https://drive.google.com/file/d/0BzKgu_YpO6uZYnpGY0d0NHNHQWc/view?usp=sharing

Booklet_Ryuugan+.pdf 763 KB (781,446 bytes): https://drive.google.com/file/d/0BzKgu_YpO6uZb1VsSC02cE9WUlk/view?usp=sharing

F:\TEXTUAL_MADNESS\OLD_DOCs\Nakamichi_Ryuugan+_(32bit_64bit_source)>"Nakamichi_Ryuugan+_(1xQWORD+1xXMM)_Intel_15.0_32bit_SSE41.exe" University_of_Canterbury_The_Calgary_Corpus.tar
Nakamichi 'Ryuugan+', written by Kaze, based on Nobuo Ito's LZSS source, babealicious suggestion by m^2 enforced, muffinesque suggestion by Jim Dempsey enforced.
Note0: Nakamichi 'Dragoneye' is 100% FREE, licenseless that is.
Note1: Hamid Buzidi's LzTurbo ([a] FASTEST [Textual] Decompressor, Levels 19/29/39) retains kingship, his TurboBench (2017-Apr-07) proves the supremacy of LzTurbo, Turbo-Amazing!
Note2: Conor Stokes' LZSSE2 ([a] FASTEST Textual Decompressor, Level 17) is embedded, all credits along with many thanks go to him.
Note3: 'Ryuugan' predecessors are Washigan, Okamigan, Zato, Tsubame, Tengu-Tsuyo, Tengu, Rakka, Kokuen, Kinroba, Yoko, Kinutora, Jiten, Butsuhira, Suiken, Keigan, Kumataka, Washi, Aratama, Hitomi, Nekomata, Kitsune, Kinezumi, Sanbashi, Kaiko, Inazuma, Zangetsu, Hanabi, Hanazakari, Sanshi and Kaidanji.
Note4: This compile can handle files up to 5120MB.
Note5: The matchfinder/memmem() is 'Railgun_Trolldom'.
Note6: Instead of '_mm_loadu_si128' '_mm_lddqu_si128' is used.
Note7: The lookahead 'Tsuyo' heuristic which looks one char ahead is applied thrice, still not strengthened, though.
Note8: The compile made 2017-May-04, the decompression time measuring is done in 16x8 passes choosing the top score from 64 back-to-back runs - the goal - to enter [maximal] Turbo Mode.
Note9: Just to reduce the codesize, the 3xQWORD become (in this compile) 1xQWORD+1xXMMWORD.
NoteA: Please send me (at sanmayce@sanmayce.com) decompression results obtained on machines with fast CPU-RAM subsystems.
Current priority class is REALTIME_PRIORITY_CLASS.
Allocating Source-Buffer 3 MB ...
Allocating Source-Buffer 3 MB (REVERSED) ...
Allocating Target-Buffer 35 MB ...
Allocating Verification-Buffer 3 MB ...
Compressing 3,265,536 bytes ...
/; Each rotation means 64KB are encoded; Speed: 000,588B/s; Done 100%; Compression Ratio: 2.67:1; Matches(24/32/48): 4,233/317/1,483; Washigan Matches(5bytes offsets): 8
NumberOfFullLiterals (lower-the-better): 8276
Tsuyo_HEURISTIC_APPLIED_thrice_back-to-back: 14
NumberOf(Tiny)Matches[Micro]Window (4)[16B]: 3852
NumberOfMatches[Bheema]Window [128GB window]: 1
RAM-to-RAM performance: 588 B/s.
Compressed to 1,223,800 bytes.
Source-file-Hash(FNV1A_YoshimitsuTRIAD) = 0xe6d1,9b78
Target-file-Hash(FNV1A_YoshimitsuTRIAD) = 0xeacf,b4c9
Decompressing 1,223,800 (being the compressed stream) bytes ...
RAM-to-RAM performance: 494 MB/s.
Verification (input and output sizes match) OK.
Verification (input and output blocks match) OK.
LZSSE2: Compressing with LZSSE2 (level 17) 3,265,536 bytes ...
LZSSE2: Compressed to 1,142,536 bytes.
LZSSE2: RAM-to-RAM performance: 376 KB/s.
LZSSE2: Decompressing 1,142,536 bytes (being the compressed stream) ...
LZSSE2: RAM-to-RAM performance: 671 MB/s.
LZSSE2: Verification (input and output sizes match) OK.
LZSSE2: Verification (input and output blocks match) OK.
LZSSE2 vs Nakamichi 'Ryuugan', c.size: 0.93x
LZSSE2 vs Nakamichi 'Ryuugan', d.rate: 1.36x

07/02/2017  10:06 AM         3,265,536 University_of_Canterbury_The_Calgary_Corpus.tar
07/02/2017  10:06 AM           667,746 University_of_Canterbury_The_Calgary_Corpus.tar.cc.nz
07/02/2017  10:06 AM           768,579 University_of_Canterbury_The_Calgary_Corpus.tar.co.nz
07/02/2017  10:06 AM         1,142,536 University_of_Canterbury_The_Calgary_Corpus.tar.L17.LZSSE2
07/02/2017  10:06 AM           853,356 University_of_Canterbury_The_Calgary_Corpus.tar.L9Dict512.xz
07/02/2017  10:06 AM         1,028,558 University_of_Canterbury_The_Calgary_Corpus.tar.M49Block256.lz5
07/02/2017  10:06 AM         1,079,861 University_of_Canterbury_The_Calgary_Corpus.tar.method29.zpaq
07/02/2017  10:06 AM           683,957 University_of_Canterbury_The_Calgary_Corpus.tar.method59.zpaq
07/02/2017  10:06 AM         1,015,733 University_of_Canterbury_The_Calgary_Corpus.tar.MX9.zip
07/02/2017  10:06 AM           853,789 University_of_Canterbury_The_Calgary_Corpus.tar.MX9Dict512.7z
08/27/2017  04:42 PM         1,223,800 University_of_Canterbury_The_Calgary_Corpus.tar.Nakamichi
07/02/2017  10:06 AM           784,803 University_of_Canterbury_The_Calgary_Corpus.tar.O16.PPMd_varI
07/02/2017  10:06 AM           796,337 University_of_Canterbury_The_Calgary_Corpus.tar.O6.PPMd_varI
07/02/2017  10:06 AM           825,124 University_of_Canterbury_The_Calgary_Corpus.tar.ST6Block512.bsc

04/06/2017  08:44 PM           152,089 alice29.txt
08/27/2017  12:48 PM            56,526 alice29.txt.L17.LZSSE2
08/27/2017  12:48 PM            67,579 alice29.txt.Nakamichi

04/06/2017  08:44 PM           260,569 Fahrenheit_451_-_Ray_Bradbury.txt
08/27/2017  12:50 PM           100,387 Fahrenheit_451_-_Ray_Bradbury.txt.L17.LZSSE2
08/27/2017  12:50 PM           115,818 Fahrenheit_451_-_Ray_Bradbury.txt.Nakamichi

04/06/2017  08:44 PM         1,820,160 Fleurs_du_mal.tar
08/27/2017  10:58 AM           557,422 Fleurs_du_mal.tar.L17.LZSSE2
08/27/2017  10:58 AM           414,512 Fleurs_du_mal.tar.Nakamichi

04/06/2017  08:44 PM             8,798 IBM_Deep_Blue_vs_Kasparov.pgn
08/27/2017  12:49 PM             4,803 IBM_Deep_Blue_vs_Kasparov.pgn.L17.LZSSE2
08/27/2017  12:49 PM             5,153 IBM_Deep_Blue_vs_Kasparov.pgn.Nakamichi

04/06/2017  08:44 PM         3,680,256 Resident_Evil_(10-books).tar
08/27/2017  02:49 PM         1,343,243 Resident_Evil_(10-books).tar.L17.LZSSE2
08/27/2017  02:49 PM         1,289,461 Resident_Evil_(10-books).tar.Nakamichi

04/06/2017  08:44 PM           698,072 The_Death_Ship_-_B.Traven.pdf.txt
08/27/2017  12:55 PM           258,683 The_Death_Ship_-_B.Traven.pdf.txt.L17.LZSSE2
08/27/2017  12:55 PM           277,889 The_Death_Ship_-_B.Traven.pdf.txt.Nakamichi

07/02/2017  10:06 AM         4,445,260 The_Project_Gutenberg_EBook_of_The_King_James_Bible_kjv10.txt
08/27/2017  12:44 PM         1,390,894 The_Project_Gutenberg_EBook_of_The_King_James_Bible_kjv10.txt.L17.LZSSE2
08/27/2017  12:44 PM         1,356,427 The_Project_Gutenberg_EBook_of_The_King_James_Bible_kjv10.txt.Nakamichi
Sanmayce commented 6 years ago

Thanks to Stephan Busch here comes the performance of the forgotten and the very first Nakamichi - 'Kaidanji' - the simplest and weakest of all his brethren.

Two datasets were used: www.squeezechart.com_APP.tar www.squeezechart.com_SOURCES.tar

Wanted also to share the 'Complete_works_of_Fyodor_Dostoyevsky_in_15volumes(Russian).tar' benchmark with my monster 'Ruugan+', but it is only 62% done.

// Testmachine: Laptop 'Compressionette' Lenovo Ideapad 310; i5-7200u @2.5GHz; 8GB DDR4 @1066MHz (2133MHz) CL15 CR2T; L2 cache: 2x256KB; L3 cache: 3MB

/*
D:\_KAZE_workfield\more\smashshop_2017-Jul-23_Kaidanji>lzbench171 -c4 -i1,15 -o3 -ebrotli24,11/tornado,16/blosclz,9/brieflz/crush,2/csc,5/density,3/fastlz,2/gipfeli/zstd,12,22/zstd24,12,22/lzo1b,999/lzham,4/lzham24,4/libdeflate,12/lz4/lz4hc,10,12/lizard,19,29,39,49/lzf,1/lzfse/lzg,9/lzham,1/lzjb/lzlib,9/lzma,9/lzrw,5/lzsse2,17/lzsse4,17/lzsse8,17/lzvn/pithy,9/quicklz,3/snappy/slz_zlib,3/ucl_nrv2b,9/ucl_nrv2d,9/ucl_nrv2e,9/xpack,9/xz,9/yalz77,12/yappy,99/yappy,9999/zlib,9/zling,4/shrinker/wflz/lzmat www.squeezechart.com_APP.tar
lzbench 1.7.1 (64-bit Windows)   Assembled by P.Skibinski
Compressor name         Compress. Decompress.  Orig. size  Compr. size  Ratio Filename
memcpy                  10611 MB/s 10618 MB/s   338329600    338329600 100.00 www.squeezechart.com_APP.tar
...
The results sorted by column number 4:
Compressor name         Compress. Decompress.  Orig. size  Compr. size  Ratio Filename
csc 2016-10-13 -5        3.30 MB/s    53 MB/s   338329600     86918631  25.69 www.squeezechart.com_APP.tar
lzma 16.04 -9            2.28 MB/s    59 MB/s   338329600     89255359  26.38 www.squeezechart.com_APP.tar
xz 5.2.3 -9              2.43 MB/s    52 MB/s   338329600     89257952  26.38 www.squeezechart.com_APP.tar
lzlib 1.8 -9             1.65 MB/s    44 MB/s   338329600     89480091  26.45 www.squeezechart.com_APP.tar
brotli24 2017-03-10 -11  0.34 MB/s   264 MB/s   338329600     92416117  27.32 www.squeezechart.com_APP.tar
lzham 1.0 -d26 -4        1.17 MB/s   176 MB/s   338329600     94479460  27.93 www.squeezechart.com_APP.tar
lzham24 1.0 -4           1.39 MB/s   177 MB/s   338329600     95926322  28.35 www.squeezechart.com_APP.tar
zstd 1.1.4 -22           1.87 MB/s   552 MB/s   338329600     96033510  28.38 www.squeezechart.com_APP.tar
lzham 1.0 -d26 -1        2.08 MB/s   174 MB/s   338329600     97525827  28.83 www.squeezechart.com_APP.tar
tornado 0.6a -16         2.11 MB/s   127 MB/s   338329600     97709585  28.88 www.squeezechart.com_APP.tar
zstd24 1.1.4 -22         2.14 MB/s   551 MB/s   338329600     98039421  28.98 www.squeezechart.com_APP.tar
zstd24 1.1.4 -12           20 MB/s   746 MB/s   338329600    112412610  33.23 www.squeezechart.com_APP.tar
zstd 1.1.4 -12             20 MB/s   745 MB/s   338329600    112412610  33.23 www.squeezechart.com_APP.tar
lizard 1.0 -49           1.70 MB/s  1399 MB/s   338329600    112574251  33.27 www.squeezechart.com_APP.tar
lizard 1.0 -29           1.82 MB/s  2034 MB/s   338329600    119044334  35.19 www.squeezechart.com_APP.tar
xpack 2016-06-02 -9        18 MB/s   599 MB/s   338329600    119504220  35.32 www.squeezechart.com_APP.tar
zling 2016-01-10 -4        24 MB/s   141 MB/s   338329600    121553453  35.93 www.squeezechart.com_APP.tar
crush 1.0 -2             1.25 MB/s   282 MB/s   338329600    124628758  36.84 www.squeezechart.com_APP.tar
libdeflate 0.7 -12       6.74 MB/s   515 MB/s   338329600    130224560  38.49 www.squeezechart.com_APP.tar
lzfse 2017-03-08           55 MB/s   648 MB/s   338329600    130439523  38.55 www.squeezechart.com_APP.tar
ucl_nrv2e 1.03 -9        2.22 MB/s   293 MB/s   338329600    130656237  38.62 www.squeezechart.com_APP.tar
ucl_nrv2d 1.03 -9        2.23 MB/s   295 MB/s   338329600    131137069  38.76 www.squeezechart.com_APP.tar
ucl_nrv2b 1.03 -9        2.24 MB/s   285 MB/s   338329600    132076913  39.04 www.squeezechart.com_APP.tar
zlib 1.2.11 -9             10 MB/s   266 MB/s   338329600    134316446  39.70 www.squeezechart.com_APP.tar
lzmat 1.01                 27 MB/s   314 MB/s   338329600    142517331  42.12 www.squeezechart.com_APP.tar
lizard 1.0 -39           3.19 MB/s  1811 MB/s   338329600    143654666  42.46 www.squeezechart.com_APP.tar
lzvn 2017-03-08            46 MB/s   890 MB/s   338329600    144713323  42.77 www.squeezechart.com_APP.tar
brieflz 1.1.0             119 MB/s   190 MB/s   338329600    147997915  43.74 www.squeezechart.com_APP.tar
lzg 1.0.8 -9             0.92 MB/s   503 MB/s   338329600    148887434  44.01 www.squeezechart.com_APP.tar
lzo1b 2.09 -999            10 MB/s   653 MB/s   338329600    151105525  44.66 www.squeezechart.com_APP.tar
yalz77 2015-09-19 -12      13 MB/s   433 MB/s   338329600    151185159  44.69 www.squeezechart.com_APP.tar
lz4hc 1.7.5 -12          2.41 MB/s  2626 MB/s   338329600    152528107  45.08 www.squeezechart.com_APP.tar
lz4hc 1.7.5 -10            10 MB/s  2599 MB/s   338329600    153025251  45.23 www.squeezechart.com_APP.tar
lizard 1.0 -19           3.36 MB/s  3008 MB/s   338329600    153055244  45.24 www.squeezechart.com_APP.tar
quicklz 1.5.0 -3           48 MB/s   705 MB/s   338329600    153348434  45.33 www.squeezechart.com_APP.tar
lzsse8 2016-05-14 -17    0.75 MB/s  3016 MB/s   338329600    154311619  45.61 www.squeezechart.com_APP.tar
lzsse4 2016-05-14 -17    0.81 MB/s  2684 MB/s   338329600    157425621  46.53 www.squeezechart.com_APP.tar
lzsse2 2016-05-14 -17    0.68 MB/s  1950 MB/s   338329600    159377622  47.11 www.squeezechart.com_APP.tar
pithy 2011-12-24 -9        54 MB/s  1363 MB/s   338329600    163840094  48.43 www.squeezechart.com_APP.tar
gipfeli 2016-07-13        214 MB/s   572 MB/s   338329600    166986298  49.36 www.squeezechart.com_APP.tar
slz_zlib 1.0.0 -3         228 MB/s   280 MB/s   338329600    171463349  50.68 www.squeezechart.com_APP.tar
lzrw 15-Jul-1991 -5       110 MB/s   402 MB/s   338329600    174466937  51.57 www.squeezechart.com_APP.tar
density 0.12.5 beta -3    164 MB/s   250 MB/s   338329600    179346920  53.01 www.squeezechart.com_APP.tar
yappy 2014-03-22 -9999     46 MB/s  2356 MB/s   338329600    180200889  53.26 www.squeezechart.com_APP.tar
yappy 2014-03-22 -99       67 MB/s  2351 MB/s   338329600    180334284  53.30 www.squeezechart.com_APP.tar
lz4 1.7.5                 563 MB/s  2544 MB/s   338329600    180522563  53.36 www.squeezechart.com_APP.tar
fastlz 0.1 -2             262 MB/s   529 MB/s   338329600    180631111  53.39 www.squeezechart.com_APP.tar
blosclz 2015-11-10 -9     241 MB/s   799 MB/s   338329600    184123276  54.42 www.squeezechart.com_APP.tar
lzf 3.6 -1                254 MB/s   601 MB/s   338329600    185285276  54.76 www.squeezechart.com_APP.tar
snappy 1.1.4              210 MB/s  1309 MB/s   338329600    186267653  55.06 www.squeezechart.com_APP.tar
wflz 2015-09-16           174 MB/s   864 MB/s   338329600    192518735  56.90 www.squeezechart.com_APP.tar
lzjb 2010                 256 MB/s   447 MB/s   338329600    207739252  61.40 www.squeezechart.com_APP.tar
Nakamichi 'Kaidanji'                3427 MB/s                225734933        ! outside lzbench !
shrinker 0.1             1860 MB/s  4832 MB/s   338329600    303935727  89.83 www.squeezechart.com_APP.tar

D:\_KAZE_workfield\more\smashshop_2017-Jul-23_Kaidanji>"turbobenchs_Official_v17.04_-_build_07_Apr_2017.exe" www.squeezechart.com_APP.tar -elibdeflate,12/oodle,19,49,89,112,114,116,118,129/lzsse2,17/lzturbo,19,12,29,22,39,32,49,59/zstd,12,22/lizard,19,29,39,49/brotli,11/lzma,9/bzip2/xpack,9/chameleon,2/density,3/lzham,4/trle/bsc,3,6/zpaq,2,5 -g -I1 -J31 -k1 -B2G
TurboBench:  - Mon Aug 28 16:08:19 2017
      C Size  ratio%     C MB/s     D MB/s   Name            File
    81382356    24.1       0.35       0.35   zpaq 5                           www.squeezechart.com_APP.tar.tbb
    86908951    25.7       1.01      62.74   lzturbo 49                       www.squeezechart.com_APP.tar.tbb
    89255363    26.4       2.27      59.16   lzma 9                           www.squeezechart.com_APP.tar.tbb
    91863102    27.2       0.61     985.04   oodle 129                        www.squeezechart.com_APP.tar.tbb
    91909435    27.2       0.72     977.90   oodle 89                         www.squeezechart.com_APP.tar.tbb
    92318757    27.3       1.47     184.16   lzham 4                          www.squeezechart.com_APP.tar.tbb
    93648185    27.7       1.37     925.20   lzturbo 39                       www.squeezechart.com_APP.tar.tbb
    95538927    28.2       0.42     462.05   oodle 19                         www.squeezechart.com_APP.tar.tbb
    95746053    28.3       0.33     265.07   brotli 11                        www.squeezechart.com_APP.tar.tbb
    96033514    28.4       1.87     552.86   zstd 22                          www.squeezechart.com_APP.tar.tbb
   103271940    30.5       0.26     519.84   xpack 9                          www.squeezechart.com_APP.tar.tbb
   106167203    31.4      31.13    1323.62   lzturbo 32                       www.squeezechart.com_APP.tar.tbb
   106824652    31.6       9.97       8.85   bsc 6                            www.squeezechart.com_APP.tar.tbb
   109919839    32.5       1.52    1454.65   lzturbo 29                       www.squeezechart.com_APP.tar.tbb
   110670368    32.7       8.58      14.83   lzturbo 59                       www.squeezechart.com_APP.tar.tbb
   110766938    32.7      16.98       9.61   bsc 3                            www.squeezechart.com_APP.tar.tbb
   112412614    33.2      20.47     744.93   zstd 12                          www.squeezechart.com_APP.tar.tbb
   114426330    33.8       0.81    3032.50   oodle 118                        www.squeezechart.com_APP.tar.tbb
   115821587    34.2       0.71    3380.28   oodle 116                        www.squeezechart.com_APP.tar.tbb
   119044338    35.2       1.79    2033.57   lizard 29                        www.squeezechart.com_APP.tar.tbb
   120596981    35.6       1.69    1818.92   lizard 49                        www.squeezechart.com_APP.tar.tbb
   121236898    35.8       4.62      78.77   zpaq 2                           www.squeezechart.com_APP.tar.tbb
   124762234    36.9       9.81      26.16   bzip2                            www.squeezechart.com_APP.tar.tbb
   125801945    37.2      31.72    1678.70   lzturbo 22                       www.squeezechart.com_APP.tar.tbb
   126865073    37.5      23.14    3251.27   oodle 114                        www.squeezechart.com_APP.tar.tbb
   130224570    38.5       7.09     495.47   libdeflate 12                    www.squeezechart.com_APP.tar.tbb
   134353989    39.7      34.10    3720.42   oodle 112                        www.squeezechart.com_APP.tar.tbb
   152212804    45.0       3.98    2375.98   oodle 49                         www.squeezechart.com_APP.tar.tbb
   152683097    45.1       1.60    3368.39   lzturbo 19                       www.squeezechart.com_APP.tar.tbb
   153055248    45.2       3.30    3022.84   lizard 19                        www.squeezechart.com_APP.tar.tbb
   153632665    45.4       3.14    2932.00   lizard 39                        www.squeezechart.com_APP.tar.tbb
   158988575    47.0      61.07    3370.44   lzturbo 12                       www.squeezechart.com_APP.tar.tbb
   159377626    47.1       0.84    1475.54   lzsse2 17                        www.squeezechart.com_APP.tar.tbb
   181127228    53.5     191.30     291.12   density 3                        www.squeezechart.com_APP.tar.tbb
   225734933                       3427      Nakamichi 'Kaidanji'             ! outside TurboBench !
   243747112    72.0     996.81    1688.07   chameleon 2                      www.squeezechart.com_APP.tar.tbb
   305965196    90.4     176.12    5283.80   trle                             www.squeezechart.com_APP.tar.tbb

D:\_KAZE_workfield\more\smashshop_2017-Jul-23_Kaidanji>lzbench171 -c4 -i1,15 -o3 -ebrotli24,11/tornado,16/blosclz,9/brieflz/crush,2/csc,5/density,3/fastlz,2/gipfeli/zstd,12,22/zstd24,12,22/lzo1b,999/lzham,4/lzham24,4/libdeflate,12/lz4/lz4hc,10,12/lizard,19,29,39,49/lzf,1/lzfse/lzg,9/lzham,1/lzjb/lzlib,9/lzma,9/lzrw,5/lzsse2,17/lzsse4,17/lzsse8,17/lzvn/pithy,9/quicklz,3/snappy/slz_zlib,3/ucl_nrv2b,9/ucl_nrv2d,9/ucl_nrv2e,9/xpack,9/xz,9/yalz77,12/yappy,99/yappy,9999/zlib,9/zling,4/shrinker/wflz/lzmat www.squeezechart.com_SOURCES.tar
lzbench 1.7.1 (64-bit Windows)   Assembled by P.Skibinski
Compressor name         Compress. Decompress.  Orig. size  Compr. size  Ratio Filename
memcpy                  10896 MB/s 10873 MB/s   209825792    209825792 100.00 www.squeezechart.com_SOURCES.tar
...
The results sorted by column number 4:
Compressor name         Compress. Decompress.  Orig. size  Compr. size  Ratio Filename
lzlib 1.8 -9             1.68 MB/s    65 MB/s   209825792     36701060  17.49 www.squeezechart.com_SOURCES.tar
xz 5.2.3 -9              2.81 MB/s    83 MB/s   209825792     36917349  17.59 www.squeezechart.com_SOURCES.tar
lzma 16.04 -9            2.55 MB/s    91 MB/s   209825792     36924693  17.60 www.squeezechart.com_SOURCES.tar
brotli24 2017-03-10 -11  0.48 MB/s   429 MB/s   209825792     37325474  17.79 www.squeezechart.com_SOURCES.tar
zstd 1.1.4 -22           1.74 MB/s   875 MB/s   209825792     38033002  18.13 www.squeezechart.com_SOURCES.tar
csc 2016-10-13 -5        3.76 MB/s    85 MB/s   209825792     38192929  18.20 www.squeezechart.com_SOURCES.tar
lzham 1.0 -d26 -4        1.43 MB/s   309 MB/s   209825792     38637032  18.41 www.squeezechart.com_SOURCES.tar
tornado 0.6a -16         2.13 MB/s   210 MB/s   209825792     38861022  18.52 www.squeezechart.com_SOURCES.tar
zstd24 1.1.4 -22         1.84 MB/s   867 MB/s   209825792     38952486  18.56 www.squeezechart.com_SOURCES.tar
lzham24 1.0 -4           1.58 MB/s   305 MB/s   209825792     39290082  18.73 www.squeezechart.com_SOURCES.tar
lzham 1.0 -d26 -1        2.70 MB/s   290 MB/s   209825792     40907583  19.50 www.squeezechart.com_SOURCES.tar
zstd 1.1.4 -12             25 MB/s  1061 MB/s   209825792     43543522  20.75 www.squeezechart.com_SOURCES.tar
zstd24 1.1.4 -12           26 MB/s  1061 MB/s   209825792     43543522  20.75 www.squeezechart.com_SOURCES.tar
zling 2016-01-10 -4        44 MB/s   232 MB/s   209825792     43869965  20.91 www.squeezechart.com_SOURCES.tar
lizard 1.0 -49           1.44 MB/s  1771 MB/s   209825792     44509333  21.21 www.squeezechart.com_SOURCES.tar
crush 1.0 -2             0.96 MB/s   439 MB/s   209825792     46567262  22.19 www.squeezechart.com_SOURCES.tar
lizard 1.0 -29           1.55 MB/s  2496 MB/s   209825792     48335185  23.04 www.squeezechart.com_SOURCES.tar
xpack 2016-06-02 -9        19 MB/s   856 MB/s   209825792     49087266  23.39 www.squeezechart.com_SOURCES.tar
lzfse 2017-03-08           61 MB/s   786 MB/s   209825792     52845473  25.19 www.squeezechart.com_SOURCES.tar
libdeflate 0.7 -12       4.98 MB/s   743 MB/s   209825792     52948822  25.23 www.squeezechart.com_SOURCES.tar
ucl_nrv2e 1.03 -9        2.36 MB/s   385 MB/s   209825792     54344570  25.90 www.squeezechart.com_SOURCES.tar
ucl_nrv2d 1.03 -9        2.39 MB/s   387 MB/s   209825792     54631206  26.04 www.squeezechart.com_SOURCES.tar
zlib 1.2.11 -9             12 MB/s   356 MB/s   209825792     55071341  26.25 www.squeezechart.com_SOURCES.tar
ucl_nrv2b 1.03 -9        2.31 MB/s   372 MB/s   209825792     55101272  26.26 www.squeezechart.com_SOURCES.tar
lizard 1.0 -39           2.71 MB/s  2016 MB/s   209825792     57541892  27.42 www.squeezechart.com_SOURCES.tar
lzg 1.0.8 -9             0.95 MB/s   695 MB/s   209825792     57625819  27.46 www.squeezechart.com_SOURCES.tar
lzmat 1.01                 33 MB/s   378 MB/s   209825792     58077448  27.68 www.squeezechart.com_SOURCES.tar
yalz77 2015-09-19 -12      23 MB/s   509 MB/s   209825792     58605626  27.93 www.squeezechart.com_SOURCES.tar
lzo1b 2.09 -999            15 MB/s   783 MB/s   209825792     60561155  28.86 www.squeezechart.com_SOURCES.tar
brieflz 1.1.0             150 MB/s   207 MB/s   209825792     60917799  29.03 www.squeezechart.com_SOURCES.tar
lzsse8 2016-05-14 -17    2.49 MB/s  3814 MB/s   209825792     61265223  29.20 www.squeezechart.com_SOURCES.tar
lzsse4 2016-05-14 -17    2.60 MB/s  3658 MB/s   209825792     61643185  29.38 www.squeezechart.com_SOURCES.tar
lz4hc 1.7.5 -12          2.77 MB/s  2537 MB/s   209825792     61805116  29.46 www.squeezechart.com_SOURCES.tar
lizard 1.0 -19           2.88 MB/s  2866 MB/s   209825792     62118651  29.60 www.squeezechart.com_SOURCES.tar
lz4hc 1.7.5 -10            11 MB/s  2502 MB/s   209825792     62247510  29.67 www.squeezechart.com_SOURCES.tar
lzsse2 2016-05-14 -17    2.48 MB/s  2919 MB/s   209825792     62657596  29.86 www.squeezechart.com_SOURCES.tar
quicklz 1.5.0 -3           51 MB/s   895 MB/s   209825792     64326171  30.66 www.squeezechart.com_SOURCES.tar
lzvn 2017-03-08            54 MB/s   976 MB/s   209825792     64822535  30.89 www.squeezechart.com_SOURCES.tar
pithy 2011-12-24 -9       393 MB/s  1720 MB/s   209825792     65840207  31.38 www.squeezechart.com_SOURCES.tar
gipfeli 2016-07-13        334 MB/s   545 MB/s   209825792     69387174  33.07 www.squeezechart.com_SOURCES.tar
slz_zlib 1.0.0 -3         237 MB/s   296 MB/s   209825792     75440572  35.95 www.squeezechart.com_SOURCES.tar
lzrw 15-Jul-1991 -5       128 MB/s   512 MB/s   209825792     75735750  36.09 www.squeezechart.com_SOURCES.tar
density 0.12.5 beta -3    153 MB/s   280 MB/s   209825792     77393218  36.88 www.squeezechart.com_SOURCES.tar
fastlz 0.1 -2             314 MB/s   548 MB/s   209825792     78515504  37.42 www.squeezechart.com_SOURCES.tar
lz4 1.7.5                 552 MB/s  2416 MB/s   209825792     78534596  37.43 www.squeezechart.com_SOURCES.tar
yappy 2014-03-22 -9999     48 MB/s  2240 MB/s   209825792     78661616  37.49 www.squeezechart.com_SOURCES.tar
yappy 2014-03-22 -99       70 MB/s  2230 MB/s   209825792     78800941  37.56 www.squeezechart.com_SOURCES.tar
snappy 1.1.4              397 MB/s  1324 MB/s   209825792     80334243  38.29 www.squeezechart.com_SOURCES.tar
blosclz 2015-11-10 -9     280 MB/s   909 MB/s   209825792     80801973  38.51 www.squeezechart.com_SOURCES.tar
lzf 3.6 -1                307 MB/s   632 MB/s   209825792     82264589  39.21 www.squeezechart.com_SOURCES.tar
wflz 2015-09-16           251 MB/s   777 MB/s   209825792     89283223  42.55 www.squeezechart.com_SOURCES.tar
lzjb 2010                 274 MB/s   457 MB/s   209825792    101949122  48.59 www.squeezechart.com_SOURCES.tar
Nakamichi 'Kaidanji'                3133 MB/s                103791792        ! outside lzbench !
shrinker 0.1              956 MB/s  2357 MB/s   209825792    166863382  79.52 www.squeezechart.com_SOURCES.tar

D:\_KAZE_workfield\more\smashshop_2017-Jul-23_Kaidanji>"turbobenchs_Official_v17.04_-_build_07_Apr_2017.exe" www.squeezechart.com_SOURCES.tar -elibdeflate,12/oodle,19,49,89,112,114,116,118,129/lzsse2,17/lzturbo,19,12,29,22,39,32,49,59/zstd,12,22/lizard,19,29,39,49/brotli,11/lzma,9/bzip2/xpack,9/chameleon,2/density,3/lzham,4/trle/bsc,3,6/zpaq,2,5 -g -I1 -J31 -k1 -B2G
TurboBench:  - Mon Aug 28 23:00:52 2017
      C Size  ratio%     C MB/s     D MB/s   Name            File
    31867486    15.2       0.39       0.38   zpaq 5                           www.squeezechart.com_SOURCES.tar.tbb
    35533637    16.9       0.89     101.83   lzturbo 49                       www.squeezechart.com_SOURCES.tar.tbb
    36924697    17.6       2.54      92.03   lzma 9                           www.squeezechart.com_SOURCES.tar.tbb
    37624056    17.9       1.21    1652.12   lzturbo 39                       www.squeezechart.com_SOURCES.tar.tbb
    38027612    18.1       0.46     425.05   brotli 11                        www.squeezechart.com_SOURCES.tar.tbb
    38033006    18.1       1.76    1045.51   zstd 22                          www.squeezechart.com_SOURCES.tar.tbb
    38205054    18.2       1.63     323.68   lzham 4                          www.squeezechart.com_SOURCES.tar.tbb
    38240952    18.2       0.42     735.38   oodle 19                         www.squeezechart.com_SOURCES.tar.tbb
    38286333    18.2       0.72    1269.52   oodle 129                        www.squeezechart.com_SOURCES.tar.tbb
    38301672    18.3       0.86    1262.60   oodle 89                         www.squeezechart.com_SOURCES.tar.tbb
    39763512    19.0      11.17      19.66   lzturbo 59                       www.squeezechart.com_SOURCES.tar.tbb
    39875924    19.0      17.45      12.95   bsc 6                            www.squeezechart.com_SOURCES.tar.tbb
    41614971    19.8       0.45     862.80   xpack 9                          www.squeezechart.com_SOURCES.tar.tbb
    42734064    20.4       1.35    2244.80   lzturbo 29                       www.squeezechart.com_SOURCES.tar.tbb
    43543526    20.8      24.79    1208.95   zstd 12                          www.squeezechart.com_SOURCES.tar.tbb
    43611417    20.8      56.44    1456.48   lzturbo 32                       www.squeezechart.com_SOURCES.tar.tbb
    43659756    20.8      24.98      16.21   bsc 3                            www.squeezechart.com_SOURCES.tar.tbb
    44461915    21.2      11.10      35.34   bzip2                            www.squeezechart.com_SOURCES.tar.tbb
    45142780    21.5       6.60     104.13   zpaq 2                           www.squeezechart.com_SOURCES.tar.tbb
    47697770    22.7       0.93    3297.00   oodle 118                        www.squeezechart.com_SOURCES.tar.tbb
    48335189    23.0       1.52    2483.10   lizard 29                        www.squeezechart.com_SOURCES.tar.tbb
    48386253    23.1       1.43    2308.75   lizard 49                        www.squeezechart.com_SOURCES.tar.tbb
    48570256    23.1       0.76    3555.26   oodle 116                        www.squeezechart.com_SOURCES.tar.tbb
    52812548    25.2      56.50    1884.21   lzturbo 22                       www.squeezechart.com_SOURCES.tar.tbb
    52948832    25.2       5.30     706.19   libdeflate 12                    www.squeezechart.com_SOURCES.tar.tbb
    54530384    26.0      32.36    3368.12   oodle 114                        www.squeezechart.com_SOURCES.tar.tbb
    60920223    29.0      77.00    3775.99   oodle 112                        www.squeezechart.com_SOURCES.tar.tbb
    61839546    29.5       3.97    2276.18   oodle 49                         www.squeezechart.com_SOURCES.tar.tbb
    61951466    29.5       1.39    3257.79   lzturbo 19                       www.squeezechart.com_SOURCES.tar.tbb
    62118655    29.6       2.80    2892.13   lizard 19                        www.squeezechart.com_SOURCES.tar.tbb
    62619130    29.8       2.67    2897.50   lizard 39                        www.squeezechart.com_SOURCES.tar.tbb
    62657600    29.9       2.32    2181.65   lzsse2 17                        www.squeezechart.com_SOURCES.tar.tbb
    66140228    31.5      76.02    3356.79   lzturbo 12                       www.squeezechart.com_SOURCES.tar.tbb
    77218134    36.8     168.21     312.87   density 3                        www.squeezechart.com_SOURCES.tar.tbb
   103791792                       3133      Nakamichi 'Kaidanji'             ! outside TurboBench !
   130585304    62.2    1008.99    2390.20   chameleon 2                      www.squeezechart.com_SOURCES.tar.tbb
   188695088    89.9     155.40    4919.48   trle                             www.squeezechart.com_SOURCES.tar.tbb

; 'Kaidanji' decompression loop, 4e-0e+2=66 bytes long, 19 instructions long:
; mark_description "Intel(R) C++ Compiler XE for applications running on Intel(R) 64, Version 15.0.0.108 Build 20140726";
; mark_description "-O3 -QxSSE2 -D_N_YMM -D_N_HIGH_PRIORITY -FAcs";

.B8.3::                         
  0000e 44 0f b7 0a      movzx r9d, WORD PTR [rdx]              
  00012 41 f7 c1 07 00 
        00 00            test r9d, 7                            
  00019 74 16            je .B8.5 
.B8.4::                         
  0001b 48 83 c2 02      add rdx, 2                             
  0001f 49 f7 d9         neg r9                                 
  00022 4c 03 c8         add r9, rax                            
  00025 4d 8b 11         mov r10, QWORD PTR [r9]                
  00028 4c 89 10         mov QWORD PTR [rax], r10               
  0002b 48 83 c0 08      add rax, 8                             
  0002f eb 1a            jmp .B8.6 
.B8.5::                         
  00031 c5 fe 6f 42 01   vmovdqu ymm0, YMMWORD PTR [1+rdx]      
  00036 45 0f b6 c9      movzx r9d, r9b                         
  0003a 41 c1 e9 03      shr r9d, 3                             
  0003e c5 fe 7f 00      vmovdqu YMMWORD PTR [rax], ymm0        
  00042 49 03 c1         add rax, r9                            
  00045 41 ff c1         inc r9d                                
  00048 49 03 d1         add rdx, r9                            
.B8.6::                         
  0004b 49 3b d0         cmp rdx, r8                            
  0004e 72 be            jb .B8.3 

C:\xx\more\smashshop_2017-Jul-23 dost>lzbench171 -c4 -i1,15 -o3 -ebrotli24,11/tornado,16/blosclz,9/brieflz/crush,2/csc,5/density,3/fastlz,2/gipfeli/zstd,12,22/zstd24,12,22/lzo1b,999/lzham,4/lzham24,4/libdeflate,12/lz4/lz4hc,10,12/lizard,19,29,39,49/lzf,1/lzfse/lzg,9/lzham,1/lzjb/lzlib,9/lzma,9/lzrw,5/lzsse2,17/lzsse4,17/lzsse8,17/lzvn/pithy,9/quicklz,3/snappy/slz_zlib,3/ucl_nrv2b,9/ucl_nrv2d,9/ucl_nrv2e,9/xpack,9/xz,9/yalz77,12/yappy,99/yappy,9999/zlib,9/zling,4/shrinker/wflz/lzmat "Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar"
lzbench 1.7.1 (64-bit Windows)   Assembled by P.Skibinski
Compressor name         Compress. Decompress.  Orig. size  Compr. size  Ratio Filename
memcpy                  11313 MB/s 11793 MB/s    40303104     40303104 100.00 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
...
The results sorted by column number 4:
Compressor name         Compress. Decompress.  Orig. size  Compr. size  Ratio Filename
lzlib 1.8 -9             1.09 MB/s    71 MB/s    40303104      6936312  17.21 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
lzma 16.04 -9            1.14 MB/s   106 MB/s    40303104      6941791  17.22 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
xz 5.2.3 -9              1.23 MB/s   102 MB/s    40303104      6941839  17.22 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
lzham 1.0 -d26 -4        0.87 MB/s   254 MB/s    40303104      6998371  17.36 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
zstd 1.1.4 -22           1.31 MB/s   600 MB/s    40303104      7051335  17.50 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
lzham24 1.0 -4           0.91 MB/s   255 MB/s    40303104      7083370  17.58 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
brotli24 2017-03-10 -11  0.39 MB/s   384 MB/s    40303104      7110635  17.64 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
tornado 0.6a -16         1.29 MB/s   215 MB/s    40303104      7132739  17.70 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
zstd24 1.1.4 -22         1.38 MB/s   598 MB/s    40303104      7133201  17.70 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
csc 2016-10-13 -5        1.25 MB/s    97 MB/s    40303104      7182316  17.82 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
lizard 1.0 -49           1.35 MB/s   925 MB/s    40303104      8061821  20.00 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
Nakamichi 'Ryuugan+' ICL v15.0      ???? MB/s                  ???????        ! outside lzbench !
Nakamichi 'Washigan+' ICL v15.0     1100 MB/s                  9063686        ! outside lzbench !
Nakamichi 'Washigan+' GCC v6.3.0    1140 MB/s                  9063686        ! outside lzbench !
zling 2016-01-10 -4        29 MB/s   171 MB/s    40303104      9154682  22.71 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
lzham 1.0 -d26 -1        2.24 MB/s   211 MB/s    40303104      9180529  22.78 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
crush 1.0 -2             0.07 MB/s   354 MB/s    40303104      9263238  22.98 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
zstd 1.1.4 -12           7.68 MB/s   730 MB/s    40303104      9298170  23.07 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
zstd24 1.1.4 -12         7.61 MB/s   730 MB/s    40303104      9298170  23.07 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
lizard 1.0 -29           1.39 MB/s  1235 MB/s    40303104      9438742  23.42 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
xpack 2016-06-02 -9      4.57 MB/s   928 MB/s    40303104      9948452  24.68 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
lizard 1.0 -39           4.39 MB/s  1828 MB/s    40303104     10064398  24.97 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
libdeflate 0.7 -12       3.50 MB/s  1013 MB/s    40303104     10073076  24.99 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
lzsse2 2016-05-14 -17    4.71 MB/s  3813 MB/s    40303104     10359314  25.70 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
lzsse4 2016-05-14 -17    5.60 MB/s  4060 MB/s    40303104     10404097  25.81 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
lzsse8 2016-05-14 -17    5.38 MB/s  3597 MB/s    40303104     10716708  26.59 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
zlib 1.2.11 -9           4.37 MB/s   307 MB/s    40303104     10771354  26.73 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
ucl_nrv2e 1.03 -9        0.35 MB/s   306 MB/s    40303104     11595193  28.77 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
lzo1b 2.09 -999          4.33 MB/s   579 MB/s    40303104     11728605  29.10 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
ucl_nrv2d 1.03 -9        0.34 MB/s   307 MB/s    40303104     11728710  29.10 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
lz4hc 1.7.5 -12          5.74 MB/s  2364 MB/s    40303104     11800407  29.28 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
lizard 1.0 -19           4.46 MB/s  2625 MB/s    40303104     11812204  29.31 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
ucl_nrv2b 1.03 -9        0.32 MB/s   294 MB/s    40303104     12008298  29.79 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
lz4hc 1.7.5 -10          9.01 MB/s  2356 MB/s    40303104     12065534  29.94 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
lzfse 2017-03-08           50 MB/s   548 MB/s    40303104     12239442  30.37 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
lzg 1.0.8 -9             0.18 MB/s   651 MB/s    40303104     12415466  30.81 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
lzmat 1.01                 12 MB/s   293 MB/s    40303104     12540407  31.12 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
yalz77 2015-09-19 -12      25 MB/s   312 MB/s    40303104     14715782  36.51 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
yappy 2014-03-22 -9999     56 MB/s  1921 MB/s    40303104     15109222  37.49 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
yappy 2014-03-22 -99       56 MB/s  1923 MB/s    40303104     15109577  37.49 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
brieflz 1.1.0             103 MB/s   135 MB/s    40303104     15196136  37.70 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
lzvn 2017-03-08            44 MB/s   772 MB/s    40303104     15512385  38.49 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
quicklz 1.5.0 -3           37 MB/s   580 MB/s    40303104     16194509  40.18 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
gipfeli 2016-07-13        234 MB/s   447 MB/s    40303104     16243953  40.30 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
lzrw 15-Jul-1991 -5        88 MB/s   410 MB/s    40303104     16370457  40.62 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
Nakamichi 'Kaidanji'                1991 MB/s                 16823815        ! outside lzbench !
pithy 2011-12-24 -9       273 MB/s  1337 MB/s    40303104     17417731  43.22 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
density 0.12.5 beta -3    392 MB/s   353 MB/s    40303104     17802720  44.17 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
lz4 1.7.5                 317 MB/s  2059 MB/s    40303104     19276004  47.83 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
snappy 1.1.4              243 MB/s   846 MB/s    40303104     19976663  49.57 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
fastlz 0.1 -2             241 MB/s   392 MB/s    40303104     20596666  51.10 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
blosclz 2015-11-10 -9     208 MB/s   596 MB/s    40303104     20867150  51.78 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
shrinker 0.1              297 MB/s   555 MB/s    40303104     20874796  51.79 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
lzf 3.6 -1                231 MB/s   482 MB/s    40303104     20904695  51.87 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
slz_zlib 1.0.0 -3         177 MB/s   210 MB/s    40303104     21287387  52.82 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
lzjb 2010                 177 MB/s   287 MB/s    40303104     25454117  63.16 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
wflz 2015-09-16           189 MB/s   469 MB/s    40303104     26169076  64.93 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar

C:\xx\more\smashshop_2017-Jul-23 dost>"turbobenchs_Official_v17.04_-_build_07_Apr_2017.exe" "Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar" -elibdeflate,12/oodle,19,49,89,112,114,116,118,129/lzsse2,17/lzturbo,19,12,29,22,39,32,49,59/zstd,12,22/lizard,19,29,39,49/brotli,11/lzma,9/bzip2/xpack,9/chameleon,2/density,3/lzham,4/trle/bsc,3,6/zpaq,2,5 -g -I1 -J31 -k1 -B2G
TurboBench:  - Tue Aug 01 04:03:32 2017
      C Size  ratio%     C MB/s     D MB/s   Name            File
     5446232    13.5       7.99      17.94   lzturbo 59                       Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
     5881208    14.6       0.42       0.41   zpaq 5                           Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
     6808608    16.9      27.11      10.62   bsc 6                            Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
     6827650    16.9       0.77     108.26   lzturbo 49                       Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
     6941795    17.2       1.13     105.79   lzma 9                           Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
     6996681    17.4       0.91     281.38   lzham 4                          Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
     7011119    17.4       0.90     810.98   lzturbo 39                       Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
     7051339    17.5       1.31     602.51   zstd 22                          Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
     7125187    17.7      10.36      27.05   bzip2                            Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
     7238603    18.0       0.59     680.56   oodle 129                        Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
     7238603    18.0       0.78     669.06   oodle 89                         Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
     7392071    18.3       0.28     439.01   oodle 19                         Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
     7405445    18.4       0.40     479.31   brotli 11                        Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
     7951427    19.7       0.96    1375.71   lzturbo 29                       Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
     8984648    22.3       4.42      86.71   zpaq 2                           Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
     9001114    22.3       1.36    1073.27   lizard 49                        Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
     ???????                       ????      Nakamichi 'Ryuugan+' ICL v15.0   ! outside TurboBench !
     9063686                       1100      Nakamichi 'Washigan+' ICL v15.0  ! outside TurboBench !
     9063686                       1140      Nakamichi 'Washigan+' GCC v6.3.0 ! outside TurboBench !
     9221434    22.9       1.23     752.75   xpack 9                          Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
     9298174    23.1       7.65     765.67   zstd 12                          Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
     9314841    23.1      35.07    1045.04   lzturbo 32                       Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
     9438746    23.4       1.39    1244.13   lizard 29                        Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
     9600504    23.8      24.27      18.49   bsc 3                            Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
     9847862    24.4       0.95    1803.10   oodle 118                        Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
    10073086    25.0       3.66     925.78   libdeflate 12                    Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
    10359314                       4300      LZSSE2 17 ICL v15.0 AVX          ! outside TurboBench !
    10359318    25.7       4.59    2789.04   lzsse2 17                        Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
    11806328    29.3       1.10    3041.11   lzturbo 19                       Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
    11810189    29.3       2.94    1773.32   oodle 49                         Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
    11812208    29.3       4.43    2659.35   lizard 19                        Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
    12052393    29.9       4.39    2580.82   lizard 39                        Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
    12459957    30.9       0.57    2302.64   oodle 116                        Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
    12677081    31.5      19.70    2575.58   oodle 114                        Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
    13042131    32.4      35.44    1207.58   lzturbo 22                       Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
    13181552    32.7      56.94    3222.46   lzturbo 12                       Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
    16823815                       1991      Nakamichi 'Kaidanji'             ! outside TurboBench !
    17802562    44.2     404.52     399.85   density 3                        Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
    17972414    44.6      59.54    3181.60   oodle 112                        Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
    21693398    53.8    1673.31    3515.18   chameleon 2                      Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
    40278349    99.9     194.71    2054.61   trle                             Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.tbb
*/
inikep commented 6 years ago

Sorry, but it's not possible to add window/offsets bigger than 16 MB (24-bit) without breaking backward compatibility. It can be done in a fork of lizard.

Sanmayce commented 6 years ago

Wanted once and for all to have the baseline for one big textual (XML) corpus, since it is not really deduplicative it is a good bench for seeing how different windows (16MB, 64MB, 512MB) affect the ratio, memory usage and timing.

Testmachine: Laptop 'Compressionette' Lenovo Ideapad 310; i5-7200u @2.5GHz; 8GB DDR4 @1066MHz (2133MHz) CL15 CR2T; L2 cache: 2x256KB; L3 cache: 3MB

Quick roster 'Single-Threaded' for English Wikipedia 2017-Jan-01:

---------------------------------------------------------------------------------------------------------
| Compressor   | Options                          | Total Compression Time | Footprint |           Size |
---------------------------------------------------------------------------------------------------------
| zpaq64       | -method 59 -threads 1            |        207,484 seconds |  6,344 MB |  7,778,448,878 |
| bsc          | -b512 -m6 -cp -Tt                |          4,097 seconds |  2,608 MB |  9,584,990,892 |
| xz           | -z -k -f -9 -e -v -v --threads=1 |         53,210 seconds |    676 MB | 10,385,748,048 |
| zstd         | --ultra -22 -T1                  |         54,975 seconds |    773 MB | 10,617,443,916 |
| zpaq64       | -method 29 -threads 1            |         18,017 seconds |  3,598 MB | 12,421,086,486 |
| lz5          | -49 -B7 -v                       |         47,045 seconds |    497 MB | 13,859,918,784 |
| rz           | -d 64m                           |       140,882+ seconds |    889 MB |  9,064,770,314 |
| uncompressed |                                  |                        |           | 60,182,193,037 |
---------------------------------------------------------------------------------------------------------

Notes: zpaq64 - v7.05 journaling archiver, compiled Apr 17 2015 bsc - Block Sorting Compressor. Version 3.1.0. 8 July 2012 (c) 2009-2012 Ilya Grebnov Ilya.Grebnov@gmail.com. xz - (XZ utils) 5.2.1 zstd - command line interface 64-bits v1.3.1, by Yann Collet lz5 - command line interface 64-bit 2.0.0 by Y.Collet & P.Skibinski (Feb 5 2017) rz - RAZOR Archiver 1.00 (2017-09-08) - DEMO/TEST version (c) Christian Martelock (christian.martelock@web.de)

C:\xx\Nigella\Nigella_Corpus>timer64.exe rz a -d 64m enwiki-20170101-pages-articles.xml.0064M.rz enwiki-20170101-pages-articles.xml

 *** RAZOR Archiver 1.00 (2017-09-08) - DEMO/TEST version ***
 *** (c) Christian Martelock (christian.martelock@web.de) ***

 Scanning c:\xx\nigella\nigella_corpus\enwiki-20170101-pages-articles.xml
 Found 0 dirs, 1 files, 60182193037 bytes.

 Creating archive enwiki-20170101-pages-articles.xml.0064M.rz
 Window : 65536K (512M..128G)
 Header : 64
 Size   : 9064770314

 Archive ok. Added 0 dirs, 1 files, 60182193037 bytes.
 CPU time = 179641.547s / wall time = 140882.532s

Kernel  Time =   334.250 =    0%
User    Time =179641.546 =  127%
Process Time =179975.796 =  127%    Virtual  Memory =    905 MB
Global  Time =140882.551 =  100%    Physical Memory =    889 MB

Feel free to ask to run other compressors/options on 'Compressionette' until I take on my latest corpus: 360,797,299,712 _GAMERA_r28_1316229_TXT_files_(English).tar

Unfortunately cannot bench the decompression rates, no SSD on my laptop, grmbl.

Edit: Filled stats for the finished Razor, plus some decompression stats:

SHA1: 7cfefa6f2c2ad080ab92d76c9bf9ea1b75aa7263 enwiki-20170101-pages-articles.xml

The time is not really representative since the 2TB 5400rpm HDD is badly fragmented - the I/O sucks.

------------------------------------------------------------------------------------------
| Decompressor | Options         | Total Decompression Time | Footprint |           Size |
------------------------------------------------------------------------------------------
| xz           | -d -f           |            2,327 seconds |     67 MB | 10,385,748,048 |
| zstd         | -d              |            3,124 seconds |    132 MB | 10,617,443,916 |
| lz5          | -d --no-sparse  |            2,444 seconds |    337 MB | 13,859,918,784 |
| rz           | e               |            1,756 seconds |    118 MB |  9,064,770,314 |
------------------------------------------------------------------------------------------

No idea what causes this:

C:\xx\Nigella\Nigella_Corpus>timer64.exe lz5 -d enwiki-20170101-pages-articles.xml.49.lz5
Decoding file enwiki-20170101-pages-articles.xml.49
Error 72 : Sparse skip error ; try --no-sparse

Exit code: 72

zstd, lz5, rz verified OK.

In my view the best performer is yet to come, Ilya Grebnov's bsc is excellent for huge textual data, however I fully expect Hamid Bouzidi's LzTurbo 59 to trump it. Razor is also impressive.

Very nice feature implemented Christian Martelock, namely the built-in deduplicator, very very handy.