search

inikep / lzbench

lzbench is an in-memory benchmark of open-source LZ77/LZSS/LZMA compressors
896 stars 182 forks source link

Feedback #59

Closed Sanmayce closed 5 years ago

Sanmayce commented 5 years ago

Hi inikep, many thanks for new releases, I appreciate your work, will play with v1.8 next week and will share results on i7-3630QM @3.4GHz turbo and i5-7200U @3.1GHz...

Sanmayce commented 5 years ago

Having tarred the sources of current version, on i7-3630QM @3.4GHz:

F:\TEXTORAMIC_benchmarking_2019-Nov-30>lzbench18 -c4 -i1,15 -o3 -elzham,4/slz_deflate,3/slz_gzip,3/slz_zlib,3/tornado,1,16/bzip2,1,9/zstd,1,22/zstdLDM,1,22/fastlzma2,1,10/brotli,0,1,11/blosclz,1,9/brieflz,1,5,9/crush,2/csc,5/density,1,3/fastlz,2/gipfeli/lzo1b,999/libdeflate,1,12/lz4,1,12/lizard,19,29,39,49/lzf,1/lzfse/lzg,9/lzjb/lzlib,9/lzma,0,1,9/lzrw,5/lzsse2,17/lzsse4,17/lzsse8,17/lzvn/pithy,9/quicklz,3/snappy/slz_zlib,3/ucl_nrv2b,9/ucl_nrv2d,9/ucl_nrv2e,9/xpack,1,9/xz,9/yalz77,12/yappy,99/zlib,1,5,9/zling,4/shrinker/wflz/lzmat/glza "lzbench-1.8.tar"
lzbench 1.8 (64-bit Windows)   Assembled by P.Skibinski
The results sorted by column number 4:
Compressor name         Compress. Decompress.  Orig. size  Compr. size  Ratio Filename
glza 0.8                 0.08 MB/s    58 MB/s    14167552      1865213  13.17 lzbench-1.8.tar
lzlib 1.11 -9            1.82 MB/s    88 MB/s    14167552      1919384  13.55 lzbench-1.8.tar
lzma 19.00 -9            2.89 MB/s   129 MB/s    14167552      1936414  13.67 lzbench-1.8.tar
xz 5.2.4 -9              2.89 MB/s   113 MB/s    14167552      1937713  13.68 lzbench-1.8.tar
fastlzma2 1.0.1 -10      3.81 MB/s   122 MB/s    14167552      1945652  13.73 lzbench-1.8.tar
brotli 2019-10-01 -11    0.49 MB/s   516 MB/s    14167552      1966508  13.88 lzbench-1.8.tar
zstd 1.4.3 -22           2.00 MB/s   848 MB/s    14167552      2012358  14.20 lzbench-1.8.tar
zstdLDM 1.4.3 -22        2.05 MB/s   853 MB/s    14167552      2023782  14.28 lzbench-1.8.tar
csc 2016-10-13 -5        3.60 MB/s    98 MB/s    14167552      2055423  14.51 lzbench-1.8.tar
tornado 0.6a -16         2.46 MB/s   281 MB/s    14167552      2068920  14.60 lzbench-1.8.tar
lzham 1.0 -d26 -4        1.65 MB/s   325 MB/s    14167552      2070818  14.62 lzbench-1.8.tar
bzip2 1.0.8 -9           9.71 MB/s    47 MB/s    14167552      2286942  16.14 lzbench-1.8.tar
zling 2018-10-12 -4        58 MB/s   252 MB/s    14167552      2304660  16.27 lzbench-1.8.tar
brieflz 1.2.0 -9         0.49 MB/s   443 MB/s    14167552      2372691  16.75 lzbench-1.8.tar
lizard 1.0 -49           1.31 MB/s  1622 MB/s    14167552      2446353  17.27 lzbench-1.8.tar
crush 1.0 -2             1.14 MB/s   439 MB/s    14167552      2474033  17.46 lzbench-1.8.tar
xpack 2016-06-02 -9        20 MB/s   846 MB/s    14167552      2541304  17.94 lzbench-1.8.tar
brieflz 1.2.0 -5           58 MB/s   410 MB/s    14167552      2574956  18.18 lzbench-1.8.tar
fastlzma2 1.0.1 -1         23 MB/s    90 MB/s    14167552      2592064  18.30 lzbench-1.8.tar
bzip2 1.0.8 -1             13 MB/s    49 MB/s    14167552      2662759  18.79 lzbench-1.8.tar
lzma 19.00 -1              29 MB/s    92 MB/s    14167552      2694749  19.02 lzbench-1.8.tar
libdeflate 1.3 -12       4.58 MB/s   893 MB/s    14167552      2721135  19.21 lzbench-1.8.tar
lzfse 2017-03-08           61 MB/s   788 MB/s    14167552      2758223  19.47 lzbench-1.8.tar
lizard 1.0 -29           1.37 MB/s  2328 MB/s    14167552      2773783  19.58 lzbench-1.8.tar
ucl_nrv2e 1.03 -9        2.34 MB/s   380 MB/s    14167552      2784747  19.66 lzbench-1.8.tar
ucl_nrv2d 1.03 -9        2.34 MB/s   403 MB/s    14167552      2808582  19.82 lzbench-1.8.tar
ucl_nrv2b 1.03 -9        2.32 MB/s   401 MB/s    14167552      2836567  20.02 lzbench-1.8.tar
zlib 1.2.11 -9           8.81 MB/s   381 MB/s    14167552      2861368  20.20 lzbench-1.8.tar

Nakamichi 'Dragoneye'    0.05 MB/s  1535 MB/s                  2869774  ! Outside lzbench, i7-3630QM @3.4GHz, Intel v15.0 !

zlib 1.2.11 -5             47 MB/s   365 MB/s    14167552      2968893  20.96 lzbench-1.8.tar
lzma 19.00 -0              30 MB/s    80 MB/s    14167552      2976267  21.01 lzbench-1.8.tar
zstdLDM 1.4.3 -1          161 MB/s   856 MB/s    14167552      2995046  21.14 lzbench-1.8.tar
lizard 1.0 -39           2.21 MB/s  1964 MB/s    14167552      3015548  21.28 lzbench-1.8.tar
zstd 1.4.3 -1             306 MB/s   839 MB/s    14167552      3042289  21.47 lzbench-1.8.tar
lzmat 1.01                 27 MB/s   378 MB/s    14167552      3048755  21.52 lzbench-1.8.tar
brotli 2019-10-01 -1      241 MB/s   412 MB/s    14167552      3129511  22.09 lzbench-1.8.tar

LZSSE2 -17                          3679 MB/s                  3134413  ! Outside lzbench, i7-3630QM @3.4GHz, Intel v15.0 !

lzsse2 2019-04-18 -17    2.41 MB/s  3670 MB/s    14167552      3134413  22.12 lzbench-1.8.tar
lzsse4 2019-04-18 -17    2.49 MB/s  3830 MB/s    14167552      3162370  22.32 lzbench-1.8.tar
lzsse8 2019-04-18 -17    2.42 MB/s  3654 MB/s    14167552      3210429  22.66 lzbench-1.8.tar
lzo1b 2.10 -999            13 MB/s   838 MB/s    14167552      3222136  22.74 lzbench-1.8.tar
xpack 2016-06-02 -1       150 MB/s   624 MB/s    14167552      3238107  22.86 lzbench-1.8.tar
lizard 1.0 -19           2.06 MB/s  2799 MB/s    14167552      3343296  23.60 lzbench-1.8.tar
libdeflate 1.3 -1         160 MB/s   804 MB/s    14167552      3358045  23.70 lzbench-1.8.tar
lzg 1.0.10 -9            1.23 MB/s   680 MB/s    14167552      3384133  23.89 lzbench-1.8.tar
brotli 2019-10-01 -0      294 MB/s   396 MB/s    14167552      3387227  23.91 lzbench-1.8.tar
brieflz 1.2.0 -1          189 MB/s   323 MB/s    14167552      3388863  23.92 lzbench-1.8.tar
quicklz 1.5.0 -3           59 MB/s   930 MB/s    14167552      3462591  24.44 lzbench-1.8.tar
yalz77 2015-09-19 -12      36 MB/s   447 MB/s    14167552      3621404  25.56 lzbench-1.8.tar
lzvn 2017-03-08            54 MB/s   925 MB/s    14167552      3632403  25.64 lzbench-1.8.tar
zlib 1.2.11 -1             98 MB/s   329 MB/s    14167552      3653876  25.79 lzbench-1.8.tar
pithy 2011-12-24 -9       392 MB/s  1508 MB/s    14167552      3925759  27.71 lzbench-1.8.tar
gipfeli 2016-07-13        322 MB/s   504 MB/s    14167552      4003047  28.26 lzbench-1.8.tar
lzrw 15-Jul-1991 -5       137 MB/s   535 MB/s    14167552      4277731  30.19 lzbench-1.8.tar
shrinker 0.1              355 MB/s   869 MB/s    14167552      4389303  30.98 lzbench-1.8.tar
slz_deflate 1.0.0 -3      284 MB/s   368 MB/s    14167552      4553143  32.14 lzbench-1.8.tar
slz_gzip 1.0.0 -3         243 MB/s   325 MB/s    14167552      4553149  32.14 lzbench-1.8.tar
slz_zlib 1.0.0 -3         230 MB/s   280 MB/s    14167552      4553161  32.14 lzbench-1.8.tar
slz_zlib 1.0.0 -3         230 MB/s   280 MB/s    14167552      4553161  32.14 lzbench-1.8.tar
tornado 0.6a -1           301 MB/s   446 MB/s    14167552      4600305  32.47 lzbench-1.8.tar
yappy 2014-03-22 -99       60 MB/s  2302 MB/s    14167552      4639392  32.75 lzbench-1.8.tar
fastlz 0.1 -2             311 MB/s   522 MB/s    14167552      4642302  32.77 lzbench-1.8.tar
lz4 1.9.2                 478 MB/s  2886 MB/s    14167552      4675144  33.00 lzbench-1.8.tar
lz4 1.9.2                 478 MB/s  2887 MB/s    14167552      4675144  33.00 lzbench-1.8.tar
snappy 2019-09-30         376 MB/s  1211 MB/s    14167552      4839349  34.16 lzbench-1.8.tar
density 0.14.2 -3         301 MB/s   297 MB/s    14167552      4952180  34.95 lzbench-1.8.tar
lzf 3.6 -1                318 MB/s   561 MB/s    14167552      5003796  35.32 lzbench-1.8.tar
wflz 2015-09-16           297 MB/s   783 MB/s    14167552      5709699  40.30 lzbench-1.8.tar
lzjb 2010                 250 MB/s   442 MB/s    14167552      6591545  46.53 lzbench-1.8.tar
blosclz 2.0.0 -9          288 MB/s   590 MB/s    14167552      7450688  52.59 lzbench-1.8.tar
density 0.14.2 -1        1021 MB/s  1619 MB/s    14167552      8355232  58.97 lzbench-1.8.tar
blosclz 2.0.0 -1         6000 MB/s  6465 MB/s    14167552     14167552 100.00 lzbench-1.8.tar
memcpy                   9772 MB/s  9701 MB/s    14167552     14167552 100.00 lzbench-1.8.tar

Speaking of my toy, not that bad, on par with 'zlib 1.2.11 -9' while 4x faster.

By the way, in next weeks I intend to finally write down the compression boost - only the NON-UNIQUE (a shocker :P) keys shall be inserted in B-trees - thus eliminating the insane memory footprint, and as a "side-effect" speeding up the compression rate. Don't know whether will find time to write the final tweak - to group all leaves of a given B-tree as one continuous block i.e. to enforce FULL LOCALITY - then all costly SSD random reads will be reduced to ... 1, eh.

To me, nicest of all add-ons is @jibsen Jørgen's Brieflz 1.2.0, you forgot to add it to the list:

Just ran on i5-7200u @3.1GHz 'Silesia', the binary is Intel v15.0 64bit:

Nakamichi 'Dragoneye'  1013 MB/s 72022153

I have inserted it in your roster (compared to your run with i7-8700K @4.7 GHz):

"The following results are obtained with lzbench 1.8 with the -t16,16 -eall options using 1 core of Intel Core i7-8700K, Ubuntu 18.04.3 64-bit, and clang 9.0.1 with "silesia.tar" which contains tarred files from Silesia compression corpus. The results are sorted by ratio."

Compressor name           Compress.   Decompress.           Compr. size        Ratio
lzlib 1.11 -9            1.82 MB/s        76 MB/s            48296889           22.79
fastlzma2 1.0.1 -10      3.99 MB/s       105 MB/s            48666065           22.96
lzma 19.00 -9            2.66 MB/s       107 MB/s            48707450           22.98
xz 5.2.4 -9              2.62 MB/s        88 MB/s            48745306           23.00
fastlzma2 1.0.1 -8       5.18 MB/s       103 MB/s            49126740           23.18
xz 5.2.4 -6              2.95 MB/s        89 MB/s            49195929           23.21
lzma 19.00 -5            3.28 MB/s       103 MB/s            49710307           23.45
lzlib 1.11 -6            2.82 MB/s        74 MB/s            49777495           23.49
csc 2016-10-13 -5        3.86 MB/s        77 MB/s            49801577           23.50
brotli 2019-10-01 -11    0.63 MB/s       451 MB/s            50412404           23.79
fastlzma2 1.0.1 -5       7.44 MB/s       103 MB/s            51209571           24.16
zstd 1.4.3 -22           2.28 MB/s       865 MB/s            52738312           24.88
tornado 0.6a -16         2.15 MB/s       207 MB/s            53257046           25.13
csc 2016-10-13 -3        9.38 MB/s        71 MB/s            53477914           25.23
zstd 1.4.3 -18           3.58 MB/s       912 MB/s            53690572           25.33
fastlzma2 1.0.1 -3         11 MB/s        94 MB/s            54023837           25.49
bzip2 1.0.8 -9             15 MB/s        41 MB/s            54572811           25.75
lzham 1.0 -d26 -1        2.98 MB/s       340 MB/s            54740589           25.83
tornado 0.6a -13         6.94 MB/s       202 MB/s            55614072           26.24
bzip2 1.0.8 -5             16 MB/s        44 MB/s            55724395           26.29
xz 5.2.4 -3              6.76 MB/s        84 MB/s            55745125           26.30
csc 2016-10-13 -1          21 MB/s        73 MB/s            56201092           26.52
lzlib 1.11 -3            6.81 MB/s        69 MB/s            56320674           26.57
brotli 2019-10-01 -8       10 MB/s       533 MB/s            57140168           26.96
zstd 1.4.3 -15           7.12 MB/s      1024 MB/s            57167422           26.97
lzma 19.00 -4              14 MB/s        95 MB/s            57201645           26.99
tornado 0.6a -10         5.73 MB/s       192 MB/s            57588241           27.17
lzma 19.00 -2              25 MB/s        91 MB/s            58867911           27.77
tornado 0.6a -7            16 MB/s       194 MB/s            59026325           27.85
fastlzma2 1.0.1 -1         23 MB/s        90 MB/s            59030954           27.85
zstd 1.4.3 -11             20 MB/s      1001 MB/s            59239357           27.95
brotli 2019-10-01 -5       35 MB/s       520 MB/s            59568603           28.11
bzip2 1.0.8 -1             18 MB/s        52 MB/s            60484813           28.54
zling 2018-10-12 -4        46 MB/s       226 MB/s            60626768           28.60
lizard 1.0 -49           1.95 MB/s      1729 MB/s            60679215           28.63
zstd 1.4.3 -8              46 MB/s      1055 MB/s            60757793           28.67
zling 2018-10-12 -3        53 MB/s       226 MB/s            60999828           28.78
xpack 2016-06-02 -9        17 MB/s      1116 MB/s            61240928           28.89
zling 2018-10-12 -2        60 MB/s       225 MB/s            61503093           29.02
zling 2018-10-12 -1        67 MB/s       221 MB/s            62022546           29.26
xpack 2016-06-02 -6        43 MB/s      1086 MB/s            62213845           29.35
tornado 0.6a -6            34 MB/s       195 MB/s            62364583           29.42
xz 5.2.4 -0                24 MB/s        70 MB/s            62579435           29.53
zling 2018-10-12 -0        75 MB/s       216 MB/s            62990590           29.72
crush 1.0 -2             0.82 MB/s       468 MB/s            63746223           30.08
lzlib 1.11 -0              36 MB/s        61 MB/s            63847386           30.12
zstd 1.4.3 -5             104 MB/s       932 MB/s            63993747           30.19
lzma 19.00 -0              34 MB/s        80 MB/s            64013917           30.20
lzham 1.0 -d26 -0          11 MB/s       271 MB/s            64089870           30.24
tornado 0.6a -5            51 MB/s       195 MB/s            64129604           30.26
libdeflate 1.3 -12       7.39 MB/s       900 MB/s            64801629           30.57
brieflz 1.2.0 -8         0.46 MB/s       473 MB/s            64912139           30.63
libdeflate 1.3 -9          16 MB/s       898 MB/s            65701539           31.00
crush 1.0 -1             6.11 MB/s       455 MB/s            66494412           31.37
brieflz 1.2.0 -6           19 MB/s       468 MB/s            67208420           31.71
lizard 1.0 -45             17 MB/s      1810 MB/s            67317588           31.76
lzfse 2017-03-08           90 MB/s       934 MB/s            67624281           31.91
zlib 1.2.11 -9             14 MB/s       404 MB/s            67644548           31.92
libdeflate 1.3 -6          99 MB/s       924 MB/s            67928189           32.05
brotli 2019-10-01 -2      154 MB/s       485 MB/s            68060686           32.11
zlib 1.2.11 -6             35 MB/s       407 MB/s            68228431           32.19
lizard 1.0 -29           2.07 MB/s      2697 MB/s            68694227           32.41
zstd 1.4.3 -2             356 MB/s      1067 MB/s            69594511           32.84
ucl_nrv2e 1.03 -9        2.13 MB/s       429 MB/s            69645134           32.86
lizard 1.0 -39           4.37 MB/s      2475 MB/s            69807522           32.94
ucl_nrv2d 1.03 -9        2.09 MB/s       422 MB/s            70053895           33.05
tornado 0.6a -4           133 MB/s       310 MB/s            70513617           33.27
libdeflate 1.3 -3         161 MB/s       912 MB/s            70668968           33.34
ucl_nrv2b 1.03 -9        2.09 MB/s       407 MB/s            71031195           33.51
xpack 2016-06-02 -1       171 MB/s       890 MB/s            71090065           33.54

Nakamichi 'Dragoneye'                   1013 MB/s            72022153           ! Outside lzbench, i5-7200U, Intel v15.0 !

tornado 0.6a -3           186 MB/s       301 MB/s            72662044           34.28
crush 1.0 -0               53 MB/s       413 MB/s            73064603           34.47
ucl_nrv2e 1.03 -6          21 MB/s       391 MB/s            73302012           34.58
libdeflate 1.3 -1         201 MB/s       865 MB/s            73318371           34.59
lizard 1.0 -42            131 MB/s      1621 MB/s            73350988           34.61
zstd 1.4.3 -1             480 MB/s      1203 MB/s            73508823           34.68
ucl_nrv2d 1.03 -6          21 MB/s       386 MB/s            73757673           34.80
ucl_nrv2b 1.03 -6          20 MB/s       375 MB/s            73902185           34.87
lizard 1.0 -35             95 MB/s      2279 MB/s            74563583           35.18
lzo1z 2.10 -999          8.67 MB/s       814 MB/s            75061331           35.42
lzo1x 2.10 -999          8.76 MB/s       827 MB/s            75301903           35.53
lzsse8 2019-04-18 -12    9.86 MB/s      4491 MB/s            75464339           35.61
lzsse8 2019-04-18 -16    9.90 MB/s      4461 MB/s            75464339           35.61
lzsse8 2019-04-18 -6       10 MB/s      4503 MB/s            75469717           35.61
lzo1y 2.10 -999          8.87 MB/s       822 MB/s            75503849           35.62
brieflz 1.2.0 -3          108 MB/s       436 MB/s            75550736           35.65
lzsse2 2019-04-18 -16    9.82 MB/s      3762 MB/s            75829973           35.78
lzsse2 2019-04-18 -12    9.74 MB/s      3754 MB/s            75829973           35.78
lzsse2 2019-04-18 -6       10 MB/s      3741 MB/s            75837101           35.78
lizard 1.0 -25             18 MB/s      2853 MB/s            75867915           35.80
lzsse4 2019-04-18 -16      10 MB/s      4291 MB/s            76113017           35.91
lzsse4 2019-04-18 -12      10 MB/s      4272 MB/s            76113017           35.91
lzsse4 2019-04-18 -6       10 MB/s      4272 MB/s            76118298           35.91
lzmat 1.01                 38 MB/s       479 MB/s            76485353           36.09
lzo1b 2.10 -999            12 MB/s       945 MB/s            76594292           36.14
zlib 1.2.11 -1            119 MB/s       383 MB/s            77259029           36.45
lz4hc 1.9.2 -12            11 MB/s      4427 MB/s            77262620           36.45
lizard 1.0 -19           4.60 MB/s      4043 MB/s            77416400           36.53
lz4hc 1.9.2 -9             33 MB/s      4378 MB/s            77884448           36.75
brotli 2019-10-01 -0      420 MB/s       419 MB/s            78433298           37.01
lizard 1.0 -32            193 MB/s      1641 MB/s            78652654           37.11
lz4hc 1.9.2 -4             81 MB/s      4210 MB/s            79807909           37.65
lzo1c 2.10 -999            24 MB/s       878 MB/s            80396741           37.93
lzvn 2017-03-08            73 MB/s      1223 MB/s            80814609           38.13
lizard 1.0 -40            354 MB/s      1497 MB/s            80843049           38.14
lzo1f 2.10 -999            21 MB/s       833 MB/s            80890206           38.17
brieflz 1.2.0 -1          197 MB/s       431 MB/s            81138803           38.28
lizard 1.0 -15             85 MB/s      4081 MB/s            81187330           38.31
ucl_nrv2e 1.03 -1          59 MB/s       330 MB/s            81195560           38.31
ucl_nrv2d 1.03 -1          59 MB/s       333 MB/s            81461976           38.43
ucl_nrv2b 1.03 -1          58 MB/s       322 MB/s            81703168           38.55
quicklz 1.5.0 -3           59 MB/s      1069 MB/s            81822241           38.60
lzsse8 2019-04-18 -1       19 MB/s      4166 MB/s            81866245           38.63
lzsse4 2019-04-18 -1       21 MB/s      3965 MB/s            82542106           38.94
lzo2a 2.10 -999            27 MB/s       667 MB/s            82809337           39.07
lzg 1.0.10 -8            9.30 MB/s       762 MB/s            83606901           39.45
lz4hc 1.9.2 -1            131 MB/s      4071 MB/s            83803769           39.54
yalz77 2015-09-19 -12      24 MB/s       518 MB/s            84050625           39.66
quicklz 1.5.0 -2          286 MB/s       708 MB/s            84555627           39.89
lizard 1.0 -22            149 MB/s      2904 MB/s            84866725           40.04
yalz77 2015-09-19 -8       35 MB/s       532 MB/s            85153287           40.18
lzo1b 2.10 -99            126 MB/s       839 MB/s            85653376           40.41
lizard 1.0 -30            453 MB/s      1414 MB/s            85727429           40.45
lizard 1.0 -12            179 MB/s      3955 MB/s            86232422           40.69
yalz77 2015-09-19 -4       56 MB/s       539 MB/s            87392632           41.23
density 0.14.2 -3         432 MB/s       529 MB/s            87649866           41.35
gipfeli 2016-07-13        403 MB/s       663 MB/s            87931759           41.49
lzsse2 2019-04-18 -1       24 MB/s      3276 MB/s            87976095           41.51
lzo1c 2.10 -99            110 MB/s       828 MB/s            88112288           41.57
lzo1b 2.10 -9             186 MB/s       816 MB/s            89261884           42.12
lzg 1.0.10 -6              29 MB/s       702 MB/s            89490220           42.22
tornado 0.6a -2           300 MB/s       488 MB/s            90076660           42.50
pithy 2011-12-24 -9       400 MB/s      2256 MB/s            90360813           42.63
lzrw 15-Jul-1991 -5       150 MB/s       677 MB/s            90818810           42.85
lzo1c 2.10 -9             169 MB/s       820 MB/s            91040386           42.95
lzo1b 2.10 -6             244 MB/s       823 MB/s            91382355           43.12
pithy 2011-12-24 -6       483 MB/s      2221 MB/s            92090898           43.45
lzo1a 2.10 -99            121 MB/s       869 MB/s            92666265           43.72
lzo1c 2.10 -6             211 MB/s       819 MB/s            93303623           44.02
yalz77 2015-09-19 -1      105 MB/s       578 MB/s            93952728           44.33
lzo1b 2.10 -3             255 MB/s       821 MB/s            94044578           44.37
quicklz 1.5.0 -1          550 MB/s       715 MB/s            94720562           44.69
lzo1 2.10 -99             123 MB/s       857 MB/s            94946129           44.80
lzg 1.0.10 -4              53 MB/s       655 MB/s            95930551           45.26
slz_zlib 1.0.0 -3         293 MB/s       379 MB/s            96187780           45.38
lzo1c 2.10 -3             262 MB/s       829 MB/s            96716153           45.63
slz_zlib 1.0.0 -2         297 MB/s       378 MB/s            96863094           45.70
lizard 1.0 -20            481 MB/s      2985 MB/s            96924204           45.73
lzo1b 2.10 -1             257 MB/s       805 MB/s            97036087           45.78
pithy 2011-12-24 -3       597 MB/s      2083 MB/s            97255186           45.89
yappy 2014-03-22 -100      96 MB/s      3001 MB/s            98672514           46.56
lzo1c 2.10 -1             269 MB/s       812 MB/s            99550904           46.97
slz_zlib 1.0.0 -1         301 MB/s       380 MB/s            99657958           47.02
lzo1f 2.10 -1             244 MB/s       793 MB/s            99743329           47.06
yappy 2014-03-22 -10      128 MB/s      2969 MB/s           100018673           47.19
lzrw 15-Jul-1991 -4       392 MB/s       630 MB/s           100131356           47.24
lzo1x 2.10 -1             680 MB/s       868 MB/s           100572537           47.45
lz4 1.9.2                 737 MB/s      4448 MB/s           100880800           47.60
fastlz 0.1 -2             368 MB/s       811 MB/s           100906072           47.61
lzo1y 2.10 -1             674 MB/s       863 MB/s           101258318           47.78
lzo1x 2.10 -15            699 MB/s       871 MB/s           101462094           47.87
density 0.14.2 -2         933 MB/s      1433 MB/s           101651444           47.96
lzf 3.6 -1                398 MB/s       914 MB/s           102041092           48.14
snappy 2019-09-30         591 MB/s      1868 MB/s           102146767           48.19
pithy 2011-12-24 -0       647 MB/s      2084 MB/s           103072463           48.63
lzo1x 2.10 -12            717 MB/s       875 MB/s           103238859           48.71
lizard 1.0 -10            635 MB/s      4173 MB/s           103402971           48.79
lzo1a 2.10 -1             309 MB/s       811 MB/s           104202251           49.16
fastlz 0.1 -1             341 MB/s       806 MB/s           104628084           49.37
lzrw 15-Jul-1991 -3       381 MB/s       726 MB/s           105424168           49.74
lzf 3.6 -0                400 MB/s       869 MB/s           105682088           49.86
yappy 2014-03-22 -1       165 MB/s      2809 MB/s           105750956           49.89
lzo1 2.10 -1              308 MB/s       799 MB/s           106474519           50.24
lzo1x 2.10 -11            735 MB/s       893 MB/s           106604629           50.30
lz4fast 1.9.2 -3          838 MB/s      4423 MB/s           107066190           50.52
tornado 0.6a -1           437 MB/s       520 MB/s           107381846           50.66
lzg 1.0.10 -1              91 MB/s       653 MB/s           108553667           51.22
wflz 2015-09-16           305 MB/s      1183 MB/s           109605264           51.71
lzrw 15-Jul-1991 -1       317 MB/s       646 MB/s           113761625           53.67
lzjb 2010                 394 MB/s       601 MB/s           122671613           57.88
lz4fast 1.9.2 -17        1201 MB/s      4632 MB/s           131732802           62.15
density 0.14.2 -1        2214 MB/s      2677 MB/s           133042166           62.77
blosclz 2.0.0 -9          403 MB/s      1037 MB/s           135557850           63.96
blosclz 2.0.0 -6          412 MB/s      1083 MB/s           137571765           64.91
shrinker 0.1              985 MB/s      3180 MB/s           172535778           81.40
blosclz 2.0.0 -3         1073 MB/s      5909 MB/s           199437330           94.10
blosclz 2.0.0 -1         6485 MB/s      7959 MB/s           211947520          100.00
Sanmayce commented 5 years ago

Oh, wanted to see how lzbench crunches a small 9MB DNA testfile from Kirill's Sequence Compression Benchmark:

http://kirr.dyndns.org/sequence-compression-benchmark/?d=Cryptosporidium+parvum+Iowa+II+GCA_000165345.1+%289.22+MB%29&doagg=1&agg=sum&cs=1&cg=1&cc=1&com=yes&src=all&nt=4&bn=1&bm=tdspeed&sm=same&tn=10&bs=100&rr=gzip-9&tm0=name&tm1=size&tm2=ratio&tm3=ctime&tm4=dtime&tm5=cdtime&tm6=tdtime&tm7=empty&gm=same&cyl=lin&ccw=1500&cch=500&sxm=ratio&sxmin=0&sxmax=0&sxl=lin&sym=dspeed&symin=0&symax=0&syl=log&button=Show+scatterplot

The run on i5-7200U:

E:\TEXTORAMIC_benchmarking_2019-Nov-30>lzbench18 -c4 -i1,15 -o3 -elzham,4/slz_deflate,3/slz_gzip,3/slz_zlib,3/tornado,1,16/bzip2,1,9/zstd,1,22/zstdLDM,1,22/fastlzma2,1,10/brotli,0,1,11/blosclz,1,9/brieflz,1,5,9/crush,2/csc,5/density,1,3/fastlz,2/gipfeli/lzo1b,999/libdeflate,1,12/lz4,1,12/lizard,19,29,39,49/lzf,1/lzfse/lzg,9/lzjb/lzlib,9/lzma,0,1,9/lzrw,5/lzsse2,17/lzsse4,17/lzsse8,17/lzvn/pithy,9/quicklz,3/snappy/slz_zlib,3/ucl_nrv2b,9/ucl_nrv2d,9/ucl_nrv2e,9/xpack,1,9/xz,9/yalz77,12/yappy,99/zlib,1,5,9/zling,4/shrinker/wflz/lzmat "GCA_000165345.1_ASM16534v1_genomic.fna"
lzbench 1.8 (64-bit Windows)   Assembled by P.Skibinski
Compressor name         Compress. Decompress.  Orig. size  Compr. size  Ratio Filename
csc 2016-10-13 -5          17 MB/s    32 MB/s     9216802      2414828  26.20 GCA_000165345.1_ASM16534v1_genomic.fna
fastlzma2 1.0.1 -10      2.83 MB/s    38 MB/s     9216802      2500310  27.13 GCA_000165345.1_ASM16534v1_genomic.fna
lzlib 1.11 -9            0.91 MB/s    50 MB/s     9216802      2537517  27.53 GCA_000165345.1_ASM16534v1_genomic.fna
tornado 0.6a -16         1.13 MB/s   143 MB/s     9216802      2544932  27.61 GCA_000165345.1_ASM16534v1_genomic.fna
xz 5.2.4 -9              0.93 MB/s    71 MB/s     9216802      2549450  27.66 GCA_000165345.1_ASM16534v1_genomic.fna
lzma 19.00 -9            0.98 MB/s    76 MB/s     9216802      2549520  27.66 GCA_000165345.1_ASM16534v1_genomic.fna
lzham 1.0 -d26 -4        0.81 MB/s   193 MB/s     9216802      2566994  27.85 GCA_000165345.1_ASM16534v1_genomic.fna
zstd 1.4.3 -22           1.16 MB/s   288 MB/s     9216802      2568193  27.86 GCA_000165345.1_ASM16534v1_genomic.fna
zstdLDM 1.4.3 -22        1.15 MB/s   288 MB/s     9216802      2568337  27.87 GCA_000165345.1_ASM16534v1_genomic.fna
brotli 2019-10-01 -11    0.29 MB/s   138 MB/s     9216802      2589623  28.10 GCA_000165345.1_ASM16534v1_genomic.fna
bzip2 1.0.8 -9             11 MB/s    19 MB/s     9216802      2699355  29.29 GCA_000165345.1_ASM16534v1_genomic.fna
libdeflate 1.3 -12       3.49 MB/s   812 MB/s     9216802      2700779  29.30 GCA_000165345.1_ASM16534v1_genomic.fna
lizard 1.0 -49           1.19 MB/s  1193 MB/s     9216802      2728503  29.60 GCA_000165345.1_ASM16534v1_genomic.fna
bzip2 1.0.8 -1             11 MB/s    29 MB/s     9216802      2730660  29.63 GCA_000165345.1_ASM16534v1_genomic.fna
lizard 1.0 -39           4.02 MB/s  1715 MB/s     9216802      2762531  29.97 GCA_000165345.1_ASM16534v1_genomic.fna
zlib 1.2.11 -9           1.16 MB/s   294 MB/s     9216802      2848699  30.91 GCA_000165345.1_ASM16534v1_genomic.fna
zling 2018-10-12 -4        24 MB/s   152 MB/s     9216802      2872321  31.16 GCA_000165345.1_ASM16534v1_genomic.fna
xpack 2016-06-02 -9      3.31 MB/s   817 MB/s     9216802      2898404  31.45 GCA_000165345.1_ASM16534v1_genomic.fna
lzma 19.00 -0              12 MB/s    51 MB/s     9216802      2944692  31.95 GCA_000165345.1_ASM16534v1_genomic.fna
lzma 19.00 -1              11 MB/s    51 MB/s     9216802      2954143  32.05 GCA_000165345.1_ASM16534v1_genomic.fna
fastlzma2 1.0.1 -1         16 MB/s    53 MB/s     9216802      2959540  32.11 GCA_000165345.1_ASM16534v1_genomic.fna
lzsse2 2019-04-18 -17    3.70 MB/s  3560 MB/s     9216802      2986618  32.40 GCA_000165345.1_ASM16534v1_genomic.fna
lzsse4 2019-04-18 -17    4.33 MB/s  3625 MB/s     9216802      2987700  32.42 GCA_000165345.1_ASM16534v1_genomic.fna
lzsse8 2019-04-18 -17    4.16 MB/s  3542 MB/s     9216802      3001551  32.57 GCA_000165345.1_ASM16534v1_genomic.fna
zlib 1.2.11 -5             12 MB/s   254 MB/s     9216802      3062704  33.23 GCA_000165345.1_ASM16534v1_genomic.fna
zstd 1.4.3 -1             191 MB/s   354 MB/s     9216802      3065124  33.26 GCA_000165345.1_ASM16534v1_genomic.fna
zstdLDM 1.4.3 -1          126 MB/s   354 MB/s     9216802      3065684  33.26 GCA_000165345.1_ASM16534v1_genomic.fna
brotli 2019-10-01 -1      198 MB/s   294 MB/s     9216802      3092927  33.56 GCA_000165345.1_ASM16534v1_genomic.fna
brieflz 1.2.0 -9         0.02 MB/s   216 MB/s     9216802      3095187  33.58 GCA_000165345.1_ASM16534v1_genomic.fna
crush 1.0 -2             0.04 MB/s   275 MB/s     9216802      3132711  33.99 GCA_000165345.1_ASM16534v1_genomic.fna
libdeflate 1.3 -1         138 MB/s   594 MB/s     9216802      3141223  34.08 GCA_000165345.1_ASM16534v1_genomic.fna
lzfse 2017-03-08           46 MB/s   470 MB/s     9216802      3156725  34.25 GCA_000165345.1_ASM16534v1_genomic.fna
xpack 2016-06-02 -1       123 MB/s   430 MB/s     9216802      3166798  34.36 GCA_000165345.1_ASM16534v1_genomic.fna

Nakamichi 'Dragoneye'                902 MB/s                  3189749        ! Outside lzbench, i5-7200U, Intel v15.0 !

brotli 2019-10-01 -0      281 MB/s   284 MB/s     9216802      3263779  35.41 GCA_000165345.1_ASM16534v1_genomic.fna
lzo1b 2.10 -999          1.19 MB/s   577 MB/s     9216802      3358483  36.44 GCA_000165345.1_ASM16534v1_genomic.fna
zlib 1.2.11 -1             60 MB/s   232 MB/s     9216802      3398830  36.88 GCA_000165345.1_ASM16534v1_genomic.fna
lizard 1.0 -29           1.20 MB/s  1808 MB/s     9216802      3472659  37.68 GCA_000165345.1_ASM16534v1_genomic.fna
brieflz 1.2.0 -5           52 MB/s   225 MB/s     9216802      3474716  37.70 GCA_000165345.1_ASM16534v1_genomic.fna
yappy 2014-03-22 -99       36 MB/s  2634 MB/s     9216802      3483065  37.79 GCA_000165345.1_ASM16534v1_genomic.fna
lizard 1.0 -19           4.05 MB/s  2622 MB/s     9216802      3486132  37.82 GCA_000165345.1_ASM16534v1_genomic.fna
lzmat 1.01               3.26 MB/s   280 MB/s     9216802      3500280  37.98 GCA_000165345.1_ASM16534v1_genomic.fna
ucl_nrv2e 1.03 -9        0.16 MB/s   249 MB/s     9216802      3690049  40.04 GCA_000165345.1_ASM16534v1_genomic.fna
brieflz 1.2.0 -1          188 MB/s   234 MB/s     9216802      3737088  40.55 GCA_000165345.1_ASM16534v1_genomic.fna
ucl_nrv2d 1.03 -9        0.16 MB/s   257 MB/s     9216802      3760675  40.80 GCA_000165345.1_ASM16534v1_genomic.fna
ucl_nrv2b 1.03 -9        0.15 MB/s   243 MB/s     9216802      3834797  41.61 GCA_000165345.1_ASM16534v1_genomic.fna
lzg 1.0.10 -9            0.05 MB/s   536 MB/s     9216802      4004074  43.44 GCA_000165345.1_ASM16534v1_genomic.fna
gipfeli 2016-07-13        223 MB/s   402 MB/s     9216802      4101181  44.50 GCA_000165345.1_ASM16534v1_genomic.fna
yalz77 2015-09-19 -12      26 MB/s   307 MB/s     9216802      4190878  45.47 GCA_000165345.1_ASM16534v1_genomic.fna
quicklz 1.5.0 -3           38 MB/s   718 MB/s     9216802      4278962  46.43 GCA_000165345.1_ASM16534v1_genomic.fna
lzvn 2017-03-08            44 MB/s   865 MB/s     9216802      4292057  46.57 GCA_000165345.1_ASM16534v1_genomic.fna
lzrw 15-Jul-1991 -5        80 MB/s   428 MB/s     9216802      4376803  47.49 GCA_000165345.1_ASM16534v1_genomic.fna
pithy 2011-12-24 -9       300 MB/s   882 MB/s     9216802      4400945  47.75 GCA_000165345.1_ASM16534v1_genomic.fna
snappy 2019-09-30         263 MB/s   757 MB/s     9216802      4672078  50.69 GCA_000165345.1_ASM16534v1_genomic.fna
density 0.14.2 -3         550 MB/s   468 MB/s     9216802      4780998  51.87 GCA_000165345.1_ASM16534v1_genomic.fna
density 0.14.2 -1        1794 MB/s  1818 MB/s     9216802      4922254  53.41 GCA_000165345.1_ASM16534v1_genomic.fna
blosclz 2.0.0 -9          239 MB/s   410 MB/s     9216802      4930865  53.50 GCA_000165345.1_ASM16534v1_genomic.fna
tornado 0.6a -1           289 MB/s   319 MB/s     9216802      5110101  55.44 GCA_000165345.1_ASM16534v1_genomic.fna
shrinker 0.1              298 MB/s   437 MB/s     9216802      5263476  57.11 GCA_000165345.1_ASM16534v1_genomic.fna
lz4 1.9.2                 341 MB/s  1837 MB/s     9216802      5269661  57.17 GCA_000165345.1_ASM16534v1_genomic.fna
lz4 1.9.2                 341 MB/s  1837 MB/s     9216802      5269661  57.17 GCA_000165345.1_ASM16534v1_genomic.fna
slz_deflate 1.0.0 -3      222 MB/s   275 MB/s     9216802      5284119  57.33 GCA_000165345.1_ASM16534v1_genomic.fna
slz_gzip 1.0.0 -3         198 MB/s   248 MB/s     9216802      5284125  57.33 GCA_000165345.1_ASM16534v1_genomic.fna
slz_zlib 1.0.0 -3         187 MB/s   224 MB/s     9216802      5284137  57.33 GCA_000165345.1_ASM16534v1_genomic.fna
slz_zlib 1.0.0 -3         187 MB/s   223 MB/s     9216802      5284137  57.33 GCA_000165345.1_ASM16534v1_genomic.fna
lzf 3.6 -1                268 MB/s   538 MB/s     9216802      5391744  58.50 GCA_000165345.1_ASM16534v1_genomic.fna
fastlz 0.1 -2             230 MB/s   402 MB/s     9216802      5536433  60.07 GCA_000165345.1_ASM16534v1_genomic.fna
lzjb 2010                 238 MB/s   363 MB/s     9216802      5941195  64.46 GCA_000165345.1_ASM16534v1_genomic.fna
wflz 2015-09-16           171 MB/s   534 MB/s     9216802      7492383  81.29 GCA_000165345.1_ASM16534v1_genomic.fna
blosclz 2.0.0 -1         4410 MB/s  8472 MB/s     9216802      9216802 100.00 GCA_000165345.1_ASM16534v1_genomic.fna
memcpy                  10725 MB/s 11146 MB/s     9216802      9216802 100.00 GCA_000165345.1_ASM16534v1_genomic.fna

Of course, bigger DNA sets are far more interesting, the Human Genome ~3.3GB long is yet to be seen.

inikep commented 5 years ago

Thanks, these are interesting results.

Sanmayce commented 4 years ago

Don't know whether will find time to write the final tweak - to group all leaves of a given B-tree as one continuous block i.e. to enforce FULL LOCALITY - then all costly SSD random reads will be reduced to ... 1, eh.

Found time and wrote it down: https://community.centminmod.com/threads/a-lzss-microdeduplicator-tagetting-huge-texts-with-c-source.16427/#post-80533

Thanks again inikep, lzbench and Lizard are so cool, glad now even more having Jørgen Ibsen's latest BriefLZ - it beats Nakamichi almost everytime ratiowise, wish I had one more laptop dedicated for his cruncher.

Sanmayce commented 4 years ago

@jibsen Jørgen, the BriefLZ 1.2.0 mode 5 crashed (within lzbench 1.8) with 'SPETSNAZ', maybe the reason is some memory insufficiency!? It is worth testing yourself, I reckon, maybe some little tweak is needed. However, your new 1.3.0 beats my Satanichi, very good indeed.

On i7-3630QM 16GB DDR3, Windows 10:

F:\S>lzbench18 -c4 -i1,15 -o3 -elzham,4/slz_deflate,3/slz_gzip,3/slz_zlib,3/tornado,1,16/bzip2,1,9/zstd,1,22/zstdLDM,1,22/fastlzma2,1,10/brotli,0,1,3,5,11/blosclz,1,9/brieflz,1,5,9/crush,2/csc,5/density,1,3/fastlz,2/gipfeli/lzo1b,999/libdeflate,1,12/lz4,1,12/lizard,11,19,21,29,31,39,41,49/lzf,1/lzfse/lzg,9/lzjb/lzlib,9/lzma,0,1,9/lzrw,5/lzsse2,17/lzsse4,17/lzsse8,17/lzvn/pithy,9/quicklz,3/snappy/slz_zlib,3/ucl_nrv2b,9/ucl_nrv2d,9/ucl_nrv2e,9/xpack,1,9/xz,9/yalz77,12/yappy,99/zlib,1,5,9/zling,4/shrinker/wflz/lzmat ""book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar""
lzbench 1.8 (64-bit Windows)   Assembled by P.Skibinski
Compressor name         Compress. Decompress.  Orig. size  Compr. size  Ratio Filename
memcpy                   9191 MB/s  9125 MB/s   832290816    832290816 100.00 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lzham 1.0 -d26 -4        0.83 MB/s   317 MB/s   832290816    108470968  13.03 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
slz_deflate 1.0.0 -3      205 MB/s   256 MB/s   832290816    453783256  54.52 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
slz_gzip 1.0.0 -3         190 MB/s   234 MB/s   832290816    453783262  54.52 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
slz_zlib 1.0.0 -3         182 MB/s   211 MB/s   832290816    453783274  54.52 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
tornado 0.6a -1           231 MB/s   293 MB/s   832290816    435175239  52.29 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
tornado 0.6a -16         1.28 MB/s   264 MB/s   832290816    108105061  12.99 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
bzip2 1.0.8 -1             12 MB/s    41 MB/s   832290816    177659512  21.35 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
bzip2 1.0.8 -9             11 MB/s    37 MB/s   832290816    145658823  17.50 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
zstd 1.4.3 -1             191 MB/s   589 MB/s   832290816    257209725  30.90 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
zstd 1.4.3 -22           1.43 MB/s   840 MB/s   832290816    106285862  12.77 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
zstdLDM 1.4.3 -1          124 MB/s   589 MB/s   832290816    257066314  30.89 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
zstdLDM 1.4.3 -22        1.44 MB/s   842 MB/s   832290816    106556756  12.80 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
fastlzma2 1.0.1 -1         19 MB/s    66 MB/s   832290816    232444725  27.93 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
fastlzma2 1.0.1 -10      2.25 MB/s   125 MB/s   832290816    112251678  13.49 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
brotli 2019-10-01 -0      215 MB/s   319 MB/s   832290816    287973706  34.60 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
brotli 2019-10-01 -1      182 MB/s   334 MB/s   832290816    266460227  32.02 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
brotli 2019-10-01 -3       79 MB/s   414 MB/s   832290816    219485981  26.37 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
brotli 2019-10-01 -5       28 MB/s   512 MB/s   832290816    179403587  21.56 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
brotli 2019-10-01 -11    0.45 MB/s   636 MB/s   832290816    132070667  15.87 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
blosclz 2.0.0 -1         6030 MB/s  6167 MB/s   832290816    832290816 100.00 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
blosclz 2.0.0 -9          223 MB/s   358 MB/s   832290816    497813982  59.81 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
brieflz 1.2.0 -1          174 MB/s   269 MB/s   832290816    253592756  30.47 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
CRASH
  C.Time |        Size | Filename                                                           | Command Line
  -------|-------------|--------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------
         | 832,290,816 | book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar                 | sha1sum: b5a38bcc70ccc77daa2b9c459a9535b2e7660e5e  
    396s | 166,179,937 | book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar.9.crush_xezz    | crush_xezz.exe -9 "%1" "%1.9.crush_xezz"
 221703s | 131,370,690 | book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar.blz             | BriefLZ_130_Intel_v19_64bit.exe --optimal -b2g "%1" "%1.blz"
   2105s |  99,152,526 | book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar.bro             | brotli_ic19.exe -q 11 --large_window=30 "%1" -o "%1.bro"
    650s | 101,121,392 | book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar.L22W2GB.zst     | zstd-v1.4.4-win64.exe --single-thread --ultra -22 --zstd=wlog=31,clog=30,hlog=30,slog=26 "%1" -o "%1.L22W2GB.zst"
    260s | 121,901,223 | book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar.method211.zpaq  | "zpaq_v7.05_x64.exe" add "%1.method211.zpaq" "%1" -method 211 -threads 1
   1845s |  87,527,794 | book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar.method511.zpaq  | "zpaq_v7.05_x64.exe" add "%1.method511.zpaq" "%1" -method 511 -threads 1
    672s |  97,805,186 | book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar.MX9Dict1024.7z  | "7za_x64_v1900.exe" a -t7z -mx9 -md=30 "%1.MX9Dict1024.7z" "%1"
1282007s | 149,041,296 | book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar.Nakamichi       | Satanichi_GCC730_64bit.exe "%1" "%1.Nakamichi" 26 80000 e
    775s | 154,632,025 | book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar.optimal.bcrush  | bcrush_ICL150_64bit.exe --optimal "%1" "%1.optimal.bcrush"
   2764s |  82,503,337 | book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar.rz              | rz.exe a -d 1023M "%1.rz" "%1"
     92s |  76,516,343 | book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar.zv              | bwtturbo -59 -t0 -b1024 "%1" "%1.zv"

Note1: I find amazing ZPAQ 511 being outperformed sizewise by bwtturbo, by a great margin! Note2: My Satanichi falls behind, if 96N = 76,581MB physical RAM are available then the compression rate is ~50KB/s.

Also, glad to enrich the TEXTUAL SHOWDOWN with a precious corpus (247 MB) - all the games by 242 chessmasters in .PGN format - tarred:

www.pgnmentor.com_242_players.txt: https://drive.google.com/file/d/1gtFnps1EU5-yBxx45GfqVljtZWgpU_wT/view?usp=sharing

www.pgnmentor.com_242_players.zip: https://drive.google.com/file/d/1dMxRd9gV_avq0_1mF_SYXGM-zxaYPFdI/view?usp=sharing

fv_www.pgnmentor.com_242_players.tar.bmp: https://drive.google.com/file/d/1qfkkPzydy9C-_SjpXWosygPiw1o-_rvl/view?usp=sharing

Dr. Mahoneys's fv generated an unseen histogram so far, thus it will tell us more about parsers and schemes.

Will share lzbench and turbobench results, in days...

jibsen commented 4 years ago

Thanks for the notice, I believe the issue is the same as you reported in https://github.com/jibsen/brieflz/issues/9 -- level 5 uses 12*N bytes of memory, which overflows the unsigned long value on Windows when the block size is larger than ~341MiB.

If you build the current master branch of lzbench it should include BriefLZ 1.3.0 which fixes this.

Sanmayce commented 4 years ago

Thanks. In the past, I encountered building problems with MinGW, should try once again. I started compressing the most definitive English language corpus - Delphi_Classics_Complete_Works_of_128_authors.tar ~ 1.5GB. Don't want to "enforce" handicapness on compressors not having 2GB windows, simply this series of complete writings happened to be this long, even the 1GB performers are not up to the task, not my fault, yet, it looks unfair. After 640h will run BriefLZ 1.3.0 either within lzbench or 

BriefLZ_130_Intel_v19_64bit.exe --optimal -b2g "%1" "%1.blz"

@inikep Could you please upload a new release/binary, if possible with adding large window options for Zstd and Brotli, it would be nice to have these within lzbench, the delta is ~32MB for Brotli.

zstd 1.4.3 -22           1.43 MB/s   841 MB/s   832290816    106285862  12.77 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
zstd-v1.4.4-win64.exe --ultra -22 --zstd=wlog=31             101121392 
brotli 2019-10-01 -11    0.45 MB/s   637 MB/s   832290816    132070667  15.87 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
brotli_ic19.exe -q 11 --large_window=30                       99152526

The DDR3 laptop gives:

F:\S>lzbench18 -c4 -i1,15 -o3 -elzham,4/slz_deflate,3/slz_gzip,3/slz_zlib,3/tornado,1,16/bzip2,1,9/zstd,1,22/zstdLDM,1,22/fastlzma2,1,10/brotli,0,1,3,5,11/blosclz,1,9/crush,2/csc,5/density,1,3/fastlz,2/gipfeli/lzo1b,999/libdeflate,1,12/lz4,1,12/lizard,11,19,21,29,31,39,41,49/lzf,1/lzfse/lzg,9/lzjb/lzlib,9/lzma,0,1,9/lzrw,5/lzsse2,17/lzsse4,17/lzsse8,17/lzvn/pithy,9/quicklz,3/snappy/slz_zlib,3/ucl_nrv2b,9/ucl_nrv2d,9/ucl_nrv2e,9/xpack,1,9/xz,9/yalz77,12/yappy,99/zlib,1,5,9/zling,4/shrinker/wflz/lzmat ""book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar""
lzbench 1.8 (64-bit Windows)   Assembled by P.Skibinski
The results sorted by column number 4:
Compressor name         Compress. Decompress.  Orig. size  Compr. size  Ratio Filename
zstd 1.4.3 -22           1.43 MB/s   841 MB/s   832290816    106285862  12.77 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
zstdLDM 1.4.3 -22        1.44 MB/s   842 MB/s   832290816    106556756  12.80 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
xz 5.2.4 -9              1.17 MB/s   139 MB/s   832290816    107796983  12.95 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lzma 19.00 -9            1.15 MB/s   143 MB/s   832290816    107849433  12.96 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
tornado 0.6a -16         1.28 MB/s   264 MB/s   832290816    108105061  12.99 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lzham 1.0 -d26 -4        0.83 MB/s   317 MB/s   832290816    108470968  13.03 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
fastlzma2 1.0.1 -10      2.27 MB/s   125 MB/s   832290816    112251678  13.49 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lzlib 1.11 -9            1.10 MB/s    96 MB/s   832290816    113146693  13.59 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
csc 2016-10-13 -5        1.49 MB/s   130 MB/s   832290816    116667138  14.02 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
brotli 2019-10-01 -11    0.45 MB/s   637 MB/s   832290816    132070667  15.87 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lizard 1.0 -49           1.55 MB/s  1110 MB/s   832290816    134883363  16.21 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
bzip2 1.0.8 -9             11 MB/s    37 MB/s   832290816    145658823  17.50 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lizard 1.0 -29           1.57 MB/s  1417 MB/s   832290816    158642651  19.06 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
crush 1.0 -2             2.52 MB/s   437 MB/s   832290816    167891024  20.17 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
zling 2018-10-12 -4        36 MB/s   216 MB/s   832290816    172424277  20.72 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
bzip2 1.0.8 -1             12 MB/s    41 MB/s   832290816    177659512  21.35 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
brotli 2019-10-01 -5       28 MB/s   512 MB/s   832290816    179403587  21.56 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
xpack 2016-06-02 -9      5.63 MB/s   788 MB/s   832290816    207604165  24.94 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lizard 1.0 -39           4.21 MB/s  1787 MB/s   832290816    209834821  25.21 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
libdeflate 1.3 -12       4.06 MB/s   938 MB/s   832290816    213213594  25.62 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lzsse2 2019-04-18 -17    2.49 MB/s  3470 MB/s   832290816    216869808  26.06 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lzsse4 2019-04-18 -17    2.58 MB/s  3467 MB/s   832290816    217770793  26.17 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
brotli 2019-10-01 -3       79 MB/s   414 MB/s   832290816    219485981  26.37 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lzsse8 2019-04-18 -17    2.47 MB/s  3109 MB/s   832290816    224389638  26.96 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
zlib 1.2.11 -9           4.68 MB/s   322 MB/s   832290816    227825566  27.37 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
fastlzma2 1.0.1 -1         19 MB/s    66 MB/s   832290816    232444725  27.93 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lzma 19.00 -1              16 MB/s    67 MB/s   832290816    233717316  28.08 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
yalz77 2015-09-19 -12      34 MB/s   401 MB/s   832290816    237645327  28.55 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
ucl_nrv2e 1.03 -9        0.44 MB/s   306 MB/s   832290816    243105097  29.21 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lzma 19.00 -0              17 MB/s    62 MB/s   832290816    244108780  29.33 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
ucl_nrv2d 1.03 -9        0.43 MB/s   326 MB/s   832290816    245933936  29.55 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lzo1b 2.10 -999          4.90 MB/s   679 MB/s   832290816    246693493  29.64 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lizard 1.0 -19           4.16 MB/s  2501 MB/s   832290816    248001307  29.80 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
zlib 1.2.11 -5             22 MB/s   296 MB/s   832290816    251891158  30.26 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
ucl_nrv2b 1.03 -9        0.43 MB/s   322 MB/s   832290816    252275771  30.31 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lzg 1.0.10 -9            0.46 MB/s   733 MB/s   832290816    254874747  30.62 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lizard 1.0 -41            151 MB/s  1159 MB/s   832290816    255405016  30.69 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
zstdLDM 1.4.3 -1          124 MB/s   589 MB/s   832290816    257066314  30.89 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
zstd 1.4.3 -1             191 MB/s   589 MB/s   832290816    257209725  30.90 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lzfse 2017-03-08           48 MB/s   518 MB/s   832290816    259743898  31.21 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
brotli 2019-10-01 -1      182 MB/s   334 MB/s   832290816    266460227  32.02 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lzmat 1.01                 12 MB/s   308 MB/s   832290816    266952375  32.07 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
libdeflate 1.3 -1         134 MB/s   665 MB/s   832290816    285713451  34.33 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
brotli 2019-10-01 -0      215 MB/s   320 MB/s   832290816    287973706  34.60 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
xpack 2016-06-02 -1       122 MB/s   410 MB/s   832290816    298137386  35.82 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lizard 1.0 -21            173 MB/s  1735 MB/s   832290816    310381370  37.29 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lizard 1.0 -31            203 MB/s  1423 MB/s   832290816    315336786  37.89 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
zlib 1.2.11 -1             67 MB/s   251 MB/s   832290816    321781027  38.66 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
yappy 2014-03-22 -99       57 MB/s  1988 MB/s   832290816    322920818  38.80 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lzvn 2017-03-08            38 MB/s   716 MB/s   832290816    331212729  39.80 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
pithy 2011-12-24 -9       321 MB/s  1396 MB/s   832290816    338709477  40.70 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
gipfeli 2016-07-13        238 MB/s   455 MB/s   832290816    344610201  41.41 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lzrw 15-Jul-1991 -5        95 MB/s   451 MB/s   832290816    346324028  41.61 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
quicklz 1.5.0 -3           39 MB/s   635 MB/s   832290816    348569403  41.88 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
density 0.14.2 -3         433 MB/s   379 MB/s   832290816    370942860  44.57 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lizard 1.0 -11            265 MB/s  1968 MB/s   832290816    387848544  46.60 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lz4 1.9.2                 305 MB/s  2322 MB/s   832290816    408975647  49.14 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lz4 1.9.2                 300 MB/s  2322 MB/s   832290816    408975647  49.14 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
snappy 2019-09-30         255 MB/s   861 MB/s   832290816    423828918  50.92 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
tornado 0.6a -1           229 MB/s   293 MB/s   832290816    435175239  52.29 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
fastlz 0.1 -2             246 MB/s   381 MB/s   832290816    436415456  52.44 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lzf 3.6 -1                240 MB/s   439 MB/s   832290816    443098437  53.24 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
density 0.14.2 -1        1641 MB/s  1922 MB/s   832290816    445806192  53.56 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
slz_deflate 1.0.0 -3      204 MB/s   256 MB/s   832290816    453783256  54.52 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
slz_gzip 1.0.0 -3         191 MB/s   234 MB/s   832290816    453783262  54.52 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
slz_zlib 1.0.0 -3         180 MB/s   210 MB/s   832290816    453783274  54.52 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
slz_zlib 1.0.0 -3         182 MB/s   211 MB/s   832290816    453783274  54.52 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
blosclz 2.0.0 -9          222 MB/s   358 MB/s   832290816    497813982  59.81 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lzjb 2010                 182 MB/s   321 MB/s   832290816    539617346  64.84 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
wflz 2015-09-16           201 MB/s   522 MB/s   832290816    557630036  67.00 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
shrinker 0.1             4767 MB/s  6254 MB/s   832290816    820229674  98.55 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
blosclz 2.0.0 -1         6036 MB/s  6180 MB/s   832290816    832290816 100.00 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
memcpy                   6986 MB/s  9184 MB/s   832290816    832290816 100.00 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
jibsen commented 4 years ago

In the past, I encountered building problems with MinGW, should try once again.

I use MSYS2 to compile it. If you do not have MSYS2, then after installing it, run the update procedure they recommend, and then

pacman -S base-devel mingw-w64-x86_64-toolchain git

to install GCC and git.