Binary needed - Githubissues

Hi, if there is not going to be binary in 'release' section, could you share inhere how to compile using ICL? https://twitter.com/Sanmayce/status/965935926735196160

For various reasons I avoid using 'make' and such, a command line compilist (to differentiate from 'compiler', heh-heh) here.

The idea is we to have a command line tool like legendary PKZIP/PKUNZIP, I would use it on a daily basis, for example, currently I am running several big textual benchmarks:

https://github.com/powturbo/TurboBench/issues/10#issuecomment-367445792 To test the parsing prowess of 1GB window, this 900MB DNA set is quite good.
https://www.reddit.com/r/datasets/comments/7cise3/reddit_october_comments_are_now_available_with/dqaq9pv/ It is an understatement to call reddit test - big, it is quite literally tera - a teracorpus indeed!

My wish is to set/present a roofline :P (as opposed to baseline) by using the full power of Brotli with 1GB, currently my testmachine 'Compressionette' (i5-7200u, 8GB DDR4) halved the teratask: zstd-v1.3.3-win64.exe -T2 -12 ... Wanna use bsc with 1024MB block, 7zip with 30bit window, and Zstd with 30bit window as well.

@Sanmayce using the auto* stack in brotli 1.0.2 should make it easier for you to switch compiler (CC=icl ./configure).

@jbergstroem Thanks, but I ask for static build compile line, not for continuation of those 'make' mumbojumboisms. I remember made myself a static executable of some of first versions (IIRC they even were .CC), I am dismayed why (now in .C) it is so difficult we to have the compile command line?! My wish is to compile and share two static executables - SSE2 and AVX2. Johan, don't you agree that not having binary (and not using it) ruins the purpose? To me, such a powerful compressor with such speedy decompression deserves stable static binary, and naturally users to start amassing experience ... and just then be opinionated.

Just found in my archives an old brotli that I was able to compile with Intel v15:

D:\brotli-master>makeEXE.bat

D:\brotli-master>cd dec

D:\brotli-master\dec>icl /O3 /c bit_reader.c decode.c huffman.c state.c streams.c
Intel(R) C++ Intel(R) 64 Compiler XE for applications running on Intel(R) 64, Version 15.0.0.108 Build 20140726
Copyright (C) 1985-2014 Intel Corporation.  All rights reserved.

bit_reader.c
decode.c
huffman.c
state.c
streams.c

D:\brotli-master\dec>cd..

D:\brotli-master>cd enc

D:\brotli-master\enc>icl /O3 /c backward_references.cc block_splitter.cc brotli_bit_stream.cc encode.cc encode_parallel.cc entropy_encode.cc histogram.cc literal_cost.cc metablock.cc static_dict.cc streams.cc
Intel(R) C++ Intel(R) 64 Compiler XE for applications running on Intel(R) 64, Version 15.0.0.108 Build 20140726
Copyright (C) 1985-2014 Intel Corporation.  All rights reserved.

backward_references.cc
block_splitter.cc
brotli_bit_stream.cc
encode.cc
encode_parallel.cc
entropy_encode.cc
histogram.cc
literal_cost.cc
metablock.cc
static_dict.cc
streams.cc

D:\brotli-master\enc>cd..

D:\brotli-master>cd tools

D:\brotli-master\tools>icl /O3 bro.cc ..\dec\bit_reader.obj ..\dec\decode.obj ..\dec\huffman.obj ..\dec\state.obj ..\dec\streams.obj ..\enc\backward_references.obj ..\enc\block_splitter.obj ..\enc\brotli_bit_stream.obj ..\enc\encode.obj ..\enc\encode_parallel.obj ..\enc\entropy_encode.obj ..\enc\histogram.obj ..\enc\literal_cost.obj ..\enc\metablock.obj ..\enc\static_dict.obj ..\enc\streams.obj
Intel(R) C++ Intel(R) 64 Compiler XE for applications running on Intel(R) 64, Version 15.0.0.108 Build 20140726
Copyright (C) 1985-2014 Intel Corporation.  All rights reserved.

bro.cc
Microsoft (R) Incremental Linker Version 10.00.30319.01
Copyright (C) Microsoft Corporation.  All rights reserved.

-out:bro.exe
bro.obj
..\dec\bit_reader.obj
..\dec\decode.obj
..\dec\huffman.obj
..\dec\state.obj
..\dec\streams.obj
..\enc\backward_references.obj
..\enc\block_splitter.obj
..\enc\brotli_bit_stream.obj
..\enc\encode.obj
..\enc\encode_parallel.obj
..\enc\entropy_encode.obj
..\enc\histogram.obj
..\enc\literal_cost.obj
..\enc\metablock.obj
..\enc\static_dict.obj
..\enc\streams.obj

D:\brotli-master\tools>dir br*.exe
 Volume in drive D is S640_Vol5
 Volume Serial Number is 5861-9E6C

 Directory of D:\brotli-master\tools

09/24/2015  06:56 AM         1,250,304 bro.exe
               1 File(s)      1,250,304 bytes
               0 Dir(s)   5,917,040,640 bytes free

D:\brotli-master\tools>bro
;
D:\brotli-master\tools>bro /?
Usage: bro [--force] [--quality n] [--decompress] [--input filename] [--output filename] [--repeat iters] [--verbose]

D:\brotli-master\tools>

AFAIR, three lines were manually modified in bro.cc:

#include "../dec/decode.h"
#include "../enc/encode.h"
#include "../enc/streams.h"

I think the brackets were replaced with quotes.

Please, share inhere similar approach for compilation, I want to have binary compiled with /O3 /arch:CORE-AVX2, also please consider making built-in benchmark as Yann's Zstd -b one. By the way I tried latest TurboBench but couldn't go past dictionary 24bit, that is, d29 failed to decompress?! Having built-in benchmark will be very informative in many ways, one of which is to test 16+MB windows heavily, some problems/drawbacks could be caught in the process. My personal experience with Conor's LZSSE2 (AVX2) and my Nakamichi was fruitful in that regard. Simply stated, benchmarking yields much more than one usually expect.

Yesterday I ran, in a hurry, bench for one 15MB html testfile and overlooked the iterations for compress, should have increased them to the default 3 in order to have stable values, stupid, now to undo the damage I reran it and demanded all the 3x11 modes - from 1 to 11 with windows 20/22/24 bit:

E:\Ryuugan_vs_lzbench_vs_TurboBench_(LzTurbo-OFFICIAL_vs_Zstd_vs_Oodle)_2018-Mar-16>"turbobench_Official_v18.03_-_build_16_Mar_2018.exe" Arabian_Nights_complete.html -eyappy/snappy_c/lzturbo,10,20,30//brotli,1d20,2d20,3d20,4d20,5d20,6d20,7d20,8d20,9d20,10d20,11d20/brotli,1d22,2d22,3d22,4d22,5d22,6d22,7d22,8d22,9d22,10d22,11d22/brotli,1d24,2d24,3d24,4d24,5d24,6d24,7d24,8d24,9d24,10d24,11d24 -g -I3 -J31 -k1 -B2G
TurboBench:  - Sun Mar 18 23:14:16 2018

      C Size  ratio%     C MB/s     D MB/s   Name            File
     3912935    25.1       0.37     274.31   brotli 11d24                     Arabian_Nights_complete.html.tbb
     4004033    25.7       0.41     318.37   brotli 11d22                     Arabian_Nights_complete.html.tbb
     4011405    25.7       0.70     243.06   brotli 10d24                     Arabian_Nights_complete.html.tbb
     4106505    26.4       0.77     280.06   brotli 10d22                     Arabian_Nights_complete.html.tbb
     4200095    27.0       0.43     344.93   brotli 11d20                     Arabian_Nights_complete.html.tbb
     4305808    27.6       0.93     298.99   brotli 10d20                     Arabian_Nights_complete.html.tbb
     4343656    27.9       2.79     280.84   brotli 9d24                      Arabian_Nights_complete.html.tbb
     4388021    28.2       3.43     301.82   brotli 9d22                      Arabian_Nights_complete.html.tbb
     4423269    28.4       4.42     291.98   brotli 8d24                      Arabian_Nights_complete.html.tbb
     4450817    28.6       4.55     322.36   brotli 8d22                      Arabian_Nights_complete.html.tbb
     4522816    29.0       6.57     303.88   brotli 7d24                      Arabian_Nights_complete.html.tbb
     4536975    29.1       7.90     328.92   brotli 7d22                      Arabian_Nights_complete.html.tbb
     4538817    29.1       7.18     373.07   brotli 9d20                      Arabian_Nights_complete.html.tbb
     4570979    29.3       9.70     360.59   brotli 8d20                      Arabian_Nights_complete.html.tbb
     4622302    29.7      11.78     358.36   brotli 7d20                      Arabian_Nights_complete.html.tbb
     4644643    29.8       9.15     309.48   brotli 6d24                      Arabian_Nights_complete.html.tbb
     4649587    29.8      12.42     342.51   brotli 6d22                      Arabian_Nights_complete.html.tbb
     4700151    30.2      16.52     357.89   brotli 6d20                      Arabian_Nights_complete.html.tbb
     4784408    30.7      13.52     327.23   brotli 5d24                      Arabian_Nights_complete.html.tbb
     4786984    30.7      17.34     336.35   brotli 5d22                      Arabian_Nights_complete.html.tbb
     4811899    30.9      21.15     356.82   brotli 5d20                      Arabian_Nights_complete.html.tbb
     4903876    31.5      21.65     294.66   brotli 4d24                      Arabian_Nights_complete.html.tbb
     4910658    31.5      29.61     331.01   brotli 4d22                      Arabian_Nights_complete.html.tbb
     4979271    32.0      29.08     354.81   brotli 4d20                      Arabian_Nights_complete.html.tbb
     5560004    35.7      59.03     309.91   brotli 3d20                      Arabian_Nights_complete.html.tbb
     5560804    35.7      62.06     262.31   brotli 3d24                      Arabian_Nights_complete.html.tbb
     5560811    35.7      56.49     302.19   brotli 3d22                      Arabian_Nights_complete.html.tbb
     5619611    36.1      67.74     284.11   brotli 2d22                      Arabian_Nights_complete.html.tbb
     5619733    36.1      76.64     300.18   brotli 2d20                      Arabian_Nights_complete.html.tbb
     5619749    36.1      72.97     253.02   brotli 2d24                      Arabian_Nights_complete.html.tbb
     5866802    37.6     107.70     233.66   brotli 1d24                      Arabian_Nights_complete.html.tbb
     5876696    37.7     119.45     264.86   brotli 1d22                      Arabian_Nights_complete.html.tbb
     5911301    37.9     133.30     280.41   brotli 1d20                      Arabian_Nights_complete.html.tbb
     6076566    39.0     166.92     724.17   lzturbo 30                       Arabian_Nights_complete.html.tbb
     8443243    54.2     268.12     916.57   lzturbo 20                       Arabian_Nights_complete.html.tbb
     8630338    55.4      67.66    1307.12   yappy                            Arabian_Nights_complete.html.tbb
     9027159    57.9     274.43     825.53   snappy_c                         Arabian_Nights_complete.html.tbb
     9115662    58.5     305.32    2535.54   lzturbo 10                       Arabian_Nights_complete.html.tbb

E:\Ryuugan_vs_lzbench_vs_TurboBench_(LzTurbo-OFFICIAL_vs_Zstd_vs_Oodle)_2018-Mar-16>"turbobench_Official_v18.03_-_build_16_Mar_2018.exe" Arabian_Nights_complete.html -elibdeflate,1,2,3,4,5,6,7,8,9,12/zlib,1,2,3,4,5,6,7,8,9 -g -I3 -J31 -k1 -B2G
TurboBench:  - Sun Mar 18 23:50:37 2018

      C Size  ratio%     C MB/s     D MB/s   Name            File
     5242942    33.6       5.29     545.43   libdeflate 12                    Arabian_Nights_complete.html.tbb
     5259086    33.7       8.39     569.09   libdeflate 9                     Arabian_Nights_complete.html.tbb
     5340204    34.3      11.51     576.37   libdeflate 8                     Arabian_Nights_complete.html.tbb
     5478319    35.2       9.53     237.37   zlib 9                           Arabian_Nights_complete.html.tbb
     5478323    35.2       9.81     237.63   zlib 8                           Arabian_Nights_complete.html.tbb
     5492801    35.2      12.54     249.26   zlib 7                           Arabian_Nights_complete.html.tbb
     5493806    35.3      37.60     594.56   libdeflate 7                     Arabian_Nights_complete.html.tbb
     5515217    35.4      15.79     236.86   zlib 6                           Arabian_Nights_complete.html.tbb
     5522003    35.4      40.77     588.90   libdeflate 6                     Arabian_Nights_complete.html.tbb
     5566382    35.7      59.15     583.56   libdeflate 5                     Arabian_Nights_complete.html.tbb
     5614102    36.0      24.20     233.67   zlib 5                           Arabian_Nights_complete.html.tbb
     5712736    36.7      69.44     629.20   libdeflate 4                     Arabian_Nights_complete.html.tbb
     5772524    37.0      84.09     621.10   libdeflate 3                     Arabian_Nights_complete.html.tbb
     5832594    37.4      38.33     249.31   zlib 4                           Arabian_Nights_complete.html.tbb
     5870312    37.7     100.24     606.29   libdeflate 2                     Arabian_Nights_complete.html.tbb
     6061515    38.9      35.85     247.95   zlib 3                           Arabian_Nights_complete.html.tbb
     6114129    39.2     118.54     576.44   libdeflate 1                     Arabian_Nights_complete.html.tbb
     6322273    40.6      47.13     236.63   zlib 2                           Arabian_Nights_complete.html.tbb
     6604448    42.4      53.34     232.78   zlib 1                           Arabian_Nights_complete.html.tbb

E:\Ryuugan_vs_lzbench_vs_TurboBench_(LzTurbo-OFFICIAL_vs_Zstd_vs_Oodle)_2018-Mar-16>

The testmachine is a laptop running Windows 10, with i5-7200u 3GHz, DDR4 2133MHz.

Another thing that interests me, is it allowed dictionary to be, say, 16bit or 18bit? Is 30 the maximum?

Just added windows binaries for v1.0.3, going to release v1.0.4 soon, and it will also feature windows binaries. Added project to create JNI binaries as well (volunteers wanted).

As before, it is possible to create something like makeExe.bat, though it now will be longer =) In this article I describe an easy way to cross-compile for windows, with minimal requirements: just bash and docker. I think it is possible to reduce it just to docker; all the things done in bash is generation of 4 small files and launch of docker.

Using icl is difficult for legal reasons - IIRC, Intel prohibits to publish the results of compilation of opensource projects by corporate users =(

Also there is no easy way to install ICL without a key...

Recently we have published "large window brotli" feature. Now window could be up to 30 bits. Perhaps this limit will be changed in future; 30 is chosen as a balance between utility and simplicity of implementation.

>Recently we have published "large window brotli" feature. Now window could be up to 30 bits.

Just downloaded and ran v 1.0.3, but couldn't see

Key changes:
    new feature: "Large Window Brotli"

The window is still up to 24, or 16MB, how one is supposed to run tests with 1GB?

As for Intel, I consider myself a fair tester since v12, Intel graciously allowed temporary key activation which, somehow, don't ask how, on my laptops reincarnated in next versions that I used to evaluate - v13, v14, v15. My belief and understanding is that Intel did good by allowing evaluating i.e. creating executables and seeing first-handedly how powerful really it is - all in the spirit of fair use, that is, no commercial or similar transgressions. For a time I used in parallel GCC 6.3.0, but this laptop died, didn't care to reinstall since new faster versions appeared.

To make sure that admins do not start using "large window brotli" for http, we made it accessible only from API in v1.0.3 In v1.0.4 (and current tip of the tree) it is possible to use the feature from CLI.

I agree, fair use is a good thing. But evaluation- and integration testing-wise the licensing is done the wrong way. One can not use icl from docker container or run it on TravisCI, just to make sure the source code compiles well on their compiler...

>But evaluation- and integration testing-wise the licensing is done the wrong way. One can not use icl from docker container or run it on TravisCI, just to make sure the source code compiles well on their compiler...

Hm, that's not good indeed, even to my amateurish standards, in my view, one excellent tool/library/API should be compileable on vast set of platforms, the first that comes to mind is LZO, Markus did make it portable AFAIK. My C/environments knowledge is really basic to express opinions on portability issues, but one phrase caught my attention "Military Grade", somewhere on LZO descriptions, very ringy!

Looking forward testing v1.0.4, funny, no one benchmarked windows 15/16, wanna know how they fare against the superb LIBDEFLATE 12. Are there plans for making some built-in benchmark similar to Zstd's one? One of the reasons to want making my own executables with Intel v15.0 was to wrap up the [de]compression invocations with some basic time stats. The other variant is to rely on Lzbench/Turbobench - which I prefer actually.

Thanks to powturbo, for the first time I witnessed the 16+MB window of Brotli:

C:\Ryuugan_vs_lzbench_vs_TurboBench_(LzTurbo-OFFICIAL_vs_Zstd_vs_Oodle)_2018-Apr-16>"turbobench_v18.04_-_build_16_Apr_2018.exe" "Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar" -ebzip2/lzlib,9d29fb273/lzham,4fb258:x4:d29/lzma,9d29:fb273:mf=bt4/libdeflate,1,5,12/oodle,19,49,89,112,114,116,118,129/lzsse2,1,17/lzturbo,19,12,29,22,39,32,49,59,59t2,59t4/zstd,1,5,12,22/lizard,19,29,39,49/brotli,1,5,11/brotli,11d29/lzma,9/xpack,1,9/chameleon,2/density,3/lzham,4/trle/bsc,3,6/zpaq,2,5 -g -I3 -J15 -k1 -B2G
TurboBench:  - Tue Apr 17 01:27:19 2018

      C Size  ratio%     C MB/s     D MB/s   Name            File
    22279381    20.7       7.75      35.37   lzturbo 59                       Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.tbb
    22280917    20.7       8.18      50.68   lzturbo 59t2                     Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.tbb
    22283621    20.7       8.34      59.18   lzturbo 59t4                     Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.tbb
    22817313    21.2       0.36       0.35   zpaq 5                           Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.tbb
    23581054    21.9      19.43       6.62   bsc 6                            Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.tbb
    27071417    25.1       0.70      71.20   lzturbo 49                       Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.tbb
    27494771    25.5       0.90      53.86   lzlib 9d29fb273                  Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.tbb
    27526894    25.5       0.94      70.72   lzma 9d29:fb273:mf=bt4           Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.tbb
    27600942    25.6       0.35     215.74   brotli 11d29                     Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.tbb
    27642496    25.6       0.96      70.66   lzma 9                           Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.tbb
    27750101    25.7       0.33     166.38   lzham 4fb258:x4:d29              Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.tbb
    27769820    25.8       0.80     167.15   lzham 4                          Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.tbb
    27942261    25.9       0.81     509.83   lzturbo 39                       Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.tbb
    28041924    26.0       1.12     522.47   zstd 22                          Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.tbb
    28654421    26.6       0.45     479.85   oodle 129                        Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.tbb
    28654421    26.6       0.52     479.77   oodle 89                         Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.tbb
    29616048    27.5       0.23     257.34   oodle 19                         Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.tbb
    30390676    28.2       0.40     292.52   brotli 11                        Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.tbb
    31079132    28.8      19.88      12.74   bsc 3                            Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.tbb
    31579501    29.3       0.83     877.02   lzturbo 29                       Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.tbb
    31734097    29.4      11.45      21.54   bzip2                            Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.tbb
    32856071    30.5      27.08     630.66   lzturbo 32                       Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.tbb
    34085035    31.6       0.47     371.09   xpack 9                          Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.tbb
    35009827    32.5       5.89     630.14   zstd 12                          Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.tbb
    35077489    32.5       1.34     719.17   lizard 49                        Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.tbb
    35786915                        773      Nakamichi 'Ryuugan-ditto-1TB'    ! Intel AVX2 compile outside TurboBench !
    35855068    33.3       3.46      67.58   zpaq 2                           Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.tbb
    36277020    33.7      18.11     316.25   brotli 5                         Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.tbb
    39674735    36.8      73.49     594.78   zstd 5                           Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.tbb
    40516809    37.6       5.93     515.48   libdeflate 12                    Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.tbb
    41420278    38.4       1.38    1072.32   lizard 29                        Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.tbb
    42104125    39.1       6.57    1505.34   lizard 39                        Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.tbb
    42528049    39.5      58.78     526.40   libdeflate 5                     Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.tbb
    42926053                       2862      LZSSE2 L17                       ! Intel AVX2 compile outside TurboBench !
    42926057    39.8       6.71    1923.31   lzsse2 17                        Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.tbb
    43339102    40.2      27.10     969.62   lzturbo 22                       Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.tbb
    44012280    40.8       0.44    2044.62   oodle 116                        Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.tbb
    44448715    41.2       0.61    2036.35   oodle 118                        Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.tbb
    44972225    41.7     120.56     263.46   brotli 1                         Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.tbb
    45747038    42.4     111.35     518.12   libdeflate 1                     Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.tbb
    45853852    42.5      87.73     523.50   xpack 1                          Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.tbb
    47817239    44.4       0.95    3001.51   lzturbo 19                       Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.tbb
    47834366    44.4       6.79    2289.38   lizard 19                        Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.tbb
    47836827    44.4       3.07    2207.38   oodle 49                         Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.tbb
    47979874    44.5     216.15     734.33   zstd 1                           Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.tbb
    48144800    44.7      17.54    2802.28   oodle 114                        Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.tbb
    50763849    47.1      42.34    3037.37   lzturbo 12                       Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.tbb
    51412000    47.7     279.76     293.62   density 3                        Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.tbb
    53894668    50.0      14.32    1589.62   lzsse2 1                         Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.tbb
    58087170    53.9      66.83    2877.54   oodle 112                        Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.tbb
    64897616    60.2    1538.32    2113.42   chameleon 2                      Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.tbb
   107743877   100.0     193.79    2003.09   trle                             Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.tbb

C:\Ryuugan_vs_lzbench_vs_TurboBench_(LzTurbo-OFFICIAL_vs_Zstd_vs_Oodle)_2018-Apr-16>

Is this not pure awesomeness: lzturbo 59 being 35.37/0.35= 101.0x faster than zpaq 5 lzturbo 59 being 35.37/6.62= 5.3x faster than bsc 6

Still, in v.1.0.4 binary the window is limited to 24?! If you wanna see a particular file benchmarked, just give me the URL ... zzx-01

No, no, cannot miss one dataset that has been crying to be benchmarked for times unremembered- the W3 itself:

mw3_c666aab0b2abf6fbc4d712d07b47bc39

https://en.wikipedia.org/wiki/Webster%27s_Third_New_International_Dictionary

EDIT: Always love to get a hold of thickish books, shrinking this thickness to 125223987:15239397 or 8:1 ...

merriam-webster3_5a9a3c464f161 image

C:\Ryuugan_vs_lzbench_vs_TurboBench_(LzTurbo-OFFICIAL_vs_Zstd_vs_Oodle)_2018-Apr-16>"turbobench_v18.04_-_build_16_Apr_2018.exe" "Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl" -ebzip2/lzlib,9d29fb273/lzham,4fb258:x4:d29/lzma,9d29:fb273:mf=bt4/libdeflate,1,5,12/oodle,19,49,89,112,114,116,118,129/lzsse2,1,17/lzturbo,19,12,29,22,39,32,49,59,59t2,59t4/zstd,1,5,12,22/lizard,19,29,39,49/brotli,1,5,11/brotli,11d29/lzma,9/xpack,1,9/chameleon,2/density,3/lzham,4/trle/bsc,3,6/zpaq,2,5 -g -I3 -J15 -k1 -B2G
TurboBench:  - Tue Apr 17 23:42:21 2018

      C Size  ratio%     C MB/s     D MB/s   Name            File
    14871463    11.9       0.38       0.38   zpaq 5                           Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    15239397    12.2       8.76      42.19   lzturbo 59                       Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    15240985    12.2       9.13      58.66   lzturbo 59t2                     Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    15243625    12.2       9.23      68.64   lzturbo 59t4                     Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    16506844    13.2      28.80      10.12   bsc 6                            Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    19204214    15.3       0.68     115.16   lzturbo 49                       Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    19480822    15.6       1.02      78.50   lzlib 9d29fb273                  Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    19519434    15.6       1.08     111.01   lzma 9d29:fb273:mf=bt4           Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    19559693    15.6       0.39     313.37   brotli 11d29                     Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    19609082    15.7       0.26     267.18   lzham 4fb258:x4:d29              Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    19650726    15.7       0.81     856.34   lzturbo 39                       Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    19975078    16.0       1.37     704.91   zstd 22                          Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    20004388    16.0       0.95     374.52   lzham 4                          Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    20099339    16.0       1.35     108.71   lzma 9                           Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    20861759    16.7       0.51     673.71   oodle 129                        Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    20861759    16.7       0.62     673.46   oodle 89                         Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    20960314    16.7       0.27     416.38   oodle 19                         Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    21473387    17.1      10.03      28.00   bzip2                            Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    21561069    17.2       0.44     428.18   brotli 11                        Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    22566183    18.0       0.83    1190.20   lzturbo 29                       Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    22733734    18.2      28.54      17.39   bsc 3                            Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    24357693    19.4       0.84     625.78   xpack 9                          Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    24807059    19.8       1.41     985.38   lizard 49                        Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    24930702    19.9      41.46     861.31   lzturbo 32                       Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    25336495    20.2      11.22     855.66   zstd 12                          Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    25518771    20.4       4.70      89.82   zpaq 2                           Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    26302175    21.0      27.11     441.27   brotli 5                         Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    29086907    23.2       3.96     746.88   libdeflate 12                    Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    29092999    23.2       1.50    1372.36   lizard 29                        Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    29886124    23.9     110.24     868.56   zstd 5                           Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    30582918    24.4       0.47    1857.72   oodle 116                        Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    30745232    24.6       0.69    1868.06   oodle 118                        Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    31131500    24.9       4.44    1613.22   lizard 39                        Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    31636977    25.3       5.66    2615.21   lzsse2 17                        Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    31670794    25.3      87.24     737.90   libdeflate 5                     Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    32592313    26.0      41.87    1206.57   lzturbo 22                       Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    34701844    27.7     163.76     358.96   brotli 1                         Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    34835884    27.8       0.93    2748.84   lzturbo 19                       Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    34858499    27.8       4.73    2233.62   lizard 19                        Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    34864016    27.8       3.49    1931.08   oodle 49                         Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    34927455    27.9     272.63     872.31   zstd 1                           Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    36016662    28.8     114.84     736.05   xpack 1                          Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    36232167    28.9     146.58     725.14   libdeflate 1                     Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    36305868    29.0      27.09    2633.08   oodle 114                        Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    39254938    31.3      60.32    2667.18   lzturbo 12                       Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    39274452    31.4     286.62     301.49   density 3                        Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    39417340    31.5      12.67    2223.11   lzsse2 1                         Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    44746189    35.7     103.65    2752.77   oodle 112                        Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    70877719    56.6    1576.73    2757.56   chameleon 2                      Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
   125223987   100.0     192.47    2127.13   trle                             Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb

C:\Ryuugan_vs_lzbench_vs_TurboBench_(LzTurbo-OFFICIAL_vs_Zstd_vs_Oodle)_2018-Apr-16>

For those who are unaware of the fact, W3 is second only to OED. Ugh, the natural next step is guess what dataset... the paragonic 534MB (d29 is to flex muscles) of Oxford are under way...

Grmbl, the file has to be under ~450MB, the insufficient 8GB on 'Compressionette' are the cause for disfiguring the [de]compression rates for OED, due to HDD thrashing, so I will run it when 16GB are available.

Continuing exploring the bigger Brotli's windows, inhere 27+bit - the full documentation folder of Intel Parallel Studio XE - 166 MB tarred file.

intel22_shop1_254787

C:\Ryuugan_vs_lzbench_vs_TurboBench_(LzTurbo-OFFICIAL_vs_Zstd_vs_Oodle)_2018-Apr-22>"turbobench_v18.04_-_build_22_Apr_2018.exe" Documentation_Composer_XE_2015.tar -ebzip2/lzlib,9d29fb273/lzham,4fb258:x4:d29/lzma,9d29:fb273:mf=bt4/libdeflate,1,5,12/oodle,19,49,89,112,114,116,118,129/lzsse2,1,17/lzturbo,19,12,29,22,39,32,49,59,59t2,59t4/zstd,1,5,12,22/lizard,19,29,39,49/brotli,1,5,11/brotli,11d29/lzma,9/xpack,1,9/chameleon,2/density,3/lzham,4/trle/bsc,3,6/zpaq,2,5 -g -I3 -J31 -k1 -B2G
TurboBench:  - Tue Apr 24 01:08:38 2018

      C Size  ratio%     C MB/s     D MB/s   Name            File
    29545224    17.0       0.75     100.68   lzturbo 49                       Documentation_Composer_XE_2015.tar.tbb
    30135901    17.3       1.52      65.17   lzlib 9d29fb273                  Documentation_Composer_XE_2015.tar.tbb
    30186666    17.3       0.49     339.88   brotli 11d29                     Documentation_Composer_XE_2015.tar.tbb
    30220005    17.4       1.71      95.64   lzma 9d29:fb273:mf=bt4           Documentation_Composer_XE_2015.tar.tbb
    30864067    17.7       0.11     415.03   lzham 4fb258:x4:d29              Documentation_Composer_XE_2015.tar.tbb
    30882973    17.7       1.21    2087.09   lzturbo 39                       Documentation_Composer_XE_2015.tar.tbb
    30890946    17.7       3.36      94.02   lzma 9                           Documentation_Composer_XE_2015.tar.tbb
    31026330    17.8       2.06    1562.69   zstd 22                          Documentation_Composer_XE_2015.tar.tbb
    31468806    18.1       1.67     411.25   lzham 4                          Documentation_Composer_XE_2015.tar.tbb
    31759169    18.2       0.95    1676.41   oodle 129                        Documentation_Composer_XE_2015.tar.tbb
    31760340    18.2       1.10    1673.90   oodle 89                         Documentation_Composer_XE_2015.tar.tbb
    31855429    18.3       9.57      33.52   lzturbo 59                       Documentation_Composer_XE_2015.tar.tbb
    31860989    18.3      10.23      46.82   lzturbo 59t2                     Documentation_Composer_XE_2015.tar.tbb
    31870733    18.3      10.30      51.38   lzturbo 59t4                     Documentation_Composer_XE_2015.tar.tbb
    32004878    18.4       0.56     808.11   oodle 19                         Documentation_Composer_XE_2015.tar.tbb
    32346682    18.6       0.37       0.37   zpaq 5                           Documentation_Composer_XE_2015.tar.tbb
    32470130    18.7      15.12      12.95   bsc 6                            Documentation_Composer_XE_2015.tar.tbb
    33013279    19.0       1.33    2896.75   lzturbo 29                       Documentation_Composer_XE_2015.tar.tbb
    33231171    19.1       0.41     927.94   xpack 9                          Documentation_Composer_XE_2015.tar.tbb
    34302252    19.7      68.09    2072.99   lzturbo 32                       Documentation_Composer_XE_2015.tar.tbb
    36088629    20.7       0.52     390.77   brotli 11                        Documentation_Composer_XE_2015.tar.tbb
    36586540    21.0       1.15    3635.88   oodle 118                        Documentation_Composer_XE_2015.tar.tbb
    36688592    21.1       0.80    3565.00   oodle 116                        Documentation_Composer_XE_2015.tar.tbb
    38389039    22.0       1.01    2605.20   lizard 49                        Documentation_Composer_XE_2015.tar.tbb
    38651612    22.2      24.69      14.14   bsc 3                            Documentation_Composer_XE_2015.tar.tbb
    39069930    22.4      34.09    1895.26   zstd 12                          Documentation_Composer_XE_2015.tar.tbb
    39182283    22.5       7.38     102.90   zpaq 2                           Documentation_Composer_XE_2015.tar.tbb
    39391791    22.6       1.10    3242.64   lizard 29                        Documentation_Composer_XE_2015.tar.tbb
    39452631    22.7      42.22     522.99   brotli 5                         Documentation_Composer_XE_2015.tar.tbb
    39982791    23.0      43.69    3941.06   oodle 114                        Documentation_Composer_XE_2015.tar.tbb
    42142284    24.2      66.35    2785.52   lzturbo 22                       Documentation_Composer_XE_2015.tar.tbb
    42928633    24.7       9.63      30.69   bzip2                            Documentation_Composer_XE_2015.tar.tbb
    43021036    24.7     142.75    1542.27   zstd 5                           Documentation_Composer_XE_2015.tar.tbb
    45864980    26.3     144.76    3990.02   oodle 112                        Documentation_Composer_XE_2015.tar.tbb
    47258638    27.1     107.68     830.78   xpack 1                          Documentation_Composer_XE_2015.tar.tbb
    47619118    27.4     469.33    1515.97   zstd 1                           Documentation_Composer_XE_2015.tar.tbb
    47898670    27.5       5.96     870.36   libdeflate 12                    Documentation_Composer_XE_2015.tar.tbb
    49143452    28.2     304.98     441.79   brotli 1                         Documentation_Composer_XE_2015.tar.tbb
    49419387    28.4       1.92    2866.65   lizard 39                        Documentation_Composer_XE_2015.tar.tbb
    49717522    28.6     119.38     869.60   libdeflate 5                     Documentation_Composer_XE_2015.tar.tbb
    51205991    29.4       4.21    3153.71   oodle 49                         Documentation_Composer_XE_2015.tar.tbb
    51242440    29.4       2.19    3787.84   lizard 19                        Documentation_Composer_XE_2015.tar.tbb
    51348728    29.5       1.21    4383.86   lzturbo 19                       Documentation_Composer_XE_2015.tar.tbb
    52436989    30.1     152.29     682.18   libdeflate 1                     Documentation_Composer_XE_2015.tar.tbb
    53565880    30.8     110.27    4151.58   lzturbo 12                       Documentation_Composer_XE_2015.tar.tbb
    57537060    33.0       2.15    2041.44   lzsse2 17                        Documentation_Composer_XE_2015.tar.tbb
    61860799    35.5      17.18    1868.29   lzsse2 1                         Documentation_Composer_XE_2015.tar.tbb
    62167830    35.7     327.17     338.38   density 3                        Documentation_Composer_XE_2015.tar.tbb
   114138236    65.6    1595.28    2355.26   chameleon 2                      Documentation_Composer_XE_2015.tar.tbb
   160701004    92.3     213.70    7290.03   trle                             Documentation_Composer_XE_2015.tar.tbb

C:\Ryuugan_vs_lzbench_vs_TurboBench_(LzTurbo-OFFICIAL_vs_Zstd_vs_Oodle)_2018-Apr-22>

The latest Brotli with larger window beats LZMA in compression ratio and in decompression rate - 339/95= 3.5x faster! Bah, just look at LzTurbo 39:

2087/95= 21.9x faster than LZMA 9d29;
2087/339= 6.1x faster than Brotli 11d29;
2087/1676= 1.2x faster than Oodle 'Hydra' (oo2core_5_win64.dll).

The next one will be the most popular dataset - SCC - Silesia Compression Corpus ...

Crunching 211,938,580 bytes long Silesia_compression_corpus ...

godlo_pozytyw_angielskie_krzywe_smaller

C:\2018-Apr-22>"turbobench_v18.04_-_build_22_Apr_2018.exe" Silesia_compression_corpus -ebzip2/lzlib,9d29fb273/lzham,4fb258:x4:d29/lzma,9d29:fb273:mf=bt4/libdeflate,1,5,12/oodle,19,49,89,112,114,116,118,129/lzsse2,1,17/lzturbo,19,12,29,22,39,32,49,59,59t2,59t4/zstd,1,5,12,22/lizard,19,29,39,49/brotli,1,5,11/brotli,11d29/lzma,9/xpack,1,9/chameleon,2/density,3/lzham,4/trle/bsc,3,6/zpaq,2,5 -g -I3 -J31 -k1 -B2G
TurboBench:  - Tue Apr 24 13:59:07 2018

      C Size  ratio%     C MB/s     D MB/s   Name            File
    40485323    19.1       0.35       0.35   zpaq 5                           Silesia_compression_corpus.tbb
    48244307    22.8       1.33      53.77   lzlib 9d29fb273                  Silesia_compression_corpus.tbb
    48382888    22.8       1.45      71.68   lzma 9d29:fb273:mf=bt4           Silesia_compression_corpus.tbb
    48577832    22.9       0.85      69.37   lzturbo 49                       Silesia_compression_corpus.tbb
    48751843    23.0       1.89      71.22   lzma 9                           Silesia_compression_corpus.tbb
    49095144    23.2      14.76      10.11   bsc 6                            Silesia_compression_corpus.tbb
    49539504    23.4       0.35     408.37   brotli 11d29                     Silesia_compression_corpus.tbb
    50124186    23.7       0.20     204.48   lzham 4fb258:x4:d29              Silesia_compression_corpus.tbb
    50424103    23.8       0.38     301.43   brotli 11                        Silesia_compression_corpus.tbb
    50599493    23.9       9.49      31.63   lzturbo 59                       Silesia_compression_corpus.tbb
    50604053    23.9      10.40      45.54   lzturbo 59t2                     Silesia_compression_corpus.tbb
    50612477    23.9      10.64      52.48   lzturbo 59t4                     Silesia_compression_corpus.tbb
    50959089    24.0       1.32     205.51   lzham 4                          Silesia_compression_corpus.tbb
    51381492    24.2       0.60    1047.70   oodle 129                        Silesia_compression_corpus.tbb
    51384209    24.2       0.73    1046.86   oodle 89                         Silesia_compression_corpus.tbb
    51676247    24.4       1.11    1055.44   lzturbo 39                       Silesia_compression_corpus.tbb
    52662820    24.8       0.36     634.43   oodle 19                         Silesia_compression_corpus.tbb
    52831997    24.9       1.97     846.76   zstd 22                          Silesia_compression_corpus.tbb
    53411512    25.2      21.20      13.18   bsc 3                            Silesia_compression_corpus.tbb
    54592581    25.8      10.38      26.89   bzip2                            Silesia_compression_corpus.tbb
    56929077    26.9       0.38     511.42   xpack 9                          Silesia_compression_corpus.tbb
    59116885    27.9      12.48     827.16   zstd 12                          Silesia_compression_corpus.tbb
    59541983    28.1      34.92     968.57   lzturbo 32                       Silesia_compression_corpus.tbb
    59583107    28.1      22.66     353.83   brotli 5                         Silesia_compression_corpus.tbb
    60694955    28.6       1.57    1155.09   lizard 49                        Silesia_compression_corpus.tbb
    61105625    28.8       1.23    1294.45   lzturbo 29                       Silesia_compression_corpus.tbb
    64188369    30.3       4.50      80.48   zpaq 2                           Silesia_compression_corpus.tbb
    64750390    30.6      95.91     782.01   zstd 5                           Silesia_compression_corpus.tbb
    64802668    30.6       5.03     586.23   libdeflate 12                    Silesia_compression_corpus.tbb
    68495938    32.3      84.44     608.60   libdeflate 5                     Silesia_compression_corpus.tbb
    68694935    32.4       1.67    1895.03   lizard 29                        Silesia_compression_corpus.tbb
    68975136    32.5       0.83    2727.86   oodle 118                        Silesia_compression_corpus.tbb
    69508100    32.8       0.66    2763.29   oodle 116                        Silesia_compression_corpus.tbb
    69815768    32.9       3.73    1663.31   lizard 39                        Silesia_compression_corpus.tbb
    70324086    33.2      94.56     564.77   xpack 1                          Silesia_compression_corpus.tbb
    73317933    34.6     131.72     575.68   libdeflate 1                     Silesia_compression_corpus.tbb
    73501173    34.7     186.29     300.28   brotli 1                         Silesia_compression_corpus.tbb
    73659096    34.8     311.23     901.83   zstd 1                           Silesia_compression_corpus.tbb
    73801396    34.8      35.99    1463.75   lzturbo 22                       Silesia_compression_corpus.tbb
    76694216    36.2      25.44    3369.40   oodle 114                        Silesia_compression_corpus.tbb
    77333037    36.5       1.41    3386.84   lzturbo 19                       Silesia_compression_corpus.tbb
    77334912    36.5       3.88    2338.74   oodle 49                         Silesia_compression_corpus.tbb
    77414284    36.5       4.02    2734.30   lizard 19                        Silesia_compression_corpus.tbb
    82890446    39.1      64.45    3390.96   lzturbo 12                       Silesia_compression_corpus.tbb
    88627496    41.8     288.51     307.01   density 3                        Silesia_compression_corpus.tbb
    89341342    42.2      85.52    3137.09   oodle 112                        Silesia_compression_corpus.tbb
   134145400    63.3    1614.11    2325.44   chameleon 2                      Silesia_compression_corpus.tbb
   200581857    94.6     218.46    4838.23   trle                             Silesia_compression_corpus.tbb

Failed:

           0     0.0       0.00       0.00   lzsse2 1         Silesia_compression_corpus
ERROR at 0:2a, 9a
           0     0.0       0.00       0.00   lzsse2 17        Silesia_compression_corpus
ERROR at 0:2a, 9a

Crunching 465,457,152 bytes long mingw-w64_x86_64-6.3.0-posix-seh-rt_v5-rev2.tar ...

gnumeditate

C:\2018-Apr-22>"turbobench_v18.04_-_build_22_Apr_2018.exe" mingw-w64_x86_64-6.3.0-posix-seh-rt_v5-rev2.tar -ebzip2/lzlib,9d29fb273/lzham,4fb258:x4:d29/lzma,9d29:fb273:mf=bt4/libdeflate,1,5,12/oodle,19,49,89,112,114,116,118,129/lzsse2,1,17/lzturbo,19,12,29,22,39,32,49,59,59t2,59t4/zstd,1,5,12,22/lizard,19,29,39,49/brotli,1,5,11/brotli,11d29/lzma,9/xpack,1,9/chameleon,2/density,3/lzham,4/trle/bsc,3,6/zpaq,2,5 -g -I3 -J31 -k1 -B2G
TurboBench:  - Tue Apr 24 08:29:39 2018

      C Size  ratio%     C MB/s     D MB/s   Name            File
    42172655     9.1       0.83     171.90   lzturbo 49                       mingw-w64_x86_64-6.3.0-posix-seh-rt_v5-rev2.tar.tbb
    42197387     9.1       1.45     104.95   lzlib 9d29fb273                  mingw-w64_x86_64-6.3.0-posix-seh-rt_v5-rev2.tar.tbb
    42457543     9.1       1.59     163.98   lzma 9d29:fb273:mf=bt4           mingw-w64_x86_64-6.3.0-posix-seh-rt_v5-rev2.tar.tbb
    42988097     9.2       0.44     439.45   brotli 11d29                     mingw-w64_x86_64-6.3.0-posix-seh-rt_v5-rev2.tar.tbb
    44372549     9.5       0.11     432.19   lzham 4fb258:x4:d29              mingw-w64_x86_64-6.3.0-posix-seh-rt_v5-rev2.tar.tbb
    45218184     9.7       0.63    1635.80   oodle 129                        mingw-w64_x86_64-6.3.0-posix-seh-rt_v5-rev2.tar.tbb
    45220234     9.7       0.71    1633.34   oodle 89                         mingw-w64_x86_64-6.3.0-posix-seh-rt_v5-rev2.tar.tbb
    45752167     9.8       1.54     434.31   lzham 4                          mingw-w64_x86_64-6.3.0-posix-seh-rt_v5-rev2.tar.tbb
    45874938     9.9       1.29    1804.96   lzturbo 39                       mingw-w64_x86_64-6.3.0-posix-seh-rt_v5-rev2.tar.tbb
    47523378    10.2       0.47     986.98   oodle 19                         mingw-w64_x86_64-6.3.0-posix-seh-rt_v5-rev2.tar.tbb
    48042802    10.3       2.64     146.84   lzma 9                           mingw-w64_x86_64-6.3.0-posix-seh-rt_v5-rev2.tar.tbb
    51380197    11.0       2.09    1212.26   zstd 22                          mingw-w64_x86_64-6.3.0-posix-seh-rt_v5-rev2.tar.tbb
    56362497    12.1       1.44    2316.65   lzturbo 29                       mingw-w64_x86_64-6.3.0-posix-seh-rt_v5-rev2.tar.tbb
    58843125    12.6      77.93    1756.14   lzturbo 32                       mingw-w64_x86_64-6.3.0-posix-seh-rt_v5-rev2.tar.tbb
    59674611    12.8       0.37       0.37   zpaq 5                           mingw-w64_x86_64-6.3.0-posix-seh-rt_v5-rev2.tar.tbb
    60760645    13.1      10.48      38.58   lzturbo 59                       mingw-w64_x86_64-6.3.0-posix-seh-rt_v5-rev2.tar.tbb
    60765065    13.1      11.16      54.02   lzturbo 59t2                     mingw-w64_x86_64-6.3.0-posix-seh-rt_v5-rev2.tar.tbb
    60775397    13.1      11.34      60.71   lzturbo 59t4                     mingw-w64_x86_64-6.3.0-posix-seh-rt_v5-rev2.tar.tbb
    64328773    13.8       0.76    4182.98   oodle 118                        mingw-w64_x86_64-6.3.0-posix-seh-rt_v5-rev2.tar.tbb
    64404059    13.8       0.66    4205.24   oodle 116                        mingw-w64_x86_64-6.3.0-posix-seh-rt_v5-rev2.tar.tbb
    66049348    14.2      19.73      13.29   bsc 6                            mingw-w64_x86_64-6.3.0-posix-seh-rt_v5-rev2.tar.tbb
    73718680    15.8       0.39     390.82   brotli 11                        mingw-w64_x86_64-6.3.0-posix-seh-rt_v5-rev2.tar.tbb
    81847492    17.6      27.66      15.76   bsc 3                            mingw-w64_x86_64-6.3.0-posix-seh-rt_v5-rev2.tar.tbb
    88006994    18.9      33.03     444.89   brotli 5                         mingw-w64_x86_64-6.3.0-posix-seh-rt_v5-rev2.tar.tbb
    88252775    19.0       1.49    2276.70   lizard 49                        mingw-w64_x86_64-6.3.0-posix-seh-rt_v5-rev2.tar.tbb
    88458606    19.0      27.15    1137.21   zstd 12                          mingw-w64_x86_64-6.3.0-posix-seh-rt_v5-rev2.tar.tbb
    93127699    20.0      44.28    4652.62   oodle 114                        mingw-w64_x86_64-6.3.0-posix-seh-rt_v5-rev2.tar.tbb
    94672486    20.3       1.60    2650.11   lizard 29                        mingw-w64_x86_64-6.3.0-posix-seh-rt_v5-rev2.tar.tbb
    97698006    21.0       5.24      97.89   zpaq 2                           mingw-w64_x86_64-6.3.0-posix-seh-rt_v5-rev2.tar.tbb
   102907230    22.1      55.54    1835.16   lzturbo 22                       mingw-w64_x86_64-6.3.0-posix-seh-rt_v5-rev2.tar.tbb
   111853281    24.0      11.11      31.73   bzip2                            mingw-w64_x86_64-6.3.0-posix-seh-rt_v5-rev2.tar.tbb
   112029945    24.1     135.35    4169.23   oodle 112                        mingw-w64_x86_64-6.3.0-posix-seh-rt_v5-rev2.tar.tbb
   114389579    24.6     122.18    1386.40   zstd 5                           mingw-w64_x86_64-6.3.0-posix-seh-rt_v5-rev2.tar.tbb
   121900684    26.2       5.60     606.43   libdeflate 12                    mingw-w64_x86_64-6.3.0-posix-seh-rt_v5-rev2.tar.tbb
   127689179    27.4      98.73     611.28   libdeflate 5                     mingw-w64_x86_64-6.3.0-posix-seh-rt_v5-rev2.tar.tbb
   131622116    28.3     401.37    1038.21   zstd 1                           mingw-w64_x86_64-6.3.0-posix-seh-rt_v5-rev2.tar.tbb
   133921751    28.8       2.65    2002.43   lizard 39                        mingw-w64_x86_64-6.3.0-posix-seh-rt_v5-rev2.tar.tbb
   134410122    28.9     236.84     340.90   brotli 1                         mingw-w64_x86_64-6.3.0-posix-seh-rt_v5-rev2.tar.tbb
   136809466    29.4     137.78     562.34   libdeflate 1                     mingw-w64_x86_64-6.3.0-posix-seh-rt_v5-rev2.tar.tbb
   143004429    30.7       4.26    2651.89   oodle 49                         mingw-w64_x86_64-6.3.0-posix-seh-rt_v5-rev2.tar.tbb
   143562842    30.8       1.32    3774.23   lzturbo 19                       mingw-w64_x86_64-6.3.0-posix-seh-rt_v5-rev2.tar.tbb
   144311136    31.0       2.97    3179.57   lizard 19                        mingw-w64_x86_64-6.3.0-posix-seh-rt_v5-rev2.tar.tbb
   152900626    32.8      84.32    3488.06   lzturbo 12                       mingw-w64_x86_64-6.3.0-posix-seh-rt_v5-rev2.tar.tbb
   183278062    39.4     278.27     291.55   density 3                        mingw-w64_x86_64-6.3.0-posix-seh-rt_v5-rev2.tar.tbb
   302031406    64.9    1592.24    2319.67   chameleon 2                      mingw-w64_x86_64-6.3.0-posix-seh-rt_v5-rev2.tar.tbb
   409060060    87.9     232.49    3342.10   trle                             mingw-w64_x86_64-6.3.0-posix-seh-rt_v5-rev2.tar.tbb

Failed:

           0     0.0       0.00       0.00   lzsse2 1         mingw-w64_x86_64-6.3.0-posix-seh-rt_v5-rev2.tar
ERROR at 0:6d, 88
           0     0.0       0.00       0.00   lzsse2 17        mingw-w64_x86_64-6.3.0-posix-seh-rt_v5-rev2.tar
ERROR at 0:6d, 88
   121181325    26.0     106.03    6955.01   xpack 1          mingw-w64_x86_64-6.3.0-posix-seh-rt_v5-rev2.tar
ERROR at 32465155:f1, e
    57070696    12.3       0.82   28854.82   xpack 9          mingw-w64_x86_64-6.3.0-posix-seh-rt_v5-rev2.tar
ERROR at 12914960:ca, 35

The standout is oodle 118, it amazes having no counterpart/rival in the roster, Oodle 'Selkie' excels at binary-ish data!

Next is www.holybooks.com_70_PDFs.tar - a 195,587,584 bytes collection of 70 Yoga ebooks non-scanned:

yogauntitled-1

Not at all interested in binary decompression, yet, it is interesting to see how poorly some decompressors behave.

C:\Ryuugan_vs_lzbench_vs_TurboBench_(LzTurbo-OFFICIAL_vs_Zstd_vs_Oodle)_2018-Apr-22>"turbobench_v18.04_-_build_22_Apr_2018.exe" www.holybooks.com_70_PDFs.tar -ebzip2/lzlib,9d29fb273/lzham,4fb258:x4:d29/lzma,9d29:fb273:mf=bt4/libdeflate,1,5,12/oodle,19,49,89,112,114,116,118,129/lzsse2,1,17/lzturbo,19,12,29,22,39,32,49,59,59t2,59t4/zstd,1,5,12,22/lizard,19,29,39,49/brotli,1,5,11/brotli,11d29/lzma,9/xpack,1,9/chameleon,2/density,3/lzham,4/trle/bsc,3,6/zpaq,2,5 -g -I3 -J31 -k1 -B2G
...
TurboBench:  - Tue Apr 24 19:52:22 2018
      C Size  ratio%     C MB/s     D MB/s   Name            File
   153693341    78.6       0.27     146.56   brotli 11d29                     www.holybooks.com_70_PDFs.tar.tbb
   154043243    78.8       1.52      21.96   lzturbo 49                       www.holybooks.com_70_PDFs.tar.tbb
   154390722    78.9       1.99    3312.57   lzturbo 39                       www.holybooks.com_70_PDFs.tar.tbb
   154478896    79.0       0.39     178.47   lzham 4fb258:x4:d29              www.holybooks.com_70_PDFs.tar.tbb
   154629187    79.1       1.07    3723.21   oodle 89                         www.holybooks.com_70_PDFs.tar.tbb
   154632800    79.1       0.98    3730.17   oodle 129                        www.holybooks.com_70_PDFs.tar.tbb
   154657312    79.1       0.75     632.43   oodle 19                         www.holybooks.com_70_PDFs.tar.tbb
   154716487    79.1       1.95     180.24   lzham 4                          www.holybooks.com_70_PDFs.tar.tbb
   154955814    79.2       1.79      16.45   lzlib 9d29fb273                  www.holybooks.com_70_PDFs.tar.tbb
   155014084    79.3       2.40      21.53   lzma 9d29:fb273:mf=bt4           www.holybooks.com_70_PDFs.tar.tbb
   155466880    79.5      42.68    4247.66   lzturbo 32                       www.holybooks.com_70_PDFs.tar.tbb
   155773776    79.6       2.30    5107.39   lzturbo 29                       www.holybooks.com_70_PDFs.tar.tbb
   156341293    79.9       0.28       0.27   zpaq 5                           www.holybooks.com_70_PDFs.tar.tbb
   156627734    80.1       1.13    6609.03   oodle 118                        www.holybooks.com_70_PDFs.tar.tbb
   157307263    80.4       2.66    2646.87   zstd 22                          www.holybooks.com_70_PDFs.tar.tbb
   157485342    80.5       1.03    6859.59   oodle 116                        www.holybooks.com_70_PDFs.tar.tbb
   157684598    80.6       2.51      21.18   lzma 9                           www.holybooks.com_70_PDFs.tar.tbb
   158949343    81.3      17.09    7019.11   oodle 114                        www.holybooks.com_70_PDFs.tar.tbb
   160104128    81.9      45.79    5453.74   lzturbo 22                       www.holybooks.com_70_PDFs.tar.tbb
   160367431    82.0       0.32     164.61   brotli 11                        www.holybooks.com_70_PDFs.tar.tbb
   160410795    82.0       2.41    5069.27   lizard 49                        www.holybooks.com_70_PDFs.tar.tbb
   160607455    82.1       2.47    6037.59   lizard 29                        www.holybooks.com_70_PDFs.tar.tbb
   160871714    82.3       4.19      56.25   zpaq 2                           www.holybooks.com_70_PDFs.tar.tbb
   162059305    82.9      39.67    4093.67   zstd 12                          www.holybooks.com_70_PDFs.tar.tbb
   162401859    83.0      45.69     379.02   brotli 5                         www.holybooks.com_70_PDFs.tar.tbb
   163125766    83.4       3.18       4.63   bsc 6                            www.holybooks.com_70_PDFs.tar.tbb
   163805148    83.8      56.52    6788.64   oodle 112                        www.holybooks.com_70_PDFs.tar.tbb
   165996410    84.9       6.82       3.66   bsc 3                            www.holybooks.com_70_PDFs.tar.tbb
   166500697    85.1       4.58      11.54   lzturbo 59                       www.holybooks.com_70_PDFs.tar.tbb
   166508493    85.1       5.52      19.45   lzturbo 59t2                     www.holybooks.com_70_PDFs.tar.tbb
   166524061    85.1       5.76      25.66   lzturbo 59t4                     www.holybooks.com_70_PDFs.tar.tbb
   171171401    87.5     132.51    3985.56   zstd 5                           www.holybooks.com_70_PDFs.tar.tbb
   172232600    88.1      55.05     474.87   xpack 1                          www.holybooks.com_70_PDFs.tar.tbb
   172591996    88.2      18.12     434.06   libdeflate 12                    www.holybooks.com_70_PDFs.tar.tbb
   172983427    88.4       7.61      14.78   bzip2                            www.holybooks.com_70_PDFs.tar.tbb
   173165757    88.5      75.93     446.40   libdeflate 5                     www.holybooks.com_70_PDFs.tar.tbb
   173517815    88.7     682.58    4449.82   zstd 1                           www.holybooks.com_70_PDFs.tar.tbb
   173598021    88.8     424.30     519.73   brotli 1                         www.holybooks.com_70_PDFs.tar.tbb
   173651914    88.8       2.34    6857.19   lzturbo 19                       www.holybooks.com_70_PDFs.tar.tbb
   173668355    88.8       4.47    6438.04   oodle 49                         www.holybooks.com_70_PDFs.tar.tbb
   173728773    88.8      79.76     444.37   libdeflate 1                     www.holybooks.com_70_PDFs.tar.tbb
   174113040    89.0     117.40    7015.84   lzturbo 12                       www.holybooks.com_70_PDFs.tar.tbb
   174132004    89.0      12.05    6063.23   lizard 39                        www.holybooks.com_70_PDFs.tar.tbb
   174534130    89.2      12.82    6625.37   lizard 19                        www.holybooks.com_70_PDFs.tar.tbb
   180588056    92.3     429.56     371.83   density 3                        www.holybooks.com_70_PDFs.tar.tbb
   190260978    97.3    1511.72    1946.90   chameleon 2                      www.holybooks.com_70_PDFs.tar.tbb
   195189562    99.8     200.72    5894.92   trle                             www.holybooks.com_70_PDFs.tar.tbb

Failed:

           0     0.0       0.00       0.00   lzsse2 1         www.holybooks.com_70_PDFs.tar
ERROR at 0:77, 97
           0     0.0       0.00       0.00   lzsse2 17        www.holybooks.com_70_PDFs.tar
ERROR at 0:77, 97
   158720272    81.2       0.06    2150.35   xpack 9          www.holybooks.com_70_PDFs.tar
ERROR at 41743301:51, ae

Having downloaded RC_2008-08 (from https://files.pushshift.io/reddit/comments/), which is 346,626,502 bytes, next to be seen is an .JSON dump of reddit...

reddit-snoo-vinyl-toy-11-1422482500-233254815383 y7ocwxpkqp2oll2dlqhp

C:\Ryuugan_vs_lzbench_vs_TurboBench_(LzTurbo-OFFICIAL_vs_Zstd_vs_Oodle)_2018-Apr-22>"turbobench_v18.04_-_build_22_Apr_2018.exe" RC_2008-08 -ebzip2/lzlib,9d29fb273/lzham,4fb258:x4:d29/lzma,9d29:fb273:mf=bt4/libdeflate,1,5,12/oodle,19,49,89,112,114,116,118,129/lzsse2,1,17/lzturbo,19,12,29,22,39,32,49,59,59t2,59t4/zstd,1,5,12,22/lizard,19,29,39,49/brotli,1,5,11/brotli,11d29/lzma,9/xpack,1,9/chameleon,2/density,3/lzham,4/trle/bsc,3,6/zpaq,2,5 -g -I3 -J31 -k1 -B2G
...
TurboBench:  - Wed Apr 25 15:06:17 2018
      C Size  ratio%     C MB/s     D MB/s   Name            File
    36787350    10.6       0.39       0.39   zpaq 5                           RC_2008-08.tbb
    37963185    11.0       7.17      39.21   lzturbo 59                       RC_2008-08.tbb
    37964885    11.0       7.37      53.85   lzturbo 59t2                     RC_2008-08.tbb
    37968465    11.0       7.43      56.65   lzturbo 59t4                     RC_2008-08.tbb
    39984248    11.5      24.07      10.28   bsc 6                            RC_2008-08.tbb
    47496850    13.7       0.64     122.69   lzturbo 49                       RC_2008-08.tbb
    48563712    14.0       0.83      83.42   lzlib 9d29fb273                  RC_2008-08.tbb
    48680881    14.0       0.94     119.17   lzma 9d29:fb273:mf=bt4           RC_2008-08.tbb
    48791397    14.1       0.23     279.88   lzham 4fb258:x4:d29              RC_2008-08.tbb
    48849144    14.1       0.36     306.14   brotli 11d29                     RC_2008-08.tbb
    49536033    14.3       0.75     850.54   lzturbo 39                       RC_2008-08.tbb
    50365139    14.5       0.88     269.48   lzham 4                          RC_2008-08.tbb
    50900116    14.7       1.17     780.13   zstd 22                          RC_2008-08.tbb
    51154124    14.8       1.49     115.53   lzma 9                           RC_2008-08.tbb
    51449617    14.8       8.33      30.98   bzip2                            RC_2008-08.tbb
    52767834    15.2       0.38    1037.65   oodle 129                        RC_2008-08.tbb
    52767834    15.2       0.42     724.03   oodle 89                         RC_2008-08.tbb
    52887047    15.3       0.23     453.05   oodle 19                         RC_2008-08.tbb
    53273852    15.4      30.60      17.71   bsc 3                            RC_2008-08.tbb
    53574727    15.5       0.41     468.17   brotli 11                        RC_2008-08.tbb
    57126304    16.5       0.79    1228.27   lzturbo 29                       RC_2008-08.tbb
    61220390    17.7       0.82     551.73   xpack 9                          RC_2008-08.tbb
    63310638    18.3      12.90     977.83   zstd 12                          RC_2008-08.tbb
    63637661    18.4       1.11    1031.78   lizard 49                        RC_2008-08.tbb
    65042080    18.8       4.25      93.47   zpaq 2                           RC_2008-08.tbb
    65149686    18.8      46.29     856.93   lzturbo 32                       RC_2008-08.tbb
    65663240    18.9      29.89     477.78   brotli 5                         RC_2008-08.tbb
    73841496    21.3       3.89     999.76   libdeflate 12                    RC_2008-08.tbb
    74160950    21.4       1.17    1568.09   lizard 29                        RC_2008-08.tbb
    76425289    22.0     119.94     812.81   zstd 5                           RC_2008-08.tbb
    78467692    22.6       4.11    2010.38   lizard 39                        RC_2008-08.tbb
    80133746    23.1       0.36    2242.20   oodle 116                        RC_2008-08.tbb
    81118226    23.4       0.45    2420.88   oodle 118                        RC_2008-08.tbb
    81229949    23.4      94.18    1036.65   libdeflate 5                     RC_2008-08.tbb
    84425756    24.4      46.88    1304.51   lzturbo 22                       RC_2008-08.tbb
    85886667    24.8     312.01     903.79   zstd 1                           RC_2008-08.tbb
    86514711    25.0       0.90    2983.14   lzturbo 19                       RC_2008-08.tbb
    86580199    25.0       3.60    2728.08   oodle 49                         RC_2008-08.tbb
    86611403    25.0       4.44    2430.01   lizard 19                        RC_2008-08.tbb
    88367497    25.5     182.98     389.72   brotli 1                         RC_2008-08.tbb
    93560292    27.0      30.50    3386.94   oodle 114                        RC_2008-08.tbb
    95371965    27.5     104.87     658.61   xpack 1                          RC_2008-08.tbb
    96962015    28.0     149.05    1035.09   libdeflate 1                     RC_2008-08.tbb
   100452734    29.0     280.42     313.01   density 3                        RC_2008-08.tbb
   101171406    29.2      68.38    2716.47   lzturbo 12                       RC_2008-08.tbb
   118301287    34.1     107.15    3361.65   oodle 112                        RC_2008-08.tbb
   196844768    56.8    1631.01    3022.66   chameleon 2                      RC_2008-08.tbb
   346296552    99.9     189.96    1982.25   trle                             RC_2008-08.tbb

Failed:

           0     0.0       0.00       0.00   lzsse2 1         RC_2008-08
ERROR at 0:7b, 90
           0     0.0       0.00       0.00   lzsse2 17        RC_2008-08
ERROR at 0:7b, 90

One .log type file:

nasaflat 800x800 075 f u4

C:\Ryuugan_vs_lzbench_vs_TurboBench_(LzTurbo-OFFICIAL_vs_Zstd_vs_Oodle)_2018-Apr-22>"turbobench_v18.04_-_build_22_Apr_2018.exe" NASA_Kennedy_Space_Center_WWW_server_access_log_Jul95 -ebzip2/lzlib,9d29fb273/lzham,4fb258:x4:d29/lzma,9d29:fb273:mf=bt4/libdeflate,1,5,12/oodle,19,49,89,112,114,116,118,129/lzsse2,1,17/lzturbo,19,12,29,22,39,32,49,59,59t2,59t4/zstd,1,5,12,22/lizard,19,29,39,49/brotli,1,5,11/brotli,11d29/lzma,9/xpack,1,9/chameleon,2/density,3/lzham,4/trle/bsc,3,6/zpaq,2,5 -g -I3 -J31 -k1 -B2G
...
TurboBench:  - Wed Apr 25 20:06:28 2018
      C Size  ratio%     C MB/s     D MB/s   Name            File
     6875621     3.4       0.42       0.42   zpaq 5                           NASA_Kennedy_Space_Center_WWW_server_access_log_Jul95.tbb
     9628481     4.7       8.95      49.80   lzturbo 59                       NASA_Kennedy_Space_Center_WWW_server_access_log_Jul95.tbb
     9628909     4.7       9.03      63.96   lzturbo 59t2                     NASA_Kennedy_Space_Center_WWW_server_access_log_Jul95.tbb
     9630257     4.7       9.15      72.74   lzturbo 59t4                     NASA_Kennedy_Space_Center_WWW_server_access_log_Jul95.tbb
     9953030     4.8      50.27      19.03   bsc 6                            NASA_Kennedy_Space_Center_WWW_server_access_log_Jul95.tbb
    11355945     5.5       0.74     296.39   lzturbo 49                       NASA_Kennedy_Space_Center_WWW_server_access_log_Jul95.tbb
    11832205     5.8       0.89     146.77   lzlib 9d29fb273                  NASA_Kennedy_Space_Center_WWW_server_access_log_Jul95.tbb
    11883945     5.8       1.06     251.55   lzma 9d29:fb273:mf=bt4           NASA_Kennedy_Space_Center_WWW_server_access_log_Jul95.tbb
    11901404     5.8       0.88    1814.02   lzturbo 39                       NASA_Kennedy_Space_Center_WWW_server_access_log_Jul95.tbb
    11922873     5.8       0.22     701.10   lzham 4fb258:x4:d29              NASA_Kennedy_Space_Center_WWW_server_access_log_Jul95.tbb
    11960483     5.8       8.48      43.68   bzip2                            NASA_Kennedy_Space_Center_WWW_server_access_log_Jul95.tbb
    12236072     6.0       0.40     566.41   brotli 11d29                     NASA_Kennedy_Space_Center_WWW_server_access_log_Jul95.tbb
    12252608     6.0      53.58      26.84   bsc 3                            NASA_Kennedy_Space_Center_WWW_server_access_log_Jul95.tbb
    12639080     6.2       1.13    1385.37   zstd 22                          NASA_Kennedy_Space_Center_WWW_server_access_log_Jul95.tbb
    12711281     6.2       0.46     914.72   brotli 11                        NASA_Kennedy_Space_Center_WWW_server_access_log_Jul95.tbb
    12820437     6.2       0.31    1114.88   oodle 19                         NASA_Kennedy_Space_Center_WWW_server_access_log_Jul95.tbb
    13598062     6.6       2.03     230.21   lzma 9                           NASA_Kennedy_Space_Center_WWW_server_access_log_Jul95.tbb
    13651218     6.7       0.95     586.45   lzham 4                          NASA_Kennedy_Space_Center_WWW_server_access_log_Jul95.tbb
    14601890     7.1       0.69    1122.20   oodle 89                         NASA_Kennedy_Space_Center_WWW_server_access_log_Jul95.tbb
    14604989     7.1       0.57    1125.80   oodle 129                        NASA_Kennedy_Space_Center_WWW_server_access_log_Jul95.tbb
    15064566     7.3       0.97    2465.97   lzturbo 29                       NASA_Kennedy_Space_Center_WWW_server_access_log_Jul95.tbb
    15185234     7.4       2.78    1336.67   xpack 9                          NASA_Kennedy_Space_Center_WWW_server_access_log_Jul95.tbb
    15884624     7.7       1.25    2224.49   lizard 49                        NASA_Kennedy_Space_Center_WWW_server_access_log_Jul95.tbb
    16164083     7.9      35.19    1641.20   zstd 12                          NASA_Kennedy_Space_Center_WWW_server_access_log_Jul95.tbb
    16665926     8.1      64.65    1022.66   brotli 5                         NASA_Kennedy_Space_Center_WWW_server_access_log_Jul95.tbb
    17344456     8.5       6.65     125.95   zpaq 2                           NASA_Kennedy_Space_Center_WWW_server_access_log_Jul95.tbb
    18280070     8.9     106.05    1767.49   lzturbo 32                       NASA_Kennedy_Space_Center_WWW_server_access_log_Jul95.tbb
    18473887     9.0       3.35    1324.87   libdeflate 12                    NASA_Kennedy_Space_Center_WWW_server_access_log_Jul95.tbb
    19226560     9.4       1.30    2864.47   lizard 29                        NASA_Kennedy_Space_Center_WWW_server_access_log_Jul95.tbb
    20125477     9.8     223.77    1184.60   zstd 5                           NASA_Kennedy_Space_Center_WWW_server_access_log_Jul95.tbb
    21399767    10.4       3.97    2639.37   lizard 39                        NASA_Kennedy_Space_Center_WWW_server_access_log_Jul95.tbb
    21693167    10.6       0.70    2648.22   oodle 118                        NASA_Kennedy_Space_Center_WWW_server_access_log_Jul95.tbb
    21714886    10.6       0.53    2850.19   oodle 116                        NASA_Kennedy_Space_Center_WWW_server_access_log_Jul95.tbb
    21836877    10.6     173.14    1388.45   libdeflate 5                     NASA_Kennedy_Space_Center_WWW_server_access_log_Jul95.tbb
    22756147    11.1     523.95    1174.83   zstd 1                           NASA_Kennedy_Space_Center_WWW_server_access_log_Jul95.tbb
    24331363    11.9       1.11    4033.61   lzturbo 19                       NASA_Kennedy_Space_Center_WWW_server_access_log_Jul95.tbb
    24368285    11.9       4.00    3183.38   oodle 49                         NASA_Kennedy_Space_Center_WWW_server_access_log_Jul95.tbb
    24408147    11.9       4.37    3488.86   lizard 19                        NASA_Kennedy_Space_Center_WWW_server_access_log_Jul95.tbb
    24713548    12.0     173.90    1008.09   xpack 1                          NASA_Kennedy_Space_Center_WWW_server_access_log_Jul95.tbb
    25275862    12.3     373.79     812.27   brotli 1                         NASA_Kennedy_Space_Center_WWW_server_access_log_Jul95.tbb
    26164436    12.7     112.66    2462.71   lzturbo 22                       NASA_Kennedy_Space_Center_WWW_server_access_log_Jul95.tbb
    27183252    13.2     229.98    1179.78   libdeflate 1                     NASA_Kennedy_Space_Center_WWW_server_access_log_Jul95.tbb
    29731553    14.5      73.97    3901.65   oodle 114                        NASA_Kennedy_Space_Center_WWW_server_access_log_Jul95.tbb
    30227172    14.7     417.58     400.41   density 3                        NASA_Kennedy_Space_Center_WWW_server_access_log_Jul95.tbb
    31188794    15.2     163.41    3435.94   lzturbo 12                       NASA_Kennedy_Space_Center_WWW_server_access_log_Jul95.tbb
    36226032    17.7     262.57    3282.72   oodle 112                        NASA_Kennedy_Space_Center_WWW_server_access_log_Jul95.tbb
   111357412    54.3    1669.41    3680.49   chameleon 2                      NASA_Kennedy_Space_Center_WWW_server_access_log_Jul95.tbb
   205062892    99.9     184.03    1983.13   trle                             NASA_Kennedy_Space_Center_WWW_server_access_log_Jul95.tbb

Failed:

           0     0.0       0.00       0.00   lzsse2 1         NASA_Kennedy_Space_Center_WWW_server_access_log_Jul95
ERROR at 0:31, c8
           0     0.0       0.00       0.00   lzsse2 17        NASA_Kennedy_Space_Center_WWW_server_access_log_Jul95
ERROR at 0:31, c8

The idea is to cover at least these formats:

.TXT (Plain ASCII English Text)
.DSL
.XML
.JSON
.PDF
.TAR (executables, libraries, sources)

It is a good idea to juxtapose two ~300MB .XML dumps...

Next one is gonna be the .XML dump of "Wikipedia's evil twin", according to the article: https://en.wikipedia.org/wiki/Encyclopedia_Dramatica

Crunching 295,515,893 bytes long encyclopediadramaticase-20150628-current.xml ...

TurboBench: - Thu Apr 26 11:44:32 2018

(bold = pareto) MB=1.000.000

C Size	ratio	C MB/s	D MB/s	Name
42526637	14.4	8.16	37.84	lzturbo 59
42529461	14.4	8.51	50.85	lzturbo 59t2
42534181	14.4	8.64	58.31	lzturbo 59t4
42763459	14.5	0.38	0.38	zpaq 5
45296512	15.3	24.94	9.23	bsc 6
48331622	16.4	0.61	103.39	lzturbo 49
48665325	16.5	0.92	74.37	lzlib 9d29fb273
48768664	16.5	0.99	103.98	lzma 9d29:fb273:mf=bt4
49285547	16.7	0.33	270.88	brotli 11d29
49356398	16.7	0.20	242.04	lzham 4fb258:x4:d29
50392272	17.1	0.81	770.60	lzturbo 39
50394670	17.1	0.93	238.38	lzham 4
51225518	17.3	1.34	100.09	lzma 9
51712719	17.5	1.22	728.34	zstd 22
52145611	17.6	0.44	677.25	oodle 89
52145724	17.6	0.39	676.07	oodle 129
53752244	18.2	0.25	390.78	oodle 19
55929229	18.9	0.37	390.60	brotli 11
56276672	19.0	28.38	16.80	bsc 3
58523735	19.8	0.86	1139.22	lzturbo 29
59832988	20.2	8.60	29.57	bzip2
60558802	20.5	40.19	771.73	lzturbo 32
62343390	21.1	0.64	499.89	xpack 9
64984048	22.0	10.58	865.71	zstd 12
66447845	22.5	25.35	409.12	brotli 5
66764312	22.6	1.16	966.61	lizard 49
68204206	23.1	4.38	83.95	zpaq 2
73098453	24.7	104.00	836.44	zstd 5
75448396	25.5	5.58	643.28	libdeflate 12
78892029	26.7	1.22	1755.37	lizard 29
79571370	26.9	39.15	1430.72	lzturbo 22
79697531	27.0	69.43	654.61	libdeflate 5
80920761	27.4	3.56	2063.53	lizard 39
81086153	27.4	0.40	2649.13	oodle 116
81908602	27.7	0.47	2657.88	oodle 118
83472503	28.2	175.59	363.39	brotli 1
84696417	28.7	106.19	694.04	xpack 1
86085983	29.1	316.32	977.61	zstd 1
86923697	29.4	110.45	647.82	libdeflate 1
89494542	30.3	26.54	3241.69	oodle 114
90013089	30.5	3.96	2946.61	lizard 19
90047833	30.5	0.97	3515.12	lzturbo 19
90100574	30.5	3.65	2628.28	oodle 49
96748726	32.7	62.50	3527.16	lzturbo 12
104639536	35.4	91.49	2949.38	oodle 112
109982235	37.2	282.19	590.40	density 3
175697176	59.5	1558.09	2218.82	chameleon 2
280443803	94.9	195.96	2045.29	trle

Having finished the main run, above table was done by running:

C:\>turbobench_v18.04_-_build_16_Apr_2018.exe -p4 encyclopediadramaticase-20150628-current.xml.tbb

Still prefer the RAW output, however the Pareto Frontier performers are not in bold:

C:\>"turbobench_v18.04_-_build_22_Apr_2018.exe" encyclopediadramaticase-20150628-current.xml -ebzip2/lzlib,9d29fb273/lzham,4fb258:x4:d29/lzma,9d29:fb273:mf=bt4/libdeflate,1,5,12/oodle,19,49,89,112,114,116,118,129/lzsse2,1,17/lzturbo,19,12,29,22,39,32,49,59,59t2,59t4/zstd,1,5,12,22/lizard,19,29,39,49/brotli,1,5,11/brotli,11d29/lzma,9/xpack,1,9/chameleon,2/density,3/lzham,4/trle/bsc,3,6/zpaq,2,5 -I3 -J31 -k1 -B2G
...
TurboBench:  - Thu Apr 26 02:09:14 2018
      C Size  ratio%     C MB/s     D MB/s   Name            File
    42526637    14.4       8.16      37.84   lzturbo 59                       encyclopediadramaticase-20150628-current.xml
    42529461    14.4       8.51      50.85   lzturbo 59t2                     encyclopediadramaticase-20150628-current.xml
    42534181    14.4       8.64      58.31   lzturbo 59t4                     encyclopediadramaticase-20150628-current.xml
    42763459    14.5       0.38       0.38   zpaq 5                           encyclopediadramaticase-20150628-current.xml
    45296512    15.3      24.94       9.23   bsc 6                            encyclopediadramaticase-20150628-current.xml
    48331622    16.4       0.61     103.39   lzturbo 49                       encyclopediadramaticase-20150628-current.xml
    48665325    16.5       0.92      74.37   lzlib 9d29fb273                  encyclopediadramaticase-20150628-current.xml
    48768664    16.5       0.99     103.98   lzma 9d29:fb273:mf=bt4           encyclopediadramaticase-20150628-current.xml
    49285547    16.7       0.33     270.88   brotli 11d29                     encyclopediadramaticase-20150628-current.xml
    49356398    16.7       0.20     242.04   lzham 4fb258:x4:d29              encyclopediadramaticase-20150628-current.xml
    50392272    17.1       0.81     770.60   lzturbo 39                       encyclopediadramaticase-20150628-current.xml
    50394670    17.1       0.93     238.38   lzham 4                          encyclopediadramaticase-20150628-current.xml
    51225518    17.3       1.34     100.09   lzma 9                           encyclopediadramaticase-20150628-current.xml
    51712719    17.5       1.22     728.34   zstd 22                          encyclopediadramaticase-20150628-current.xml
    52145611    17.6       0.44     677.25   oodle 89                         encyclopediadramaticase-20150628-current.xml
    52145724    17.6       0.39     676.07   oodle 129                        encyclopediadramaticase-20150628-current.xml
    53752244    18.2       0.25     390.78   oodle 19                         encyclopediadramaticase-20150628-current.xml
    55929229    18.9       0.37     390.60   brotli 11                        encyclopediadramaticase-20150628-current.xml
    56276672    19.0      28.38      16.80   bsc 3                            encyclopediadramaticase-20150628-current.xml
    58523735    19.8       0.86    1139.22   lzturbo 29                       encyclopediadramaticase-20150628-current.xml
    59832988    20.2       8.60      29.57   bzip2                            encyclopediadramaticase-20150628-current.xml
    60558802    20.5      40.19     771.73   lzturbo 32                       encyclopediadramaticase-20150628-current.xml
    62343390    21.1       0.64     499.89   xpack 9                          encyclopediadramaticase-20150628-current.xml
    64984048    22.0      10.58     865.71   zstd 12                          encyclopediadramaticase-20150628-current.xml
    66447845    22.5      25.35     409.12   brotli 5                         encyclopediadramaticase-20150628-current.xml
    66764312    22.6       1.16     966.61   lizard 49                        encyclopediadramaticase-20150628-current.xml
    68204206    23.1       4.38      83.95   zpaq 2                           encyclopediadramaticase-20150628-current.xml
    73098453    24.7     104.00     836.44   zstd 5                           encyclopediadramaticase-20150628-current.xml
    75448396    25.5       5.58     643.28   libdeflate 12                    encyclopediadramaticase-20150628-current.xml
    78892029    26.7       1.22    1755.37   lizard 29                        encyclopediadramaticase-20150628-current.xml
    79571370    26.9      39.15    1430.72   lzturbo 22                       encyclopediadramaticase-20150628-current.xml
    79697531    27.0      69.43     654.61   libdeflate 5                     encyclopediadramaticase-20150628-current.xml
    80920761    27.4       3.56    2063.53   lizard 39                        encyclopediadramaticase-20150628-current.xml
    81086153    27.4       0.40    2649.13   oodle 116                        encyclopediadramaticase-20150628-current.xml
    81908602    27.7       0.47    2657.88   oodle 118                        encyclopediadramaticase-20150628-current.xml
    83472503    28.2     175.59     363.39   brotli 1                         encyclopediadramaticase-20150628-current.xml
    84696417    28.7     106.19     694.04   xpack 1                          encyclopediadramaticase-20150628-current.xml
    86085983    29.1     316.32     977.61   zstd 1                           encyclopediadramaticase-20150628-current.xml
    86923697    29.4     110.45     647.82   libdeflate 1                     encyclopediadramaticase-20150628-current.xml
    89494542    30.3      26.54    3241.69   oodle 114                        encyclopediadramaticase-20150628-current.xml
    90013089    30.5       3.96    2946.61   lizard 19                        encyclopediadramaticase-20150628-current.xml
    90047833    30.5       0.97    3515.12   lzturbo 19                       encyclopediadramaticase-20150628-current.xml
    90100574    30.5       3.65    2628.28   oodle 49                         encyclopediadramaticase-20150628-current.xml
    96748726    32.7      62.50    3527.16   lzturbo 12                       encyclopediadramaticase-20150628-current.xml
   104639536    35.4      91.49    2949.38   oodle 112                        encyclopediadramaticase-20150628-current.xml
   109982235    37.2     282.19     590.40   density 3                        encyclopediadramaticase-20150628-current.xml
   175697176    59.5    1558.09    2218.82   chameleon 2                      encyclopediadramaticase-20150628-current.xml
   280443803    94.9     195.96    2045.29   trle                             encyclopediadramaticase-20150628-current.xml

Failed:

           0     0.0       0.00       0.00   lzsse2 1         encyclopediadramaticase-20150628-current.xml
ERROR at 0:3c, 96
           0     0.0       0.00       0.00   lzsse2 17        encyclopediadramaticase-20150628-current.xml
ERROR at 0:3c, 96

Another .XML dump is coming, in order to scramble the biased enwik8 notions, the first 300MB of 60,182,193,037 bytes long enwiki-20170101-pages-articles.xml will "promote" the stronger ones/performers... RIGHTFULLY SO, those who overtuned their codecs to ENWIK8 could also witness the outcome when 200 extra kilometers are to be trodden :P

enwiki2000px-wikipedia-logo-v2-en svg

TurboBench: - Thu Apr 26 23:11:13 2018

(bold = pareto) MB=1.000.000

C Size	ratio	C MB/s	D MB/s	Name
62419065	19.8	6.89	31.64	lzturbo 59
62422029	19.8	7.20	45.47	lzturbo 59t2
62427257	19.8	7.35	51.07	lzturbo 59t4
62517772	19.9	0.36	0.36	zpaq 5
65779018	20.9	18.17	6.53	bsc 6
70989698	22.6	0.65	77.48	lzturbo 49
72099599	22.9	0.87	57.24	lzlib 9d29fb273
72205998	23.0	0.91	75.21	lzma 9d29:fb273:mf=bt4
72439347	23.0	0.33	215.37	brotli 11d29
72824487	23.2	0.30	176.87	lzham 4fb258:x4:d29
73048249	23.2	0.75	177.06	lzham 4
73736920	23.4	0.75	572.32	lzturbo 39
74558511	23.7	1.06	72.63	lzma 9
75554555	24.0	1.18	554.52	zstd 22
76107283	24.2	0.36	506.87	oodle 89
76107283	24.2	0.33	506.67	oodle 129
79003351	25.1	0.21	284.37	oodle 19
81912155	26.0	0.39	297.87	brotli 11
83638722	26.6	20.97	12.30	bsc 3
84326002	26.8	0.78	858.99	lzturbo 29
88091533	28.0	28.70	573.10	lzturbo 32
88186503	28.0	11.65	24.05	bzip2
91599453	29.1	0.45	363.90	xpack 9
93365278	29.7	8.05	939.19	zstd 12
95240121	30.3	19.73	625.24	brotli 5
95326645	30.3	1.49	747.10	lizard 49
97029330	30.8	3.45	71.38	zpaq 2
104138699	33.1	81.11	941.45	zstd 5
105403772	33.5	5.80	521.50	libdeflate 12
110214795	35.0	67.73	540.69	libdeflate 5
110688754	35.2	1.55	1170.56	lizard 29
112942068	35.9	6.28	1537.93	lizard 39
113673394	36.1	28.67	903.00	lzturbo 22
113925755	36.2	0.36	2209.71	oodle 116
115347740	36.7	0.40	2199.13	oodle 118
118052171	37.5	132.86	287.65	brotli 1
119288025	37.9	85.34	564.82	xpack 1
119708816	38.1	118.16	530.93	libdeflate 1
122191084	38.8	239.91	774.52	zstd 1
124642944	39.6	0.89	3092.78	lzturbo 19
124684991	39.6	3.21	2160.23	oodle 49
124723830	39.6	6.62	2505.48	lizard 19
125164938	39.8	19.97	3104.19	oodle 114
132235497	42.0	47.56	3084.77	lzturbo 12
146775844	46.7	173.95	272.87	density 3
146814788	46.7	68.45	2967.95	oodle 112
197027074	62.6	1544.23	1812.78	chameleon 2
312661033	99.4	188.27	2157.15	trle

Failed:

           0     0.0       0.00       0.00   lzsse2 1         enwiki-20170101-pages-articles.xml_first_300MB
ERROR at 0:3c, 96
           0     0.0       0.00       0.00   lzsse2 17        enwiki-20170101-pages-articles.xml_first_300MB
ERROR at 0:3c, 96

Oh, and I almost forgot the most interesting to me format - English .TXT tarred anthologies - reckon the Star Trek fans have to be allowed to see what decompressors to choose when the need of opening AT WARP SPEED (WARP is not just a word, you know, there was OS/2 on top of that) the ULTIMATE Collection Star Trek ebook arises ...

Crunching 325,071,872 bytes long StarTrek-_737_Ebooks.tar ...

startrek20140312-212254

The Markdown generated by TurboBench:

TurboBench: - Thu Apr 26 11:44:08 2018

(bold = pareto) MB=1.000.000

C Size	ratio	C MB/s	D MB/s	Name
59411401	18.3	6.47	31.32	lzturbo 59
59412829	18.3	6.66	45.71	lzturbo 59t2
59415117	18.3	6.85	50.84	lzturbo 59t4
63159822	19.4	0.37	0.36	zpaq 5
65359558	20.1	21.95	5.83	bsc 6
70822994	21.8	0.57	85.43	lzturbo 49
71865410	22.1	0.76	61.94	lzlib 9d29fb273
71943246	22.1	0.78	83.41	lzma 9d29:fb273:mf=bt4
71952589	22.1	0.34	232.59	brotli 11d29
72229583	22.2	0.25	186.70	lzham 4fb258:x4:d29
72311551	22.2	0.68	183.77	lzham 4
72346887	22.3	0.65	564.06	lzturbo 39
74998925	23.1	1.00	569.41	zstd 22
75251291	23.1	0.88	81.02	lzma 9
75312368	23.2	0.31	489.95	oodle 89
75312445	23.2	0.29	491.69	oodle 129
78374256	24.1	0.18	277.11	oodle 19
81393957	25.0	0.68	905.84	lzturbo 29
84113610	25.9	0.41	649.77	brotli 11
84321413	25.9	11.44	22.53	bzip2
85156578	26.2	21.93	14.16	bsc 3
89815371	27.6	29.29	574.73	lzturbo 32
92993537	28.6	1.25	974.66	lizard 49
94480420	29.1	0.50	422.00	xpack 9
97560652	30.0	5.19	697.70	zstd 12
100567346	30.9	3.55	70.40	zpaq 2
102104435	31.4	20.12	359.96	brotli 5
111798931	34.4	5.40	559.52	libdeflate 12
111947098	34.4	78.30	606.49	zstd 5
112249518	34.5	1.29	1294.11	lizard 29
113118398	34.8	5.94	1515.44	lizard 39
114912970	35.4	0.32	2262.46	oodle 116
117119332	36.0	0.34	2260.52	oodle 118
119465242	36.8	56.94	598.41	libdeflate 5
124577277	38.3	127.44	281.09	brotli 1
125409207	38.6	28.93	938.48	lzturbo 22
130464112	40.1	213.17	759.57	zstd 1
130569470	40.2	97.02	527.99	xpack 1
130652429	40.2	118.39	573.86	libdeflate 1
131564768	40.5	18.76	2848.23	oodle 114
132340112	40.7	0.74	3106.01	lzturbo 19
132391865	40.7	6.12	2539.96	lizard 19
132396180	40.7	3.01	2235.15	oodle 49
142275352	43.8	44.33	3153.25	lzturbo 12
148445114	45.7	337.76	639.95	density 3
169736713	52.2	87.01	2888.01	oodle 112
183444616	56.4	1519.26	2866.97	chameleon 2
324553919	99.8	191.19	2253.41	trle

The RAW:

C:\>"turbobench_v18.04_-_build_22_Apr_2018.exe" Star_Trek_-_737_Ebooks.tar -ebzip2/lzlib,9d29fb273/lzham,4fb258:x4:d29/lzma,9d29:fb273:mf=bt4/libdeflate,1,5,12/oodle,19,49,89,112,114,116,118,129/lzsse2,1,17/lzturbo,19,12,29,22,39,32,49,59,59t2,59t4/zstd,1,5,12,22/lizard,19,29,39,49/brotli,1,5,11/brotli,11d29/lzma,9/xpack,1,9/chameleon,2/density,3/lzham,4/trle/bsc,3,6/zpaq,2,5 -I3 -J31 -k1 -B2G
...
TurboBench:  - Thu Apr 26 10:13:34 2018
      C Size  ratio%     C MB/s     D MB/s   Name            File
    59411401    18.3       6.47      31.32   lzturbo 59                       Star_Trek_-_737_Ebooks.tar
    59412829    18.3       6.66      45.71   lzturbo 59t2                     Star_Trek_-_737_Ebooks.tar
    59415117    18.3       6.85      50.84   lzturbo 59t4                     Star_Trek_-_737_Ebooks.tar
    63159822    19.4       0.37       0.36   zpaq 5                           Star_Trek_-_737_Ebooks.tar
    65359558    20.1      21.95       5.83   bsc 6                            Star_Trek_-_737_Ebooks.tar
    70822994    21.8       0.57      85.43   lzturbo 49                       Star_Trek_-_737_Ebooks.tar
    71865410    22.1       0.76      61.94   lzlib 9d29fb273                  Star_Trek_-_737_Ebooks.tar
    71943246    22.1       0.78      83.41   lzma 9d29:fb273:mf=bt4           Star_Trek_-_737_Ebooks.tar
    71952589    22.1       0.34     232.59   brotli 11d29                     Star_Trek_-_737_Ebooks.tar
    72229583    22.2       0.25     186.70   lzham 4fb258:x4:d29              Star_Trek_-_737_Ebooks.tar
    72311551    22.2       0.68     183.77   lzham 4                          Star_Trek_-_737_Ebooks.tar
    72346887    22.3       0.65     564.06   lzturbo 39                       Star_Trek_-_737_Ebooks.tar
    74998925    23.1       1.00     569.41   zstd 22                          Star_Trek_-_737_Ebooks.tar
    75251291    23.1       0.88      81.02   lzma 9                           Star_Trek_-_737_Ebooks.tar
    75312368    23.2       0.31     489.95   oodle 89                         Star_Trek_-_737_Ebooks.tar
    75312445    23.2       0.29     491.69   oodle 129                        Star_Trek_-_737_Ebooks.tar
    78374256    24.1       0.18     277.11   oodle 19                         Star_Trek_-_737_Ebooks.tar
    81393957    25.0       0.68     905.84   lzturbo 29                       Star_Trek_-_737_Ebooks.tar
    84113610    25.9       0.41     649.77   brotli 11                        Star_Trek_-_737_Ebooks.tar
    84321413    25.9      11.44      22.53   bzip2                            Star_Trek_-_737_Ebooks.tar
    85156578    26.2      21.93      14.16   bsc 3                            Star_Trek_-_737_Ebooks.tar
    89815371    27.6      29.29     574.73   lzturbo 32                       Star_Trek_-_737_Ebooks.tar
    92993537    28.6       1.25     974.66   lizard 49                        Star_Trek_-_737_Ebooks.tar
    94480420    29.1       0.50     422.00   xpack 9                          Star_Trek_-_737_Ebooks.tar
    97560652    30.0       5.19     697.70   zstd 12                          Star_Trek_-_737_Ebooks.tar
   100567346    30.9       3.55      70.40   zpaq 2                           Star_Trek_-_737_Ebooks.tar
   102104435    31.4      20.12     359.96   brotli 5                         Star_Trek_-_737_Ebooks.tar
   111798931    34.4       5.40     559.52   libdeflate 12                    Star_Trek_-_737_Ebooks.tar
   111947098    34.4      78.30     606.49   zstd 5                           Star_Trek_-_737_Ebooks.tar
   112249518    34.5       1.29    1294.11   lizard 29                        Star_Trek_-_737_Ebooks.tar
   113118398    34.8       5.94    1515.44   lizard 39                        Star_Trek_-_737_Ebooks.tar
   114912970    35.4       0.32    2262.46   oodle 116                        Star_Trek_-_737_Ebooks.tar
   117119332    36.0       0.34    2260.52   oodle 118                        Star_Trek_-_737_Ebooks.tar
   119465242    36.8      56.94     598.41   libdeflate 5                     Star_Trek_-_737_Ebooks.tar
   124577277    38.3     127.44     281.09   brotli 1                         Star_Trek_-_737_Ebooks.tar
   125409207    38.6      28.93     938.48   lzturbo 22                       Star_Trek_-_737_Ebooks.tar
   130464112    40.1     213.17     759.57   zstd 1                           Star_Trek_-_737_Ebooks.tar
   130569470    40.2      97.02     527.99   xpack 1                          Star_Trek_-_737_Ebooks.tar
   130652429    40.2     118.39     573.86   libdeflate 1                     Star_Trek_-_737_Ebooks.tar
   131564768    40.5      18.76    2848.23   oodle 114                        Star_Trek_-_737_Ebooks.tar
   132340112    40.7       0.74    3106.01   lzturbo 19                       Star_Trek_-_737_Ebooks.tar
   132391865    40.7       6.12    2539.96   lizard 19                        Star_Trek_-_737_Ebooks.tar
   132396180    40.7       3.01    2235.15   oodle 49                         Star_Trek_-_737_Ebooks.tar
   142275352    43.8      44.33    3153.25   lzturbo 12                       Star_Trek_-_737_Ebooks.tar
   148445114    45.7     337.76     639.95   density 3                        Star_Trek_-_737_Ebooks.tar
   169736713    52.2      87.01    2888.01   oodle 112                        Star_Trek_-_737_Ebooks.tar
   183444616    56.4    1519.26    2866.97   chameleon 2                      Star_Trek_-_737_Ebooks.tar
   324553919    99.8     191.19    2253.41   trle                             Star_Trek_-_737_Ebooks.tar

Failed:

           0     0.0       0.00       0.00   lzsse2 1         Star_Trek_-_737_Ebooks.tar
ERROR at 0:53, a0
           0     0.0       0.00       0.00   lzsse2 17        Star_Trek_-_737_Ebooks.tar
ERROR at 0:53, a0

Okay, let us see how the results for the two 300MB .XML encyclopedia dumps (Wikipedia and Dramatica) relate:

    48768664    16.5       0.99     103.98   lzma 9d29:fb273:mf=bt4           encyclopediadramaticase-20150628-current.xml
    49285547    16.7       0.33     270.88   brotli 11d29                     encyclopediadramaticase-20150628-current.xml
    49356398    16.7       0.20     242.04   lzham 4fb258:x4:d29              encyclopediadramaticase-20150628-current.xml
    50392272    17.1       0.81     770.60   lzturbo 39                       encyclopediadramaticase-20150628-current.xml

    72205998    23.0       0.91      75.21   lzma 9d29:fb273:mf=bt4           enwiki-20170101-pages-articles.xml_first_300MB
    72439347    23.0       0.33     215.37   brotli 11d29                     enwiki-20170101-pages-articles.xml_first_300MB
    72824487    23.2       0.30     176.87   lzham 4fb258:x4:d29              enwiki-20170101-pages-articles.xml_first_300MB
    73736920    23.4       0.75     572.32   lzturbo 39                       enwiki-20170101-pages-articles.xml_first_300MB

    71943246    22.1       0.78      83.41   lzma 9d29:fb273:mf=bt4           Star_Trek_-_737_Ebooks.tar
    71952589    22.1       0.34     232.59   brotli 11d29                     Star_Trek_-_737_Ebooks.tar
    72229583    22.2       0.25     186.70   lzham 4fb258:x4:d29              Star_Trek_-_737_Ebooks.tar
    72346887    22.3       0.65     564.06   lzturbo 39                       Star_Trek_-_737_Ebooks.tar

Dramatica:

Brotli decompresses 270/103= 2.6x faster than LZMA LzTurbo decompresses 770/270= 2.8x faster than Brotli

Wikipedia:

Brotli decompresses 215/75= 2.8x faster than LZMA LzTurbo decompresses 572/215= 2.6x faster than Brotli

Star_Trek:

Brotli decompresses 232/83= 2.7x faster than LZMA LzTurbo decompresses 564/232= 2.4x faster than Brotli

So, three things:

Large Window Brotli proves very strong but is no match to LzTurbo;
XML results are not fully uniform, the [de]compression rates vary;
After finishing all the testsets, my intent is to provide a table (Markdown style) pointing out all the decompression rates speedups for Oodle 'Hydra', LzTurbo 39, Brotli and LZMA.

In my previous tests I witnessed difference between Oodle 129 and Oodle 89, AFAIK being respectively 'Hydra' and 'Kraken', the former being much faster on DNA data, also in Reddit test:

    52767834    15.2       0.38    1037.65   oodle 129                        RC_2008-08.tbb
    52767834    15.2       0.42     724.03   oodle 89                         RC_2008-08.tbb

So, the final outcome will be a single table showing how the 4 performers relate.

As the French say, "L'appétit vient en mangeant.": Idiomatic translation: The more you have, the more you want. Literal meaning: Appetite comes while eating.

Reckon, the .DNA and .CSV should also contribute to the drawing of the picture...

EDIT: It turns out, I overlooked the TurboBench ability to benchmark an arbitrary long chunk at beginning by using the option "-B" . Ex. "-B200" for 200.000.000 bytes. Only the first 200MB will be used in the benchmark.

Also, to feint the drawback (the insufficient 8GB on my testmachine 'Compressionette') wrote the console utility 'Chunkerito', can download its .C source and executable, Get_The_First_300MB_chunk.zip (38,741 bytes):

C:\Get_The_First_300MB_chunk>Get_The_First_300MB_chunk.bat C:\_KAZE\Textual_corpora\enwiki-20170101-pages-articles.xml
Chunkerito, revision 1+, written by Kaze.
Purpose: To chunkize/split any file to 'ChunkSize' long chunks.
Usage: Chunker filename ChunkSize
Note: For 128MB chunks use ChunkSize = 134217728
Size of Input TEXTual file: 60,182,193,037
^CTerminate batch job (Y/N)? y...

C:\Get_The_First_300MB_chunk>dir

04/26/2018  11:47 AM       314,572,800 Chunkerito.000,001
04/26/2018  11:47 AM       314,572,800 Chunkerito.000,002
04/26/2018  11:47 AM       314,572,800 Chunkerito.000,003
04/26/2018  11:47 AM       314,572,800 Chunkerito.000,004
03/17/2018  12:15 AM            12,895 Chunkerito.c
03/17/2018  12:15 AM            62,976 Chunkerito.exe
04/26/2018  11:38 AM                30 Get_The_First_300MB_chunk.bat
03/29/2018  12:49 AM             1,631 MokujIN GREEN 224 prompt.lnk

C:\Get_The_First_300MB_chunk>type "Chunkerito.000,001"|more
<mediawiki xmlns="http://www.mediawiki.org/xml/export-0.10/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.mediawiki.org/xml/export-0.10/ http://www.mediawiki.org/xml/export-0.10.xsd" v
ersion="0.10" xml:lang="en">
  <siteinfo>
    <sitename>Wikipedia</sitename>
    <dbname>enwiki</dbname>
    <base>https://en.wikipedia.org/wiki/Main_Page</base>
    <generator>MediaWiki 1.29.0-wmf.6</generator>
    <case>first-letter</case>
    <namespaces>
      <namespace key="-2" case="first-letter">Media</namespace>
      <namespace key="-1" case="first-letter">Special</namespace>
      <namespace key="0" case="first-letter" />
      <namespace key="1" case="first-letter">Talk</namespace>
      <namespace key="2" case="first-letter">User</namespace>
...

The same chunkenization done on the 'Dragonfly' DNA sequence ~960MB long:

C:\Get_The_First_300MB_chunk>Get_The_First_300MB_chunk.bat "c:\xx\DCC_(Deathship_Compression_Corpus)_644-files\www.ncbi.nlm.nih.gov_Dragonfly_(Ladona_fulva)_whole_genome_shotgun.tar"
Chunkerito, revision 1+, written by Kaze.
Purpose: To chunkize/split any file to 'ChunkSize' long chunks.
Usage: Chunker filename ChunkSize
Note: For 128MB chunks use ChunkSize = 134217728
Size of Input TEXTual file: 975,021,056
\; Chunk # 4 has been created ...

C:\Get_The_First_300MB_chunk>type "www.ncbi.nlm.nih.gov_Dragonfly_(Ladona_fulva)_whole_genome_shotgun.tar_first_300MB"|more
www.ncbi.nlm.nih.gov_Dragonfly_(Ladona_fulva)_whole_genome_shotgun/APVN01.1.fsa_nt
...
>gi|481454603|gb|APVN01000001.1| Ladona fulva Contig1, whole genome shotgun sequence
TTAGCCGTCAAGGGGATCACGATACGATTTATATTTGTTCATGTAATGAGCCGTGACACATATCGATCGA
TGTAACATGTTACTAAAATACACAATACGAATCGTAGAATAAATCAAAAAGTTAATATTTTGCTTCCGAA
TGAGAAACGTCGAATGCACAACGCTGAGTATTGGTAGTTGAAATATTTTATTTCCATGAATAATTGCCCT
TGTCATCCCACGTAACGAGAACTTTCATAGATTATATGGGGATGCTGGCGAAACTGAGCAAAAATAATAA
GCTATAGGAGGGCGATTCGGTCGGCAACAGAGGTTGATCGAAATATGATCTGACTTGGGCCATATTATGG
AATACATTTCATAACATCAAGATCCCAAAATGACATGCATCACTCTAGGAAAAAAGTTGCAAAATGTTCG
TTAACTTAGCACGAAGGTATATATTAGAAATCCCGCGCGGTACACGCTCAAGGCGCTGTTTGCGCTCAAA
CCTTGCGATCAGCTGCGATCATACAACAAATGAACACCTACTTTCGAAATAGCATATGTCATATGTTGTG
AGAGATTTTTGGAATGCAGATTTGGTATAGGATGAAGCATTAGAATTAAAGGAAATTTTAAAATGGGAGG
TAAACATAAACAGGCTCAAAGAACCAAAAATAACGCACGGGTTAGTTGATATCTTACTATGATAATTTTC
ATTTAACATTATTGACAAGTATACTTAAAAGAAACTTGAAAATACCAGTTAATGTTAATTATTATTTCAT
...

Crunching 314,572,800 bytes long www.ncbi.nlm.nih.govDragonfly(Ladona_fulva)_whole_genome_shotgun.tar_first_300MB ...

dragonfly_8975rmtx-1415241634

TurboBench: - Fri Apr 27 21:51:45 2018

(bold = pareto) MB=1.000.000

C Size	ratio	C MB/s	D MB/s	Name	File
74844587	23.8	0.32	0.31	zpaq 5	www.ncbi.nlm.nih.govDragonfly(Ladona_fulva)_whole_genome_shotgun.tar_first_300MB
76363252	24.3	0.71	1090.38	lzturbo 39	www.ncbi.nlm.nih.govDragonfly(Ladona_fulva)_whole_genome_shotgun.tar_first_300MB
79031404	25.1	0.48	51.82	lzlib 9d29fb273	www.ncbi.nlm.nih.govDragonfly(Ladona_fulva)_whole_genome_shotgun.tar_first_300MB
79048973	25.1	5.62	27.45	lzturbo 59	www.ncbi.nlm.nih.govDragonfly(Ladona_fulva)_whole_genome_shotgun.tar_first_300MB
79049497	25.1	6.02	45.81	lzturbo 59t2	www.ncbi.nlm.nih.govDragonfly(Ladona_fulva)_whole_genome_shotgun.tar_first_300MB
79050237	25.1	6.14	52.16	lzturbo 59t4	www.ncbi.nlm.nih.govDragonfly(Ladona_fulva)_whole_genome_shotgun.tar_first_300MB
79050805	25.1	0.62	73.48	lzturbo 49	www.ncbi.nlm.nih.govDragonfly(Ladona_fulva)_whole_genome_shotgun.tar_first_300MB
79106644	25.1	0.50	70.35	lzma 9d29:fb273:mf=bt4	www.ncbi.nlm.nih.govDragonfly(Ladona_fulva)_whole_genome_shotgun.tar_first_300MB
79843615	25.4	0.57	70.32	lzma 9	www.ncbi.nlm.nih.govDragonfly(Ladona_fulva)_whole_genome_shotgun.tar_first_300MB
79847082	25.4	0.20	119.64	brotli 11d29	www.ncbi.nlm.nih.govDragonfly(Ladona_fulva)_whole_genome_shotgun.tar_first_300MB
79952387	25.4	0.25	154.97	lzham 4fb258:x4:d29	www.ncbi.nlm.nih.govDragonfly(Ladona_fulva)_whole_genome_shotgun.tar_first_300MB
80243366	25.5	21.40	20.75	bsc 3	www.ncbi.nlm.nih.govDragonfly(Ladona_fulva)_whole_genome_shotgun.tar_first_300MB
80327238	25.5	20.96	11.23	bsc 6	www.ncbi.nlm.nih.govDragonfly(Ladona_fulva)_whole_genome_shotgun.tar_first_300MB
80595766	25.6	0.53	162.27	lzham 4	www.ncbi.nlm.nih.govDragonfly(Ladona_fulva)_whole_genome_shotgun.tar_first_300MB
81447524	25.9	0.67	504.33	zstd 22	www.ncbi.nlm.nih.govDragonfly(Ladona_fulva)_whole_genome_shotgun.tar_first_300MB
82460233	26.2	0.26	127.11	brotli 11	www.ncbi.nlm.nih.govDragonfly(Ladona_fulva)_whole_genome_shotgun.tar_first_300MB
83368983	26.5	0.23	619.11	oodle 89	www.ncbi.nlm.nih.govDragonfly(Ladona_fulva)_whole_genome_shotgun.tar_first_300MB
83985046	26.7	0.14	344.22	oodle 19	www.ncbi.nlm.nih.govDragonfly(Ladona_fulva)_whole_genome_shotgun.tar_first_300MB
85499438	27.2	0.21	1156.87	oodle 129	www.ncbi.nlm.nih.govDragonfly(Ladona_fulva)_whole_genome_shotgun.tar_first_300MB
87322818	27.8	0.94	1173.42	lizard 49	www.ncbi.nlm.nih.govDragonfly(Ladona_fulva)_whole_genome_shotgun.tar_first_300MB
87362764	27.8	3.53	753.81	libdeflate 12	www.ncbi.nlm.nih.govDragonfly(Ladona_fulva)_whole_genome_shotgun.tar_first_300MB
88497987	28.1	11.13	19.68	bzip2	www.ncbi.nlm.nih.govDragonfly(Ladona_fulva)_whole_genome_shotgun.tar_first_300MB
88952252	28.3	4.03	1673.78	lizard 39	www.ncbi.nlm.nih.govDragonfly(Ladona_fulva)_whole_genome_shotgun.tar_first_300MB
92010880	29.2	34.46	1237.04	lzturbo 32	www.ncbi.nlm.nih.govDragonfly(Ladona_fulva)_whole_genome_shotgun.tar_first_300MB
92351753	29.4	0.76	788.88	lzturbo 29	www.ncbi.nlm.nih.govDragonfly(Ladona_fulva)_whole_genome_shotgun.tar_first_300MB
93411984	29.7	7.41	689.57	zstd 12	www.ncbi.nlm.nih.govDragonfly(Ladona_fulva)_whole_genome_shotgun.tar_first_300MB
93610827	29.8	2.05	749.11	xpack 9	www.ncbi.nlm.nih.govDragonfly(Ladona_fulva)_whole_genome_shotgun.tar_first_300MB
94243984	30.0	17.06	370.65	brotli 5	www.ncbi.nlm.nih.govDragonfly(Ladona_fulva)_whole_genome_shotgun.tar_first_300MB
96420245	30.7	54.29	632.27	libdeflate 5	www.ncbi.nlm.nih.govDragonfly(Ladona_fulva)_whole_genome_shotgun.tar_first_300MB
96702059	30.7	97.46	537.50	zstd 5	www.ncbi.nlm.nih.govDragonfly(Ladona_fulva)_whole_genome_shotgun.tar_first_300MB
99382236	31.6	223.03	313.42	brotli 1	www.ncbi.nlm.nih.govDragonfly(Ladona_fulva)_whole_genome_shotgun.tar_first_300MB
99685993	31.7	191.18	669.66	zstd 1	www.ncbi.nlm.nih.govDragonfly(Ladona_fulva)_whole_genome_shotgun.tar_first_300MB
100188891	31.8	138.51	517.49	libdeflate 1	www.ncbi.nlm.nih.govDragonfly(Ladona_fulva)_whole_genome_shotgun.tar_first_300MB
100198117	31.9	110.34	433.78	xpack 1	www.ncbi.nlm.nih.govDragonfly(Ladona_fulva)_whole_genome_shotgun.tar_first_300MB
103311753	32.8	3.56	66.08	zpaq 2	www.ncbi.nlm.nih.govDragonfly(Ladona_fulva)_whole_genome_shotgun.tar_first_300MB
112080411	35.6	0.95	1692.71	lizard 29	www.ncbi.nlm.nih.govDragonfly(Ladona_fulva)_whole_genome_shotgun.tar_first_300MB
114000407	36.2	0.24	3186.52	oodle 116	www.ncbi.nlm.nih.govDragonfly(Ladona_fulva)_whole_genome_shotgun.tar_first_300MB
114399132	36.4	0.92	3380.76	lzturbo 19	www.ncbi.nlm.nih.govDragonfly(Ladona_fulva)_whole_genome_shotgun.tar_first_300MB
114443457	36.4	2.53	2816.00	oodle 49	www.ncbi.nlm.nih.govDragonfly(Ladona_fulva)_whole_genome_shotgun.tar_first_300MB
114506143	36.4	4.12	2469.72	lizard 19	www.ncbi.nlm.nih.govDragonfly(Ladona_fulva)_whole_genome_shotgun.tar_first_300MB
119070578	37.9	0.26	3109.41	oodle 118	www.ncbi.nlm.nih.govDragonfly(Ladona_fulva)_whole_genome_shotgun.tar_first_300MB
127571232	40.6	34.94	1021.69	lzturbo 22	www.ncbi.nlm.nih.govDragonfly(Ladona_fulva)_whole_genome_shotgun.tar_first_300MB
130954049	41.6	66.45	3119.00	lzturbo 12	www.ncbi.nlm.nih.govDragonfly(Ladona_fulva)_whole_genome_shotgun.tar_first_300MB
132077784	42.0	20.10	3794.70	oodle 114	www.ncbi.nlm.nih.govDragonfly(Ladona_fulva)_whole_genome_shotgun.tar_first_300MB
162941764	51.8	584.20	555.17	density 3	www.ncbi.nlm.nih.govDragonfly(Ladona_fulva)_whole_genome_shotgun.tar_first_300MB
167155790	53.1	1754.57	4351.24	chameleon 2	www.ncbi.nlm.nih.govDragonfly(Ladona_fulva)_whole_genome_shotgun.tar_first_300MB
199758502	63.5	121.24	2141.26	oodle 112	www.ncbi.nlm.nih.govDragonfly(Ladona_fulva)_whole_genome_shotgun.tar_first_300MB
284139766	90.3	180.76	875.99	trle	www.ncbi.nlm.nih.govDragonfly(Ladona_fulva)_whole_genome_shotgun.tar_first_300MB

Failed:

           0     0.0       0.00       0.00   lzsse2 1         www.ncbi.nlm.nih.gov_Dragonfly_(Ladona_fulva)_whole_genome_shotgun.tar_first_300MB
ERROR at 0:77, 91
           0     0.0       0.00       0.00   lzsse2 17        www.ncbi.nlm.nih.gov_Dragonfly_(Ladona_fulva)_whole_genome_shotgun.tar_first_300MB
ERROR at 0:77, 91

So, the testsets that are to be processed:

04/26/2018  11:47 AM       314,572,800 enwiki-20170101-pages-articles.xml_first_300MB
11/13/2017  01:08 AM       405,610,647 Machine-Learning_Douban_Movie_Short_Comments_(Chinese).csv
11/13/2017  01:08 AM       150,950,913 Machine-Learning_Global_Terrorism_Database_(more_than_170000_terrorist_attacks_worldwide_1970-2016).csv
11/13/2017  01:08 AM       203,288,144 Machine-Learning_www.kaggle.com_examine-the-examiner_(headlines_of_3_million_articles).csv
11/13/2017  01:08 AM       282,218,054 Machine-Learning_www.kaggle.com_opencorpora-russian_(A_Tagged_1.5_Million_Word_Corpus_of_Russian).txt
04/26/2018  12:00 PM       314,572,800 www.ncbi.nlm.nih.gov_Dragonfly_(Ladona_fulva)_whole_genome_shotgun.tar_first_300MB

Chinese comments in .CSV, a must-see one it is. porco_chines13

Rethinking how scattered all the benchmark packages are, once I did above ones, surely will put all the corpora in one RAZOR archive, to pay tribute to the excellent work done by Christian Martelock, thus as a "by-product" or bonus, we can see how the whole/solid compression goes on all the testsets as a whole!

Okay, today's post will be the last since I finished the benchmarking and just 2 hours away from uploading the corpus housing all the testsets benchmarked on this thread.

The rest are given one by one, after that the promised table of speedups, and in the very end, the link to the corpus will be given.

The first one is very interesting being superimportant languagewise/inflectionwise, each word being inflected and tagged, a precious corpus indeed!

Crunching 282,218,054 bytes long Machine-Learning_www.kaggle.comopencorpora-russian(A_Tagged_1.5_Million_Word_Corpus_of_Russian).txt ...

russianwords_

TurboBench: - Sun Apr 29 17:12:27 2018

(bold = pareto) MB=1.000.000

C Size	ratio	C MB/s	D MB/s	Name	File
6189155	2.2	0.39	0.38	zpaq 5	Machine-Learning_www.kaggle.comopencorpora-russian(A_Tagged_1.5_Million_Word_Corpus_of_Russian).txt
6955472	2.5	62.39	27.44	bsc 6	Machine-Learning_www.kaggle.comopencorpora-russian(A_Tagged_1.5_Million_Word_Corpus_of_Russian).txt
8657900	3.1	64.44	34.62	bsc 3	Machine-Learning_www.kaggle.comopencorpora-russian(A_Tagged_1.5_Million_Word_Corpus_of_Russian).txt
11705323	4.1	0.25	562.50	brotli 11d29	Machine-Learning_www.kaggle.comopencorpora-russian(A_Tagged_1.5_Million_Word_Corpus_of_Russian).txt
11737167	4.2	0.98	354.07	lzturbo 49	Machine-Learning_www.kaggle.comopencorpora-russian(A_Tagged_1.5_Million_Word_Corpus_of_Russian).txt
12167321	4.3	0.31	776.03	brotli 11	Machine-Learning_www.kaggle.comopencorpora-russian(A_Tagged_1.5_Million_Word_Corpus_of_Russian).txt
12588067	4.5	0.17	601.03	lzham 4fb258:x4:d29	Machine-Learning_www.kaggle.comopencorpora-russian(A_Tagged_1.5_Million_Word_Corpus_of_Russian).txt
12677250	4.5	0.39	163.37	lzlib 9d29fb273	Machine-Learning_www.kaggle.comopencorpora-russian(A_Tagged_1.5_Million_Word_Corpus_of_Russian).txt
12925754	4.6	0.60	269.19	lzma 9d29:fb273:mf=bt4	Machine-Learning_www.kaggle.comopencorpora-russian(A_Tagged_1.5_Million_Word_Corpus_of_Russian).txt
13806329	4.9	1.20	2193.35	lzturbo 39	Machine-Learning_www.kaggle.comopencorpora-russian(A_Tagged_1.5_Million_Word_Corpus_of_Russian).txt
14352030	5.1	0.47	1270.26	zstd 22	Machine-Learning_www.kaggle.comopencorpora-russian(A_Tagged_1.5_Million_Word_Corpus_of_Russian).txt
14561448	5.2	8.66	54.65	bzip2	Machine-Learning_www.kaggle.comopencorpora-russian(A_Tagged_1.5_Million_Word_Corpus_of_Russian).txt
14666520	5.2	1.42	256.24	lzma 9	Machine-Learning_www.kaggle.comopencorpora-russian(A_Tagged_1.5_Million_Word_Corpus_of_Russian).txt
15004589	5.3	8.63	53.40	lzturbo 59	Machine-Learning_www.kaggle.comopencorpora-russian(A_Tagged_1.5_Million_Word_Corpus_of_Russian).txt
15004965	5.3	8.73	67.17	lzturbo 59t2	Machine-Learning_www.kaggle.comopencorpora-russian(A_Tagged_1.5_Million_Word_Corpus_of_Russian).txt
15005373	5.3	8.79	70.43	lzturbo 59t4	Machine-Learning_www.kaggle.comopencorpora-russian(A_Tagged_1.5_Million_Word_Corpus_of_Russian).txt
15340888	5.4	0.64	845.55	lzham 4	Machine-Learning_www.kaggle.comopencorpora-russian(A_Tagged_1.5_Million_Word_Corpus_of_Russian).txt
17171617	6.1	0.38	1266.82	oodle 19	Machine-Learning_www.kaggle.comopencorpora-russian(A_Tagged_1.5_Million_Word_Corpus_of_Russian).txt
18327223	6.5	0.53	1939.92	oodle 129	Machine-Learning_www.kaggle.comopencorpora-russian(A_Tagged_1.5_Million_Word_Corpus_of_Russian).txt
18441799	6.5	0.65	1749.74	oodle 89	Machine-Learning_www.kaggle.comopencorpora-russian(A_Tagged_1.5_Million_Word_Corpus_of_Russian).txt
20900839	7.4	50.00	1683.05	zstd 12	Machine-Learning_www.kaggle.comopencorpora-russian(A_Tagged_1.5_Million_Word_Corpus_of_Russian).txt
21090625	7.5	60.01	783.55	brotli 5	Machine-Learning_www.kaggle.comopencorpora-russian(A_Tagged_1.5_Million_Word_Corpus_of_Russian).txt
21806722	7.7	6.35	1544.74	xpack 9	Machine-Learning_www.kaggle.comopencorpora-russian(A_Tagged_1.5_Million_Word_Corpus_of_Russian).txt
22268287	7.9	2.77	1694.52	libdeflate 12	Machine-Learning_www.kaggle.comopencorpora-russian(A_Tagged_1.5_Million_Word_Corpus_of_Russian).txt
24748719	8.8	1.38	3019.80	lzturbo 29	Machine-Learning_www.kaggle.comopencorpora-russian(A_Tagged_1.5_Million_Word_Corpus_of_Russian).txt
24908224	8.8	253.39	1055.81	zstd 5	Machine-Learning_www.kaggle.comopencorpora-russian(A_Tagged_1.5_Million_Word_Corpus_of_Russian).txt
24937942	8.8	0.43	2538.46	lizard 49	Machine-Learning_www.kaggle.comopencorpora-russian(A_Tagged_1.5_Million_Word_Corpus_of_Russian).txt
26740710	9.5	162.99	1644.26	libdeflate 5	Machine-Learning_www.kaggle.comopencorpora-russian(A_Tagged_1.5_Million_Word_Corpus_of_Russian).txt
26859396	9.5	550.76	992.83	zstd 1	Machine-Learning_www.kaggle.comopencorpora-russian(A_Tagged_1.5_Million_Word_Corpus_of_Russian).txt
26883379	9.5	7.58	126.70	zpaq 2	Machine-Learning_www.kaggle.comopencorpora-russian(A_Tagged_1.5_Million_Word_Corpus_of_Russian).txt
26981077	9.6	134.21	1964.55	lzturbo 32	Machine-Learning_www.kaggle.comopencorpora-russian(A_Tagged_1.5_Million_Word_Corpus_of_Russian).txt
27658395	9.8	3.69	2622.31	lizard 39	Machine-Learning_www.kaggle.comopencorpora-russian(A_Tagged_1.5_Million_Word_Corpus_of_Russian).txt
30779166	10.9	0.44	3363.62	lizard 29	Machine-Learning_www.kaggle.comopencorpora-russian(A_Tagged_1.5_Million_Word_Corpus_of_Russian).txt
33495585	11.9	185.44	891.65	xpack 1	Machine-Learning_www.kaggle.comopencorpora-russian(A_Tagged_1.5_Million_Word_Corpus_of_Russian).txt
34028735	12.1	1.55	4339.48	lzturbo 19	Machine-Learning_www.kaggle.comopencorpora-russian(A_Tagged_1.5_Million_Word_Corpus_of_Russian).txt
34084809	12.1	4.02	3373.59	lizard 19	Machine-Learning_www.kaggle.comopencorpora-russian(A_Tagged_1.5_Million_Word_Corpus_of_Russian).txt
34089562	12.1	4.06	3085.94	oodle 49	Machine-Learning_www.kaggle.comopencorpora-russian(A_Tagged_1.5_Million_Word_Corpus_of_Russian).txt
34142961	12.1	249.54	1688.79	libdeflate 1	Machine-Learning_www.kaggle.comopencorpora-russian(A_Tagged_1.5_Million_Word_Corpus_of_Russian).txt
35012543	12.4	416.25	704.67	brotli 1	Machine-Learning_www.kaggle.comopencorpora-russian(A_Tagged_1.5_Million_Word_Corpus_of_Russian).txt
35081145	12.4	0.81	4493.70	oodle 118	Machine-Learning_www.kaggle.comopencorpora-russian(A_Tagged_1.5_Million_Word_Corpus_of_Russian).txt
35125184	12.4	0.60	4500.01	oodle 116	Machine-Learning_www.kaggle.comopencorpora-russian(A_Tagged_1.5_Million_Word_Corpus_of_Russian).txt
37707356	13.4	442.78	416.86	density 3	Machine-Learning_www.kaggle.comopencorpora-russian(A_Tagged_1.5_Million_Word_Corpus_of_Russian).txt
40311881	14.3	151.08	3017.31	lzturbo 22	Machine-Learning_www.kaggle.comopencorpora-russian(A_Tagged_1.5_Million_Word_Corpus_of_Russian).txt
42929059	15.2	74.33	4511.37	oodle 114	Machine-Learning_www.kaggle.comopencorpora-russian(A_Tagged_1.5_Million_Word_Corpus_of_Russian).txt
45343935	16.1	176.73	3934.89	lzturbo 12	Machine-Learning_www.kaggle.comopencorpora-russian(A_Tagged_1.5_Million_Word_Corpus_of_Russian).txt
57572531	20.4	313.65	4971.43	oodle 112	Machine-Learning_www.kaggle.comopencorpora-russian(A_Tagged_1.5_Million_Word_Corpus_of_Russian).txt
150259256	53.2	1602.37	3632.66	chameleon 2	Machine-Learning_www.kaggle.comopencorpora-russian(A_Tagged_1.5_Million_Word_Corpus_of_Russian).txt
282203529	100.0	193.01	2209.35	trle	Machine-Learning_www.kaggle.comopencorpora-russian(A_Tagged_1.5_Million_Word_Corpus_of_Russian).txt

Failed:

           0     0.0       0.00       0.00   lzsse2 1         Machine-Learning_www.kaggle.com_opencorpora-russian_(A_Tagged_1.5_Million_Word_Corpus_of_Russian).txt
ERROR at 0:31, 2f
           0     0.0       0.00       0.00   lzsse2 17        Machine-Learning_www.kaggle.com_opencorpora-russian_(A_Tagged_1.5_Million_Word_Corpus_of_Russian).txt
ERROR at 0:31, 2f

Inhere, lzturbo 39 dominates, however for my surprise lzturbo 59 fails to compete with bsc 6, that's one of the gifts of benchmarking, it always shows something overlooked, a weakness which needs addressing.

The next one holds precious code from the legendary Watcom C. Crunching 259,707,904 bytes long open_watcom_1.9.0-src.tar ...

watcom5

TurboBench: - Mon Apr 30 07:51:32 2018

(bold = pareto) MB=1.000.000

C Size	ratio	C MB/s	D MB/s	Name	File
17324107	6.7	0.43	0.43	zpaq 5	open_watcom_1.9.0-src.tar
20328299	7.8	0.80	218.74	lzturbo 49	open_watcom_1.9.0-src.tar
20737792	8.0	1.48	119.33	lzlib 9d29fb273	open_watcom_1.9.0-src.tar
20839672	8.0	1.69	191.64	lzma 9d29:fb273:mf=bt4	open_watcom_1.9.0-src.tar
21117099	8.1	0.52	515.13	brotli 11d29	open_watcom_1.9.0-src.tar
21670652	8.3	0.06	488.38	lzham 4fb258:x4:d29	open_watcom_1.9.0-src.tar
21759173	8.4	1.14	1778.47	lzturbo 39	open_watcom_1.9.0-src.tar
21865190	8.4	3.01	184.26	lzma 9	open_watcom_1.9.0-src.tar
22249772	8.6	2.32	1271.71	zstd 22	open_watcom_1.9.0-src.tar
22298640	8.6	1.49	480.21	lzham 4	open_watcom_1.9.0-src.tar
22756320	8.8	40.43	23.21	bsc 6	open_watcom_1.9.0-src.tar
22898691	8.8	0.57	661.98	brotli 11	open_watcom_1.9.0-src.tar
23001107	8.9	0.79	1488.99	oodle 129	open_watcom_1.9.0-src.tar
23001719	8.9	0.93	1488.94	oodle 89	open_watcom_1.9.0-src.tar
23016893	8.9	0.51	949.12	oodle 19	open_watcom_1.9.0-src.tar
23741009	9.1	12.24	46.80	lzturbo 59	open_watcom_1.9.0-src.tar
23744593	9.1	12.91	64.21	lzturbo 59t2	open_watcom_1.9.0-src.tar
23752157	9.1	13.02	69.19	lzturbo 59t4	open_watcom_1.9.0-src.tar
24976908	9.6	49.19	31.81	bsc 3	open_watcom_1.9.0-src.tar
26963637	10.4	36.90	1481.61	zstd 12	open_watcom_1.9.0-src.tar
26998731	10.4	49.17	661.76	brotli 5	open_watcom_1.9.0-src.tar
27134856	10.4	1.29	2212.31	lzturbo 29	open_watcom_1.9.0-src.tar
27342951	10.5	88.93	1706.24	lzturbo 32	open_watcom_1.9.0-src.tar
27724424	10.7	13.52	52.46	bzip2	open_watcom_1.9.0-src.tar
29421038	11.3	1.16	2306.98	lizard 49	open_watcom_1.9.0-src.tar
29666433	11.4	9.01	125.90	zpaq 2	open_watcom_1.9.0-src.tar
31215741	12.0	200.70	1510.10	zstd 5	open_watcom_1.9.0-src.tar
32725514	12.6	1.22	3019.16	lizard 29	open_watcom_1.9.0-src.tar
33113590	12.8	0.85	4016.14	oodle 116	open_watcom_1.9.0-src.tar
33148507	12.8	1.02	4155.13	oodle 118	open_watcom_1.9.0-src.tar
33794944	13.0	4.64	1269.00	libdeflate 12	open_watcom_1.9.0-src.tar
36519427	14.1	504.41	1532.00	zstd 1	open_watcom_1.9.0-src.tar
36611911	14.1	87.40	2278.18	lzturbo 22	open_watcom_1.9.0-src.tar
36644803	14.1	140.16	1265.61	libdeflate 5	open_watcom_1.9.0-src.tar
37588412	14.5	1.70	2491.54	lizard 39	open_watcom_1.9.0-src.tar
38251459	14.7	56.40	4540.19	oodle 114	open_watcom_1.9.0-src.tar
38310721	14.8	332.12	778.85	brotli 1	open_watcom_1.9.0-src.tar
41036396	15.8	4.83	2797.46	oodle 49	open_watcom_1.9.0-src.tar
41246889	15.9	1.34	4139.37	lzturbo 19	open_watcom_1.9.0-src.tar
41300306	15.9	201.39	979.11	libdeflate 1	open_watcom_1.9.0-src.tar
41480035	16.0	1.85	3523.42	lizard 19	open_watcom_1.9.0-src.tar
45860222	17.7	125.45	3876.30	lzturbo 12	open_watcom_1.9.0-src.tar
48281758	18.6	204.26	4150.81	oodle 112	open_watcom_1.9.0-src.tar
59997812	23.1	360.93	360.90	density 3	open_watcom_1.9.0-src.tar
148200642	57.1	1659.20	2860.75	chameleon 2	open_watcom_1.9.0-src.tar
165724209	63.8	298.97	3894.14	trle	open_watcom_1.9.0-src.tar

Failed:

           0     0.0       0.00       0.00   lzsse2 1         open_watcom_1.9.0-src.tar
ERROR at 0:6f, a0
           0     0.0       0.00       0.00   lzsse2 17        open_watcom_1.9.0-src.tar
ERROR at 0:6f, a0

    36805790    14.2     158.01    1671.04   xpack 1          open_watcom_1.9.0-src.tar
ERROR at 144211230:34, cb
    25917386    10.0       1.57    1818.47   xpack 9          open_watcom_1.9.0-src.tar
ERROR at 144315853:20, 4c

Two standouts (except the usual number one lzturbo 39), zstd 1 and oodle 114 shine, or rather darken the rest casting long shadows.

Crunching 405,610,647 bytes long Machine-Learning_Douban_Movie_ShortComments(Chinese).csv ...

douban-min

TurboBench: - Sun Apr 29 08:33:59 2018

(bold = pareto) MB=1.000.000

C Size	ratio	C MB/s	D MB/s	Name	File
70413122	17.4	0.36	0.36	zpaq 5	Machine-Learning_Douban_Movie_ShortComments(Chinese).csv
70488916	17.4	17.91	6.58	bsc 6	Machine-Learning_Douban_Movie_ShortComments(Chinese).csv
72381729	17.8	8.05	33.19	lzturbo 59	Machine-Learning_Douban_Movie_ShortComments(Chinese).csv
72384949	17.8	8.43	46.65	lzturbo 59t2	Machine-Learning_Douban_Movie_ShortComments(Chinese).csv
72389457	17.8	8.50	50.31	lzturbo 59t4	Machine-Learning_Douban_Movie_ShortComments(Chinese).csv
82449790	20.3	24.30	11.46	bsc 3	Machine-Learning_Douban_Movie_ShortComments(Chinese).csv
84857934	20.9	0.54	61.41	lzlib 9d29fb273	Machine-Learning_Douban_Movie_ShortComments(Chinese).csv
85031287	21.0	0.69	81.20	lzma 9d29:fb273:mf=bt4	Machine-Learning_Douban_Movie_ShortComments(Chinese).csv
85620216	21.1	0.68	68.71	lzturbo 49	Machine-Learning_Douban_Movie_ShortComments(Chinese).csv
86692998	21.4	0.27	174.55	lzham 4fb258:x4:d29	Machine-Learning_Douban_Movie_ShortComments(Chinese).csv
87700851	21.6	0.62	173.62	lzham 4	Machine-Learning_Douban_Movie_ShortComments(Chinese).csv
87834073	21.7	1.12	80.04	lzma 9	Machine-Learning_Douban_Movie_ShortComments(Chinese).csv
88748119	21.9	0.36	227.64	brotli 11d29	Machine-Learning_Douban_Movie_ShortComments(Chinese).csv
89783640	22.1	0.79	557.91	lzturbo 39	Machine-Learning_Douban_Movie_ShortComments(Chinese).csv
91795746	22.6	0.39	498.64	oodle 89	Machine-Learning_Douban_Movie_ShortComments(Chinese).csv
91795746	22.6	0.35	498.01	oodle 129	Machine-Learning_Douban_Movie_ShortComments(Chinese).csv
92414852	22.8	0.68	581.26	zstd 22	Machine-Learning_Douban_Movie_ShortComments(Chinese).csv
96123845	23.7	11.32	29.21	bzip2	Machine-Learning_Douban_Movie_ShortComments(Chinese).csv
96594457	23.8	0.23	284.19	oodle 19	Machine-Learning_Douban_Movie_ShortComments(Chinese).csv
99010755	24.4	0.44	331.85	brotli 11	Machine-Learning_Douban_Movie_ShortComments(Chinese).csv
108092512	26.6	31.16	569.19	lzturbo 32	Machine-Learning_Douban_Movie_ShortComments(Chinese).csv
108840695	26.8	0.82	815.90	lzturbo 29	Machine-Learning_Douban_Movie_ShortComments(Chinese).csv
113508557	28.0	0.46	362.85	xpack 9	Machine-Learning_Douban_Movie_ShortComments(Chinese).csv
119520522	29.5	7.62	666.10	zstd 12	Machine-Learning_Douban_Movie_ShortComments(Chinese).csv
120370688	29.7	19.52	337.67	brotli 5	Machine-Learning_Douban_Movie_ShortComments(Chinese).csv
122781661	30.3	3.83	73.46	zpaq 2	Machine-Learning_Douban_Movie_ShortComments(Chinese).csv
128802175	31.8	0.80	813.77	lizard 49	Machine-Learning_Douban_Movie_ShortComments(Chinese).csv
133841998	33.0	81.91	684.08	zstd 5	Machine-Learning_Douban_Movie_ShortComments(Chinese).csv
138385262	34.1	6.59	579.17	libdeflate 12	Machine-Learning_Douban_Movie_ShortComments(Chinese).csv
144271916	35.6	30.75	998.53	lzturbo 22	Machine-Learning_Douban_Movie_ShortComments(Chinese).csv
145629166	35.9	0.87	1244.51	lizard 29	Machine-Learning_Douban_Movie_ShortComments(Chinese).csv
145838302	36.0	6.96	2010.12	lizard 39	Machine-Learning_Douban_Movie_ShortComments(Chinese).csv
147769335	36.4	71.94	538.64	libdeflate 5	Machine-Learning_Douban_Movie_ShortComments(Chinese).csv
152898024	37.7	148.34	297.69	brotli 1	Machine-Learning_Douban_Movie_ShortComments(Chinese).csv
153973028	38.0	83.79	611.91	xpack 1	Machine-Learning_Douban_Movie_ShortComments(Chinese).csv
155196655	38.3	0.41	2586.92	oodle 116	Machine-Learning_Douban_Movie_ShortComments(Chinese).csv
156708263	38.6	118.11	519.29	libdeflate 1	Machine-Learning_Douban_Movie_ShortComments(Chinese).csv
157678551	38.9	0.44	2834.20	oodle 118	Machine-Learning_Douban_Movie_ShortComments(Chinese).csv
160981785	39.7	3.69	2305.93	oodle 49	Machine-Learning_Douban_Movie_ShortComments(Chinese).csv
161019023	39.7	0.94	3615.84	lzturbo 19	Machine-Learning_Douban_Movie_ShortComments(Chinese).csv
161352654	39.8	7.42	2612.24	lizard 19	Machine-Learning_Douban_Movie_ShortComments(Chinese).csv
164070077	40.5	20.14	3209.58	oodle 114	Machine-Learning_Douban_Movie_ShortComments(Chinese).csv
169407769	41.8	257.61	283.99	density 3	Machine-Learning_Douban_Movie_ShortComments(Chinese).csv
169680970	41.8	50.34	3573.32	lzturbo 12	Machine-Learning_Douban_Movie_ShortComments(Chinese).csv
174713968	43.1	261.51	790.18	zstd 1	Machine-Learning_Douban_Movie_ShortComments(Chinese).csv
189771059	46.8	62.13	3238.49	oodle 112	Machine-Learning_Douban_Movie_ShortComments(Chinese).csv
253924877	62.6	1578.96	1830.11	chameleon 2	Machine-Learning_Douban_Movie_ShortComments(Chinese).csv
405200855	99.9	192.69	1980.43	trle	Machine-Learning_Douban_Movie_ShortComments(Chinese).csv

Failed:

           0     0.0       0.00       0.00   lzsse2 1         Machine-Learning_Douban_Movie_Short_Comments_(Chinese).csv
ERROR at 0:ef, bb
           0     0.0       0.00       0.00   lzsse2 17        Machine-Learning_Douban_Movie_Short_Comments_(Chinese).csv
ERROR at 0:ef, bb

Inhere, one particular thing got me thinking, the 1GB/s difference between the superb performers lzturbo 19 and lizard 19, being on par sizewise, sometimes decompression rates are on par NOT at all.

Crunching 150,950,913 bytes long Machine-Learning_Global_TerrorismDatabase(more_than_170000_terrorist_attacks_worldwide_1970-2016).csv ...

terrorismyemen conflict

TurboBench: - Sat Apr 28 07:35:55 2018

(bold = pareto) MB=1.000.000

C Size	ratio	C MB/s	D MB/s	Name	File
10212045	6.8	0.39	0.39	zpaq 5	Machine-Learning_Global_TerrorismDatabase(more_than_170000_terrorist_attacks_worldwide_1970-2016).csv
10539341	7.0	9.61	46.96	lzturbo 59	Machine-Learning_Global_TerrorismDatabase(more_than_170000_terrorist_attacks_worldwide_1970-2016).csv
10540357	7.0	9.84	62.24	lzturbo 59t2	Machine-Learning_Global_TerrorismDatabase(more_than_170000_terrorist_attacks_worldwide_1970-2016).csv
10542429	7.0	9.96	68.95	lzturbo 59t4	Machine-Learning_Global_TerrorismDatabase(more_than_170000_terrorist_attacks_worldwide_1970-2016).csv
12296870	8.1	37.02	12.66	bsc 6	Machine-Learning_Global_TerrorismDatabase(more_than_170000_terrorist_attacks_worldwide_1970-2016).csv
12665445	8.4	0.48	202.64	lzturbo 49	Machine-Learning_Global_TerrorismDatabase(more_than_170000_terrorist_attacks_worldwide_1970-2016).csv
12935430	8.6	1.01	116.07	lzlib 9d29fb273	Machine-Learning_Global_TerrorismDatabase(more_than_170000_terrorist_attacks_worldwide_1970-2016).csv
12968621	8.6	1.13	188.33	lzma 9d29:fb273:mf=bt4	Machine-Learning_Global_TerrorismDatabase(more_than_170000_terrorist_attacks_worldwide_1970-2016).csv
13085028	8.7	0.41	428.02	brotli 11d29	Machine-Learning_Global_TerrorismDatabase(more_than_170000_terrorist_attacks_worldwide_1970-2016).csv
13102376	8.7	0.16	593.84	lzham 4fb258:x4:d29	Machine-Learning_Global_TerrorismDatabase(more_than_170000_terrorist_attacks_worldwide_1970-2016).csv
13289188	8.8	0.64	1492.75	lzturbo 39	Machine-Learning_Global_TerrorismDatabase(more_than_170000_terrorist_attacks_worldwide_1970-2016).csv
13671840	9.1	1.27	1201.26	zstd 22	Machine-Learning_Global_TerrorismDatabase(more_than_170000_terrorist_attacks_worldwide_1970-2016).csv
13795560	9.1	1.14	448.98	lzham 4	Machine-Learning_Global_TerrorismDatabase(more_than_170000_terrorist_attacks_worldwide_1970-2016).csv
13968253	9.3	2.10	177.63	lzma 9	Machine-Learning_Global_TerrorismDatabase(more_than_170000_terrorist_attacks_worldwide_1970-2016).csv
14154723	9.4	0.45	706.85	brotli 11	Machine-Learning_Global_TerrorismDatabase(more_than_170000_terrorist_attacks_worldwide_1970-2016).csv
14399499	9.5	0.30	875.43	oodle 19	Machine-Learning_Global_TerrorismDatabase(more_than_170000_terrorist_attacks_worldwide_1970-2016).csv
14775481	9.8	0.54	1188.96	oodle 129	Machine-Learning_Global_TerrorismDatabase(more_than_170000_terrorist_attacks_worldwide_1970-2016).csv
14775481	9.8	0.61	1086.76	oodle 89	Machine-Learning_Global_TerrorismDatabase(more_than_170000_terrorist_attacks_worldwide_1970-2016).csv
15908060	10.5	0.68	1942.49	lzturbo 29	Machine-Learning_Global_TerrorismDatabase(more_than_170000_terrorist_attacks_worldwide_1970-2016).csv
16424770	10.9	1.59	892.51	xpack 9	Machine-Learning_Global_TerrorismDatabase(more_than_170000_terrorist_attacks_worldwide_1970-2016).csv
16685531	11.1	9.70	33.11	bzip2	Machine-Learning_Global_TerrorismDatabase(more_than_170000_terrorist_attacks_worldwide_1970-2016).csv
17071177	11.3	1.01	1662.62	lizard 49	Machine-Learning_Global_TerrorismDatabase(more_than_170000_terrorist_attacks_worldwide_1970-2016).csv
17461845	11.6	6.31	119.43	zpaq 2	Machine-Learning_Global_TerrorismDatabase(more_than_170000_terrorist_attacks_worldwide_1970-2016).csv
17551699	11.6	20.23	1305.42	zstd 12	Machine-Learning_Global_TerrorismDatabase(more_than_170000_terrorist_attacks_worldwide_1970-2016).csv
18082278	12.0	43.98	671.06	brotli 5	Machine-Learning_Global_TerrorismDatabase(more_than_170000_terrorist_attacks_worldwide_1970-2016).csv
18411473	12.2	1.09	2103.96	lizard 29	Machine-Learning_Global_TerrorismDatabase(more_than_170000_terrorist_attacks_worldwide_1970-2016).csv
18689946	12.4	64.61	1332.54	lzturbo 32	Machine-Learning_Global_TerrorismDatabase(more_than_170000_terrorist_attacks_worldwide_1970-2016).csv
20499531	13.6	0.64	2402.22	oodle 118	Machine-Learning_Global_TerrorismDatabase(more_than_170000_terrorist_attacks_worldwide_1970-2016).csv
20515457	13.6	0.46	2407.17	oodle 116	Machine-Learning_Global_TerrorismDatabase(more_than_170000_terrorist_attacks_worldwide_1970-2016).csv
21950930	14.5	34.14	19.87	bsc 3	Machine-Learning_Global_TerrorismDatabase(more_than_170000_terrorist_attacks_worldwide_1970-2016).csv
22115676	14.7	159.31	1185.41	zstd 5	Machine-Learning_Global_TerrorismDatabase(more_than_170000_terrorist_attacks_worldwide_1970-2016).csv
24421187	16.2	65.85	1758.64	lzturbo 22	Machine-Learning_Global_TerrorismDatabase(more_than_170000_terrorist_attacks_worldwide_1970-2016).csv
25862420	17.1	4.49	876.02	libdeflate 12	Machine-Learning_Global_TerrorismDatabase(more_than_170000_terrorist_attacks_worldwide_1970-2016).csv
25918012	17.2	2.79	2225.69	lizard 39	Machine-Learning_Global_TerrorismDatabase(more_than_170000_terrorist_attacks_worldwide_1970-2016).csv
25959905	17.2	45.43	2997.85	oodle 114	Machine-Learning_Global_TerrorismDatabase(more_than_170000_terrorist_attacks_worldwide_1970-2016).csv
26849738	17.8	378.74	1190.33	zstd 1	Machine-Learning_Global_TerrorismDatabase(more_than_170000_terrorist_attacks_worldwide_1970-2016).csv
27351877	18.1	3.78	3168.97	lzsse2 17	Machine-Learning_Global_TerrorismDatabase(more_than_170000_terrorist_attacks_worldwide_1970-2016).csv
27912385	18.5	112.14	891.83	libdeflate 5	Machine-Learning_Global_TerrorismDatabase(more_than_170000_terrorist_attacks_worldwide_1970-2016).csv
27942491	18.5	249.75	509.24	brotli 1	Machine-Learning_Global_TerrorismDatabase(more_than_170000_terrorist_attacks_worldwide_1970-2016).csv
28214866	18.7	0.79	3391.70	lzturbo 19	Machine-Learning_Global_TerrorismDatabase(more_than_170000_terrorist_attacks_worldwide_1970-2016).csv
28252030	18.7	3.20	2856.59	lizard 19	Machine-Learning_Global_TerrorismDatabase(more_than_170000_terrorist_attacks_worldwide_1970-2016).csv
28300750	18.7	3.74	2438.66	oodle 49	Machine-Learning_Global_TerrorismDatabase(more_than_170000_terrorist_attacks_worldwide_1970-2016).csv
28825769	19.1	127.85	827.57	xpack 1	Machine-Learning_Global_TerrorismDatabase(more_than_170000_terrorist_attacks_worldwide_1970-2016).csv
31883592	21.1	99.32	3167.78	lzturbo 12	Machine-Learning_Global_TerrorismDatabase(more_than_170000_terrorist_attacks_worldwide_1970-2016).csv
32754775	21.7	167.06	825.85	libdeflate 1	Machine-Learning_Global_TerrorismDatabase(more_than_170000_terrorist_attacks_worldwide_1970-2016).csv
33431164	22.1	14.30	2715.53	lzsse2 1	Machine-Learning_Global_TerrorismDatabase(more_than_170000_terrorist_attacks_worldwide_1970-2016).csv
34436777	22.8	150.79	3149.93	oodle 112	Machine-Learning_Global_TerrorismDatabase(more_than_170000_terrorist_attacks_worldwide_1970-2016).csv
38597585	25.6	219.05	301.62	density 3	Machine-Learning_Global_TerrorismDatabase(more_than_170000_terrorist_attacks_worldwide_1970-2016).csv
85030355	56.3	1466.40	2573.19	chameleon 2	Machine-Learning_Global_TerrorismDatabase(more_than_170000_terrorist_attacks_worldwide_1970-2016).csv
139891643	92.7	201.44	1755.37	trle	Machine-Learning_Global_TerrorismDatabase(more_than_170000_terrorist_attacks_worldwide_1970-2016).csv

Inhere, my favorite LzTurbo mode, 29, performs over the top.

Crunching 203,288,144 bytes long Machine-Learning_www.kaggle.comexamine-the-examiner(headlines_of_3_million_articles).csv ...

headlines1

TurboBench: - Sat Apr 28 12:29:23 2018

(bold = pareto) MB=1.000.000

C Size	ratio	C MB/s	D MB/s	Name	File
38828239	19.1	0.37	0.36	zpaq 5	Machine-Learning_www.kaggle.comexamine-the-examiner(headlines_of_3_million_articles).csv
41932989	20.6	7.32	34.47	lzturbo 59	Machine-Learning_www.kaggle.comexamine-the-examiner(headlines_of_3_million_articles).csv
41934805	20.6	7.78	50.03	lzturbo 59t2	Machine-Learning_www.kaggle.comexamine-the-examiner(headlines_of_3_million_articles).csv
41937909	20.6	7.89	55.70	lzturbo 59t4	Machine-Learning_www.kaggle.comexamine-the-examiner(headlines_of_3_million_articles).csv
42452918	20.9	19.85	6.13	bsc 6	Machine-Learning_www.kaggle.comexamine-the-examiner(headlines_of_3_million_articles).csv
48343678	23.8	0.74	73.68	lzturbo 49	Machine-Learning_www.kaggle.comexamine-the-examiner(headlines_of_3_million_articles).csv
49249568	24.2	0.83	55.87	lzlib 9d29fb273	Machine-Learning_www.kaggle.comexamine-the-examiner(headlines_of_3_million_articles).csv
49310180	24.3	0.91	72.92	lzma 9d29:fb273:mf=bt4	Machine-Learning_www.kaggle.comexamine-the-examiner(headlines_of_3_million_articles).csv
49447538	24.3	0.32	172.09	lzham 4fb258:x4:d29	Machine-Learning_www.kaggle.comexamine-the-examiner(headlines_of_3_million_articles).csv
49456618	24.3	0.37	403.50	brotli 11d29	Machine-Learning_www.kaggle.comexamine-the-examiner(headlines_of_3_million_articles).csv
49502401	24.4	0.80	172.19	lzham 4	Machine-Learning_www.kaggle.comexamine-the-examiner(headlines_of_3_million_articles).csv
49595585	24.4	0.85	528.84	lzturbo 39	Machine-Learning_www.kaggle.comexamine-the-examiner(headlines_of_3_million_articles).csv
49875692	24.5	1.02	72.31	lzma 9	Machine-Learning_www.kaggle.comexamine-the-examiner(headlines_of_3_million_articles).csv
50192452	24.7	1.05	534.32	zstd 22	Machine-Learning_www.kaggle.comexamine-the-examiner(headlines_of_3_million_articles).csv
51672595	25.4	0.43	484.22	oodle 89	Machine-Learning_www.kaggle.comexamine-the-examiner(headlines_of_3_million_articles).csv
51672595	25.4	0.38	483.98	oodle 129	Machine-Learning_www.kaggle.comexamine-the-examiner(headlines_of_3_million_articles).csv
52942015	26.0	0.23	264.53	oodle 19	Machine-Learning_www.kaggle.comexamine-the-examiner(headlines_of_3_million_articles).csv
54547164	26.8	0.44	301.57	brotli 11	Machine-Learning_www.kaggle.comexamine-the-examiner(headlines_of_3_million_articles).csv
56526268	27.8	20.71	12.50	bsc 3	Machine-Learning_www.kaggle.comexamine-the-examiner(headlines_of_3_million_articles).csv
57991532	28.5	11.69	24.61	bzip2	Machine-Learning_www.kaggle.comexamine-the-examiner(headlines_of_3_million_articles).csv
58004551	28.5	29.49	557.77	lzturbo 32	Machine-Learning_www.kaggle.comexamine-the-examiner(headlines_of_3_million_articles).csv
58404302	28.7	0.88	809.35	lzturbo 29	Machine-Learning_www.kaggle.comexamine-the-examiner(headlines_of_3_million_articles).csv
59998369	29.5	0.46	368.07	xpack 9	Machine-Learning_www.kaggle.comexamine-the-examiner(headlines_of_3_million_articles).csv
61992934	30.5	5.98	811.46	zstd 12	Machine-Learning_www.kaggle.comexamine-the-examiner(headlines_of_3_million_articles).csv
64010305	31.5	3.40	69.33	zpaq 2	Machine-Learning_www.kaggle.comexamine-the-examiner(headlines_of_3_million_articles).csv
64495601	31.7	1.37	675.19	lizard 49	Machine-Learning_www.kaggle.comexamine-the-examiner(headlines_of_3_million_articles).csv
64527683	31.7	19.34	334.10	brotli 5	Machine-Learning_www.kaggle.comexamine-the-examiner(headlines_of_3_million_articles).csv
70609594	34.7	76.85	802.87	zstd 5	Machine-Learning_www.kaggle.comexamine-the-examiner(headlines_of_3_million_articles).csv
75969812	37.4	1.42	1002.67	lizard 29	Machine-Learning_www.kaggle.comexamine-the-examiner(headlines_of_3_million_articles).csv
76227226	37.5	7.03	440.35	libdeflate 12	Machine-Learning_www.kaggle.comexamine-the-examiner(headlines_of_3_million_articles).csv
76289315	37.5	29.12	915.89	lzturbo 22	Machine-Learning_www.kaggle.comexamine-the-examiner(headlines_of_3_million_articles).csv
78834910	38.8	7.15	1447.19	lizard 39	Machine-Learning_www.kaggle.comexamine-the-examiner(headlines_of_3_million_articles).csv
79538326	39.1	62.96	523.05	libdeflate 5	Machine-Learning_www.kaggle.comexamine-the-examiner(headlines_of_3_million_articles).csv
80437351	39.6	0.42	2222.09	oodle 116	Machine-Learning_www.kaggle.comexamine-the-examiner(headlines_of_3_million_articles).csv
80714285	39.7	119.74	271.33	brotli 1	Machine-Learning_www.kaggle.comexamine-the-examiner(headlines_of_3_million_articles).csv
81145151	39.9	0.49	2203.19	oodle 118	Machine-Learning_www.kaggle.comexamine-the-examiner(headlines_of_3_million_articles).csv
82398577	40.5	88.94	552.21	xpack 1	Machine-Learning_www.kaggle.comexamine-the-examiner(headlines_of_3_million_articles).csv
84004494	41.3	113.43	538.65	libdeflate 1	Machine-Learning_www.kaggle.comexamine-the-examiner(headlines_of_3_million_articles).csv
86300419	42.5	229.18	789.33	zstd 1	Machine-Learning_www.kaggle.comexamine-the-examiner(headlines_of_3_million_articles).csv
86821732	42.7	278.33	298.04	density 3	Machine-Learning_www.kaggle.comexamine-the-examiner(headlines_of_3_million_articles).csv
87504030	43.0	18.18	2384.89	oodle 114	Machine-Learning_www.kaggle.comexamine-the-examiner(headlines_of_3_million_articles).csv
89260143	43.9	1.01	2875.81	lzturbo 19	Machine-Learning_www.kaggle.comexamine-the-examiner(headlines_of_3_million_articles).csv
89295977	43.9	7.46	2176.21	lizard 19	Machine-Learning_www.kaggle.comexamine-the-examiner(headlines_of_3_million_articles).csv
89296021	43.9	3.07	1990.82	oodle 49	Machine-Learning_www.kaggle.comexamine-the-examiner(headlines_of_3_million_articles).csv
94755194	46.6	42.87	3029.13	lzturbo 12	Machine-Learning_www.kaggle.comexamine-the-examiner(headlines_of_3_million_articles).csv
101892479	50.1	66.10	2758.84	oodle 112	Machine-Learning_www.kaggle.comexamine-the-examiner(headlines_of_3_million_articles).csv
122584080	60.3	1536.00	2231.24	chameleon 2	Machine-Learning_www.kaggle.comexamine-the-examiner(headlines_of_3_million_articles).csv
202919667	99.8	189.20	2027.89	trle	Machine-Learning_www.kaggle.comexamine-the-examiner(headlines_of_3_million_articles).csv

Failed:

           0     0.0       0.00       0.00   lzsse2 1         Machine-Learning_www.kaggle.com_examine-the-examiner_(headlines_of_3_million_articles).csv
ERROR at 0:70, 96
           0     0.0       0.00       0.00   lzsse2 17        Machine-Learning_www.kaggle.com_examine-the-examiner_(headlines_of_3_million_articles).csv
ERROR at 0:70, 96

Hm, those headlines, brotli 11d29, lzturbo 39 and zstd 22 are not that different decompression-rate-wise.

Okay, the corpus deserves separate post:

Superpig_Corpus.rz 707 MB (741,571,263 bytes), downloadable at: https://drive.google.com/file/d/1WJkur8Gv-gNk9H_nBLMUpnY7_AJkYTrp/view?usp=sharing

zzx006655441ad0

C:\>timer64 rz_1.01.exe a -d 512M Superpig_Corpus.rz Superpig_Corpus.tar

 *** RAZOR Archiver 1.01 (2017-09-14) - DEMO/TEST version ***
 *** (c) Christian Martelock (christian.martelock@web.de) ***

 Scanning c:\superpig_corpus.tar
 Found 0 dirs, 1 files, 4399079936 bytes.

 Creating archive Superpig_Corpus.rz
 Window : 524288K (4096M..1024G)
 Header : 52
 Size   : 741571263

 Archive ok. Added 0 dirs, 1 files, 4399079936 bytes.
 CPU time = 15727.328s / wall time = 12038.339s

Kernel  Time =    32.062 =    0%
User    Time = 15727.328 =  130%
Process Time = 15759.390 =  130%    Virtual  Memory =   6518 MB
Global  Time = 12038.411 =  100%    Physical Memory =   6442 MB

C:\>sha1sum.exe Superpig_Corpus.tar
4f9c15425eb5edb265b1f5bd0241cf0b94d194ef  Superpig_Corpus.tar

C:\>sha1sum.exe Superpig_Corpus.rz
2214292c9e87e546426b5b83d0e44ad501d750b9  Superpig_Corpus.rz

C:\>dir Superpig_Corpus/og/on

04/06/2017  11:44 AM        15,583,440 Arabian_Nights_complete.html
04/06/2017  11:44 AM       174,100,480 Documentation_Composer_XE_2015.tar
04/16/2018  11:04 PM       107,784,192 Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar
06/28/2015  08:54 PM       295,515,893 encyclopediadramaticase-20150628-current.xml
04/26/2018  11:47 AM       314,572,800 enwiki-20170101-pages-articles.xml_first_300MB
11/13/2017  01:08 AM       405,610,647 Machine-Learning_Douban_Movie_Short_Comments_(Chinese).csv
11/13/2017  01:08 AM       150,950,913 Machine-Learning_Global_Terrorism_Database_(more_than_170000_terrorist_attacks_worldwide_1970-2016).csv
11/13/2017  01:08 AM       203,288,144 Machine-Learning_www.kaggle.com_examine-the-examiner_(headlines_of_3_million_articles).csv
11/13/2017  01:08 AM       282,218,054 Machine-Learning_www.kaggle.com_opencorpora-russian_(A_Tagged_1.5_Million_Word_Corpus_of_Russian).txt
04/06/2017  11:44 AM       465,457,152 mingw-w64_x86_64-6.3.0-posix-seh-rt_v5-rev2.tar
04/06/2017  11:44 AM       205,242,368 NASA_Kennedy_Space_Center_WWW_server_access_log_Jul95
04/06/2017  11:44 AM       259,707,904 open_watcom_1.9.0-src.tar
04/24/2018  03:02 AM       346,626,502 RC_2008-08
04/06/2017  11:44 AM       211,938,580 Silesia_compression_corpus
04/06/2017  11:44 AM       325,071,872 Star_Trek_-_737_Ebooks.tar
04/06/2017  11:44 AM       125,234,552 Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl
04/23/2018  01:39 PM       195,587,584 www.holybooks.com_70_PDFs.tar
04/26/2018  12:00 PM       314,572,800 www.ncbi.nlm.nih.gov_Dragonfly_(Ladona_fulva)_whole_genome_shotgun.tar_first_300MB
              18 File(s)  4,399,063,877 bytes

C:\>dir Superpig_Corpus*

04/30/2018  08:04 AM    <DIR>          Superpig_Corpus
04/30/2018  11:32 AM       741,571,263 Superpig_Corpus.rz
04/30/2018  08:08 AM     4,399,079,936 Superpig_Corpus.tar

C:\>

RAZOR with its 512MB window fits perfectly in my laptop 6.8GB free RAM heap, not touching the HDD with virtual RAM.

Finally, the battle of the strongest fast decompressors on my laptop 'Compressionette':

i5-700uuntitled

The roofline/baseline is given by LzTurbo 39, thus instead of 'Speedup' the metric is called 'Speeddown' - how many times the performer is slower.

Note: Oops, just realized that AFAIK Zstd 22 is using 128MB window, should have set it to 512MB as well, to be addressed... wonder what windows Oodle 129 and LzTurbo 39 are using!

open_watcom_1.9.0-src.tar (259,707,904 bytes):

Compressor	Compression Size	Decompression Speeddown
lzma 9d29:fb273:mf=bt4	20839672	1778/191= 9.30x
brotli 11d29	21117099	1778/515= 3.45x
lzham 4fb258:x4:d29	21670652	1778/488= 3.64x
lzturbo 39	21759173	1778/1778= 1.00x
zstd 22	22249772	1778/1271= 1.39x
oodle 129	23001107	1778/1488= 1.19x

Machine-Learning_www.kaggle.comopencorpora-russian(A_Tagged_1.5_Million_Word_Corpus_of_Russian).txt (282,218,054 bytes):

Compressor	Compression Size	Decompression Speeddown
brotli 11d29	11705323	2193/562= 3.90x
lzham 4fb258:x4:d29	12588067	2193/601= 3.64x
lzma 9d29:fb273:mf=bt4	12925754	2193/269= 8.15x
lzturbo 39	13806329	2193/2193= 1.00x
zstd 22	14352030	2193/1270= 1.72x
oodle 129	18327223	2193/1939= 1.13x

Machine-Learning_Douban_Movie_ShortComments(Chinese).csv (405,610,647 bytes):

Compressor	Compression Size	Decompression Speeddown
lzma 9d29:fb273:mf=bt4	85031287	557/81= 6.87x
lzham 4fb258:x4:d29	86692998	557/174= 3.20x
brotli 11d29	88748119	557/227= 2.4x
lzturbo 39	89783640	557/557= 1.00x
oodle 129	91795746	557/498= 1.11x
zstd 22	92414852	557/581= 0.95x

Machine-Learning_www.kaggle.comexamine-the-examiner(headlines_of_3_million_articles).csv (203,288,144 bytes):

Compressor	Compression Size	Decompression Speeddown
lzma 9d29:fb273:mf=bt4	49310180	528/72= 7.33x
lzham 4fb258:x4:d29	49447538	528/172= 3.06x
brotli 11d29	49456618	528/403= 1.31x
lzturbo 39	49595585	528/528= 1.00x
zstd 22	50192452	528/534= 0.98x
oodle 129	51672595	528/483= 1.09x

Machine-Learning_Global_TerrorismDatabase(more_than_170000_terrorist_attacks_worldwide_1970-2016).csv (150,950,913 bytes):

Compressor	Compression Size	Decompression Speeddown
lzma 9d29:fb273:mf=bt4	12968621	1492/188= 7.93x
brotli 11d29	13085028	1492/428= 3.48x
lzham 4fb258:x4:d29	13102376	1492/593= 2.51x
lzturbo 39	13289188	1492/1492= 1.00x
zstd 22	13671840	1492/1201= 1.24x
oodle 129	14775481	1492/1188= 1.25x

www.ncbi.nlm.nih.govDragonfly(Ladona_fulva)_whole_genome_shotgun.tar_first_300MB (314,572,800 bytes):

Compressor	Compression Size	Decompression Speeddown
lzturbo 39	76363252	1090/1090= 1.00x
lzma 9d29:fb273:mf=bt4	79106644	1090/70= 15.57x
brotli 11d29	79847082	1090/119= 9.15x
lzham 4fb258:x4:d29	79952387	1090/154= 7.07x
zstd 22	81447524	1090/504= 2.16x
oodle 129	85499438	1090/1156= 0.94x

enwiki-20170101-pages-articles.xml_first_300MB (314,572,800 bytes):

Compressor	Compression Size	Decompression Speeddown
lzma 9d29:fb273:mf=bt4	72205998	572/75= 7.62x
brotli 11d29	72439347	572/215= 2.66x
lzham 4fb258:x4:d29	72824487	572/176= 3.25x
lzturbo 39	73736920	572/572= 1.00x
zstd 22	75554555	572/554= 1.03x
oodle 129	76107283	572/506= 1.13x

StarTrek-_737_Ebooks.tar (325,071,872 bytes):

Compressor	Compression Size	Decompression Speeddown
lzma 9d29:fb273:mf=bt4	71943246	564/83= 6.79x
brotli 11d29	71952589	564/232= 2.43x
lzham 4fb258:x4:d29	72229583	564/186= 3.03x
lzturbo 39	72346887	564/564= 1.00x
zstd 22	74998925	564/569= 0.99x
oodle 129	75312445	564/491= 1.14x

encyclopediadramaticase-20150628-current.xml (295,515,893 bytes):

Compressor	Compression Size	Decompression Speeddown
lzma 9d29:fb273:mf=bt4	48768664	770/103= 7.47x
brotli 11d29	49285547	770/270= 2.85x
lzham 4fb258:x4:d29	49356398	770/242= 3.18x
lzturbo 39	50392272	770/770= 1.00x
zstd 22	51712719	770/728= 1.05x
oodle 129	52145724	770/676= 1.13x

NASA_Kennedy_Space_Center_WWW_server_access_log_Jul95 (205,242,368 bytes):

Compressor	Compression Size	Decompression Speeddown
lzma 9d29:fb273:mf=bt4	11883945	1814/251= 7.22x
lzturbo 39	11901404	1814/1814= 1.00x
lzham 4fb258:x4:d29	11922873	1814/701= 2.58x
brotli 11d29	12236072	1814/566= 3.20x
zstd 22	12639080	1814/1385= 1.30x
oodle 129	14604989	1814/1125= 1.61x

RC_2008-08 (346,626,502 bytes):

Compressor	Compression Size	Decompression Speeddown
lzma 9d29:fb273:mf=bt4	48680881	850/119= 7.14x
lzham 4fb258:x4:d29	48791397	850/279= 3.04x
brotli 11d29	48849144	850/306= 2.77x
lzturbo 39	49536033	850/850= 1.00x
zstd 22	50900116	850/780= 1.08x
oodle 129	52767834	850/1037= 0.81x

www.holybooks.com_70_PDFs.tar (195,587,584 bytes):

Compressor	Compression Size	Decompression Speeddown
brotli 11d29	153693341	3312/146= 22.68x
lzturbo 39	154390722	3312/3312= 1.00x
lzham 4fb258:x4:d29	154478896	3312/178= 18.60x
oodle 129	154632800	3312/3730= 0.88x
lzma 9d29:fb273:mf=bt4	155014084	3312/21= 157.71x
zstd 22	157307263	3312/2646= 1.25x

EncyclopaediaJudaica(in_22_volumes)_TXT.tar (107,784,192 bytes):

Compressor	Compression Size	Decompression Speeddown
lzma 9d29:fb273:mf=bt4	27526894	509/70= 7.27x
brotli 11d29	27600942	509/215= 2.36x
lzham 4fb258:x4:d29	27750101	509/166= 3.06x
lzturbo 39	27942261	509/509= 1.00x
zstd 22	28041924	509/522= 0.97x
oodle 129	28654421	509/479= 1.06x

Webster's_Third_New_InternationalUnabridged(1961)_(En-En).dsl (125,234,552 bytes):

Compressor	Compression Size	Decompression Speeddown
lzma 9d29:fb273:mf=bt4	19519434	856/111= 7.71x
brotli 11d29	19559693	856/313= 2.73x
lzham 4fb258:x4:d29	19609082	856/267= 3.20x
lzturbo 39	19650726	856/856= 1.00x
zstd 22	19975078	856/704= 1.21x
oodle 129	20861759	856/673= 1.27x

Documentation_Composer_XE_2015.tar (174,100,480 bytes):

Compressor	Compression Size	Decompression Speeddown
brotli 11d29	30186666	2087/339= 6.15x
lzma 9d29:fb273:mf=bt4	30220005	2087/95= 21.96x
lzham 4fb258:x4:d29	30864067	2087/415= 5.02x
lzturbo 39	30882973	2087/2087= 1.00x
zstd 22	31026330	2087/1562= 1.33x
oodle 129	31759169	2087/1676= 1.24x

Silesia_compression_corpus (211,938,580 bytes):

Compressor	Compression Size	Decompression Speeddown
lzma 9d29:fb273:mf=bt4	48382888	1055/71= 14.85x
brotli 11d29	49539504	1055/408= 2.58x
lzham 4fb258:x4:d29	50124186	1055/204= 5.17x
oodle 129	51381492	1055/1047= 1.00x
lzturbo 39	51676247	1055/1055= 1.00x
zstd 22	52831997	1055/846= 1.24x

mingw-w64_x86_64-6.3.0-posix-seh-rt_v5-rev2.tar (465,457,152 bytes):

Compressor	Compression Size	Decompression Speeddown
lzma 9d29:fb273:mf=bt4	42457543	1804/163= 11.06x
brotli 11d29	42988097	1804/439= 4.10x
lzham 4fb258:x4:d29	44372549	1804/432= 4.17x
oodle 129	45218184	1804/1635= 1.10x
lzturbo 39	45874938	1804/1804= 1.00x
zstd 22	51380197	1804/1212= 1.48x

To see how better the 29bit Zstd built-in (--wlog) and the LDM/deduplicator (--long) are when compared to the vanilla 22 mode I ran:

C:\Superpig_Corpus>zstd-v1.3.4-win64.exe -22 --ultra --long=29 %1
C:\Superpig_Corpus>zstd-v1.3.4-win64.exe -22 --ultra --zstd=wlog=31,clog=30,hlog=30,slog=26 %1

The first one delivered these:

 31,301,849 Documentation_Composer_XE_2015.tar.zst
 28,044,315 Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.zst
 51,662,186 encyclopediadramaticase-20150628-current.xml.zst
 75,554,382 enwiki-20170101-pages-articles.xml_first_300MB.zst
 92,328,026 Machine-Learning_Douban_Movie_Short_Comments_(Chinese).csv.zst
 13,885,235 Machine-Learning_Global_Terrorism_Database_(more_than_170000_terrorist_attacks_worldwide_1970-2016).csv.zst
 50,191,563 Machine-Learning_www.kaggle.com_examine-the-examiner_(headlines_of_3_million_articles).csv.zst
 14,402,612 Machine-Learning_www.kaggle.com_opencorpora-russian_(A_Tagged_1.5_Million_Word_Corpus_of_Russian).txt.zst
 46,885,572 mingw-w64_x86_64-6.3.0-posix-seh-rt_v5-rev2.tar.zst
 12,636,397 NASA_Kennedy_Space_Center_WWW_server_access_log_Jul95.zst
 22,486,164 open_watcom_1.9.0-src.tar.zst
 50,937,336 RC_2008-08.zst
 52,998,135 Silesia_compression_corpus.zst
 74,589,495 Star_Trek_-_737_Ebooks.tar.zst
 19,975,078 Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.zst
154,786,855 www.holybooks.com_70_PDFs.tar.zst
 81,443,764 www.ncbi.nlm.nih.gov_Dragonfly_(Ladona_fulva)_whole_genome_shotgun.tar_first_300MB.zst

The second one delivered these:

 30,997,654 Documentation_Composer_XE_2015.tar.zst
 28,008,708 Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.zst
 50,781,657 encyclopediadramaticase-20150628-current.xml.zst
 74,086,538 enwiki-20170101-pages-articles.xml_first_300MB.zst
 ??,???,??? Machine-Learning_Douban_Movie_Short_Comments_(Chinese).csv.zst
 13,659,294 Machine-Learning_Global_Terrorism_Database_(more_than_170000_terrorist_attacks_worldwide_1970-2016).csv.zst
 49,905,102 Machine-Learning_www.kaggle.com_examine-the-examiner_(headlines_of_3_million_articles).csv.zst
 14,342,327 Machine-Learning_www.kaggle.com_opencorpora-russian_(A_Tagged_1.5_Million_Word_Corpus_of_Russian).txt.zst
 46,460,381 mingw-w64_x86_64-6.3.0-posix-seh-rt_v5-rev2.tar.zst
 12,615,526 NASA_Kennedy_Space_Center_WWW_server_access_log_Jul95.zst
 21,972,614 open_watcom_1.9.0-src.tar.zst
 50,286,423 RC_2008-08.zst
 52,800,057 Silesia_compression_corpus.zst
 72,789,313 Star_Trek_-_737_Ebooks.tar.zst
 19,941,582 Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.zst
154,805,473 www.holybooks.com_70_PDFs.tar.zst
 81,081,155 www.ncbi.nlm.nih.gov_Dragonfly_(Ladona_fulva)_whole_genome_shotgun.tar_first_300MB.zst

For 'Douban' I had to stop the HDD thrashing, it needs more than 6.8GB RAM. The Large Window (128+MB) delivers better ratio, most noticeably for 'StarTrek-_737_Ebooks.tar', almost 2MB gain.

With this I close benchmarking the 'Superpig' testsets, hope it can be instrumental. Also, closing the thread, kinda disappointed of lack of binary capable to work with 29bit and report speeds, whatever, it is what it is, as Diaz bros. say.

Reopened, in order to allow feedback. From my side I shared what I wanted. @Brotliteam Feel free to close it whenever you see fit.

To me, the superswinish benchmark reveals some hidden facets of all the supercode zzx3532

google / brotli

Binary needed #642

TurboBench: - Thu Apr 26 11:44:32 2018

TurboBench: - Thu Apr 26 23:11:13 2018

TurboBench: - Thu Apr 26 11:44:08 2018

Dramatica:

Wikipedia:

Star_Trek:

TurboBench: - Fri Apr 27 21:51:45 2018

TurboBench: - Sun Apr 29 17:12:27 2018

TurboBench: - Mon Apr 30 07:51:32 2018

TurboBench: - Sun Apr 29 08:33:59 2018

TurboBench: - Sat Apr 28 07:35:55 2018

TurboBench: - Sat Apr 28 12:29:23 2018

open_watcom_1.9.0-src.tar (259,707,904 bytes):

Machine-Learning_www.kaggle.comopencorpora-russian(A_Tagged_1.5_Million_Word_Corpus_of_Russian).txt (282,218,054 bytes):

Machine-Learning_Douban_Movie_ShortComments(Chinese).csv (405,610,647 bytes):

Machine-Learning_www.kaggle.comexamine-the-examiner(headlines_of_3_million_articles).csv (203,288,144 bytes):

Machine-Learning_Global_TerrorismDatabase(more_than_170000_terrorist_attacks_worldwide_1970-2016).csv (150,950,913 bytes):

www.ncbi.nlm.nih.govDragonfly(Ladona_fulva)_whole_genome_shotgun.tar_first_300MB (314,572,800 bytes):

enwiki-20170101-pages-articles.xml_first_300MB (314,572,800 bytes):

StarTrek-_737_Ebooks.tar (325,071,872 bytes):

encyclopediadramaticase-20150628-current.xml (295,515,893 bytes):

NASA_Kennedy_Space_Center_WWW_server_access_log_Jul95 (205,242,368 bytes):

RC_2008-08 (346,626,502 bytes):

www.holybooks.com_70_PDFs.tar (195,587,584 bytes):

EncyclopaediaJudaica(in_22_volumes)_TXT.tar (107,784,192 bytes):

Webster's_Third_New_InternationalUnabridged(1961)_(En-En).dsl (125,234,552 bytes):

Documentation_Composer_XE_2015.tar (174,100,480 bytes):

Silesia_compression_corpus (211,938,580 bytes):

mingw-w64_x86_64-6.3.0-posix-seh-rt_v5-rev2.tar (465,457,152 bytes):