search

google / brotli

Brotli compression format
MIT License
13.36k stars 1.23k forks source link

"failed to determine uncompressed file size" #327

Closed Sanmayce closed 8 years ago

Sanmayce commented 8 years ago

Hi, wanted to see how your Brotli fares again with big files, it turns out an issue occurred:

D:\z>_BROTLIFY.BAT

D:\z>dir *.*/b/a-d 1>DIRLIST

D:\z>FOR /F %G IN (DIRLIST) DO bro_Feb-10-2016_mingw64_GCC_5.3.0 -i %G -o %G.L11_W24.brotli -v --quality 11 --window 24

D:\z>bro_Feb-10-2016_mingw64_GCC_5.3.0 -i book_serie_RUSSIAN_BESTSELLER_885_novels_(Russian).tar -o book_serie_RUSSIAN_BESTSELLER_885_novels_(Russian).tar.L11_W24.brotli -v --quality 11 --window 24
Brotli compression speed: 0.260594 MB/s

D:\z>bro_Feb-10-2016_mingw64_GCC_5.3.0 -i book_serie_SPETSNAZ_803_novels_(Russian).tar -o book_serie_SPETSNAZ_803_novels_(Russian).tar.L11_W24.brotli -v --quality 11 --window 24
Brotli compression speed: 0.262719 MB/s

D:\z>bro_Feb-10-2016_mingw64_GCC_5.3.0 -i chitanka.info.epub.name-2015.12-cumulative_(36046_books)_(Bulgarian)_UTF-8.tar -o chitanka.info.epub.name-2015.12-cumulative_(36046_books)_(Bulgarian)_UTF-8.tar.L11_W24.brotli -v --quality 11 --window 24
failed to determine uncompressed file size

D:\z>bro_Feb-10-2016_mingw64_GCC_5.3.0 -i enwiktionary-20160111-pages-articles.xml -o enwiktionary-20160111-pages-articles.xml.L11_W24.brotli -v --quality 11 --window 24
failed to determine uncompressed file size

D:\z>bro_Feb-10-2016_mingw64_GCC_5.3.0 -i GoldenDict_DSLs_(En-Ru)_plus_(Ru-En).tar -o GoldenDict_DSLs_(En-Ru)_plus_(Ru-En).tar.L11_W24.brotli -v --quality 11 --window 24
Brotli compression speed: 0.342522 MB/s

D:\z>bro_Feb-10-2016_mingw64_GCC_5.3.0 -i https_wikileaks.org_War_Diary_-_Afghanistan_War_Logs_afg.tar -o https_wikileaks.org_War_Diary_-_Afghanistan_War_Logs_afg.tar.L11_W24.brotli -v --quality 11 --window 24
failed to determine uncompressed file size

D:\z>bro_Feb-10-2016_mingw64_GCC_5.3.0 -i Sacred_Texts_7_(97830_htm_files).tar -o Sacred_Texts_7_(97830_htm_files).tar.L11_W24.brotli -v --quality 11 --window 24
Brotli compression speed: 0.323023 MB/s

D:\z>bro_Feb-10-2016_mingw64_GCC_5.3.0 -i vxheaven.org.tar -o vxheaven.org.tar.L11_W24.brotli -v --quality 11 --window 24
Brotli compression speed: 0.0560793 MB/s

D:\z>bro_Feb-10-2016_mingw64_GCC_5.3.0 -i _Deathship_textual_corpus.tar -o _Deathship_textual_corpus.tar.L11_W24.brotli -v --quality 11 --window 24
failed to determine uncompressed file size

D:\z>bro_Feb-10-2016_mingw64_GCC_5.3.0 -i _dict2txt_En_Fr_Ru_Ge_corpus.tar -o _dict2txt_En_Fr_Ru_Ge_corpus.tar.L11_W24.brotli -v --quality 11 --window 24
failed to determine uncompressed file size

D:\z>bro_Feb-10-2016_mingw64_GCC_5.3.0 -i _DSL_(Dictionary_Specification_Language_(ABBYY_Software_House))_corpus.tar -o _DSL_(Dictionary_Specification_Language_(ABBYY_Software_House))_corpus.tar.L11_W24.brotli -v --quality 11 --window 24
failed to determine uncompressed file size

The current side-by-side dump:

03/05/2016  10:34 PM       698,795,520 book_serie_RUSSIAN_BESTSELLER_885_novels_(Russian).tar
03/05/2016  11:18 PM       141,549,516 book_serie_RUSSIAN_BESTSELLER_885_novels_(Russian).tar.method28.zpaq
03/06/2016  08:25 AM        99,143,807 book_serie_RUSSIAN_BESTSELLER_885_novels_(Russian).tar.method58.zpaq
03/08/2016  04:34 AM       118,648,815 book_serie_RUSSIAN_BESTSELLER_885_novels_(Russian).tar.L21.zst
03/05/2016  12:00 PM       119,531,414 book_serie_RUSSIAN_BESTSELLER_885_novels_(Russian).tar.ST6Block256.bsc
03/08/2016  05:39 AM       124,293,485 book_serie_RUSSIAN_BESTSELLER_885_novels_(Russian).tar.L11_W24.brotli

03/05/2016  10:35 PM       681,979,392 book_serie_SPETSNAZ_803_novels_(Russian).tar
03/05/2016  11:22 PM       124,286,661 book_serie_SPETSNAZ_803_novels_(Russian).tar.method28.zpaq
03/06/2016  08:58 AM        87,398,226 book_serie_SPETSNAZ_803_novels_(Russian).tar.method58.zpaq
03/08/2016  04:46 AM       111,915,817 book_serie_SPETSNAZ_803_novels_(Russian).tar.L21.zst
03/05/2016  12:00 PM       115,613,238 book_serie_SPETSNAZ_803_novels_(Russian).tar.ST6Block256.bsc
03/08/2016  06:20 AM       119,835,493 book_serie_SPETSNAZ_803_novels_(Russian).tar.L11_W24.brotli

03/05/2016  10:42 PM     6,923,727,360 chitanka.info.epub.name-2015.12-cumulative_(36046_books)_(Bulgarian)_UTF-8.tar
03/06/2016  12:07 AM     1,317,366,597 chitanka.info.epub.name-2015.12-cumulative_(36046_books)_(Bulgarian)_UTF-8.tar.method28.zpaq
03/06/2016  03:09 PM       913,500,648 chitanka.info.epub.name-2015.12-cumulative_(36046_books)_(Bulgarian)_UTF-8.tar.method58.zpaq
03/08/2016  06:56 AM     1,090,603,355 chitanka.info.epub.name-2015.12-cumulative_(36046_books)_(Bulgarian)_UTF-8.tar.L21.zst
03/05/2016  12:00 PM     1,093,309,220 chitanka.info.epub.name-2015.12-cumulative_(36046_books)_(Bulgarian)_UTF-8.tar.ST6Block256.bsc
03/08/2016  01:18 PM     1,128,714,958 chitanka.info.epub.name-2015.12-cumulative_(36046_books)_(Bulgarian)_UTF-8.tar.L11_W24.brotli

03/05/2016  10:44 PM     4,196,304,236 enwiktionary-20160111-pages-articles.xml
03/06/2016  12:26 AM       607,352,607 enwiktionary-20160111-pages-articles.xml.method28.zpaq
03/06/2016  06:52 PM       317,020,308 enwiktionary-20160111-pages-articles.xml.method58.zpaq
03/08/2016  07:51 AM       472,855,545 enwiktionary-20160111-pages-articles.xml.L21.zst
03/05/2016  12:00 PM       372,389,584 enwiktionary-20160111-pages-articles.xml.ST6Block256.bsc
03/08/2016  04:56 PM       444,141,352 enwiktionary-20160111-pages-articles.xml.L11_W24.brotli

03/05/2016  10:45 PM     1,627,090,432 GoldenDict_DSLs_(En-Ru)_plus_(Ru-En).tar
03/06/2016  12:34 AM       279,087,696 GoldenDict_DSLs_(En-Ru)_plus_(Ru-En).tar.method28.zpaq
03/06/2016  08:26 PM       161,557,969 GoldenDict_DSLs_(En-Ru)_plus_(Ru-En).tar.method58.zpaq
03/08/2016  08:18 AM       238,122,680 GoldenDict_DSLs_(En-Ru)_plus_(Ru-En).tar.L21.zst
03/05/2016  12:00 PM       204,276,176 GoldenDict_DSLs_(En-Ru)_plus_(Ru-En).tar.ST6Block256.bsc
03/08/2016  06:11 PM       243,414,326 GoldenDict_DSLs_(En-Ru)_plus_(Ru-En).tar.L11_W24.brotli

03/05/2016  10:46 PM     3,824,270,848 https_wikileaks.org_War_Diary_-_Afghanistan_War_Logs_afg.tar
03/06/2016  12:46 AM        80,457,217 https_wikileaks.org_War_Diary_-_Afghanistan_War_Logs_afg.tar.method28.zpaq
03/07/2016  12:19 AM        52,840,005 https_wikileaks.org_War_Diary_-_Afghanistan_War_Logs_afg.tar.method58.zpaq
03/08/2016  08:52 AM        83,879,310 https_wikileaks.org_War_Diary_-_Afghanistan_War_Logs_afg.tar.L21.zst
03/05/2016  12:00 PM        64,327,890 https_wikileaks.org_War_Diary_-_Afghanistan_War_Logs_afg.tar.ST6Block256.bsc
03/08/2016  07:01 PM        80,834,479 https_wikileaks.org_War_Diary_-_Afghanistan_War_Logs_afg.tar.L11_W24.brotli

03/05/2016  10:47 PM     1,198,966,784 Sacred_Texts_7_(97830_htm_files).tar
03/06/2016  12:53 AM       253,738,906 Sacred_Texts_7_(97830_htm_files).tar.method28.zpaq
03/07/2016  01:36 AM       151,113,394 Sacred_Texts_7_(97830_htm_files).tar.method58.zpaq
03/08/2016  09:20 AM       213,134,910 Sacred_Texts_7_(97830_htm_files).tar.L21.zst
03/05/2016  12:00 PM       176,729,590 Sacred_Texts_7_(97830_htm_files).tar.ST6Block256.bsc
03/08/2016  08:00 PM       215,661,987 Sacred_Texts_7_(97830_htm_files).tar.L11_W24.brotli

03/05/2016  10:47 PM     1,428,999,168 vxheaven.org.tar
03/06/2016  12:55 AM        76,660,909 vxheaven.org.tar.method28.zpaq
03/07/2016  02:25 AM        64,309,018 vxheaven.org.tar.method58.zpaq
03/08/2016  09:23 AM        71,138,408 vxheaven.org.tar.L21.zst
03/05/2016  12:00 PM        77,156,314 vxheaven.org.tar.ST6Block256.bsc
03/09/2016  02:45 AM        70,181,174 vxheaven.org.tar.L11_W24.brotli

03/05/2016  10:55 PM     8,000,929,280 _Deathship_textual_corpus.tar
03/06/2016  01:43 AM     2,132,027,578 _Deathship_textual_corpus.tar.method28.zpaq
03/07/2016  11:03 AM     1,433,135,510 _Deathship_textual_corpus.tar.method58.zpaq
03/08/2016  11:42 AM     1,849,442,370 _Deathship_textual_corpus.tar.L21.zst
03/05/2016  12:00 PM     1,684,445,506 _Deathship_textual_corpus.tar.ST6Block256.bsc
03/09/2016  09:44 AM     1,882,729,861 _Deathship_textual_corpus.tar.L11_W24.brotli

03/05/2016  10:58 PM     4,906,654,208 _dict2txt_En_Fr_Ru_Ge_corpus.tar
03/06/2016  02:07 AM       633,428,268 _dict2txt_En_Fr_Ru_Ge_corpus.tar.method28.zpaq
03/07/2016  05:00 PM       375,300,415 _dict2txt_En_Fr_Ru_Ge_corpus.tar.method58.zpaq
03/08/2016  12:55 PM       552,303,795 _dict2txt_En_Fr_Ru_Ge_corpus.tar.L21.zst
03/05/2016  12:00 PM       448,139,006 _dict2txt_En_Fr_Ru_Ge_corpus.tar.ST6Block256.bsc
03/09/2016  01:11 PM       564,144,237 _dict2txt_En_Fr_Ru_Ge_corpus.tar.L11_W24.brotli

03/05/2016  11:04 PM     7,717,431,808 _DSL_(Dictionary_Specification_Language_(ABBYY_Software_House))_corpus.tar
03/06/2016  02:55 AM     1,367,007,877 _DSL_(Dictionary_Specification_Language_(ABBYY_Software_House))_corpus.tar.method28.zpaq
03/08/2016  12:09 AM       865,834,590 _DSL_(Dictionary_Specification_Language_(ABBYY_Software_House))_corpus.tar.method58.zpaq
03/08/2016  03:06 PM     1,171,777,463 _DSL_(Dictionary_Specification_Language_(ABBYY_Software_House))_corpus.tar.L21.zst
03/05/2016  12:00 PM     1,016,428,006 _DSL_(Dictionary_Specification_Language_(ABBYY_Software_House))_corpus.tar.ST6Block256.bsc
03/09/2016  07:35 PM     1,207,816,970 _DSL_(Dictionary_Specification_Language_(ABBYY_Software_House))_corpus.tar.L11_W24.brotli

D:\z>

The superstrong BSC despite using 256MB blocks, surprisingly, is not that strong when compared to Zstd, as for decompression speed, BSC can't stand a chance. If you fix that issue I will attempt a new run. By the way, I find no executable within release section, why so?

jyrkialakuijala commented 8 years ago

Would you be willing repeat the test so that every algorithm is limited to roughly 16 MB of decoding context?

Sanmayce commented 8 years ago

Would you be willing repeat the test so that every algorithm is limited to roughly 16 MB of decoding context?

No problema, the idea is this torture to present a clearer picture of mainstream textual [de]compression, yet my desire is to go beyond the funless 4GB of RAM and see what future holds. With higher densities of DDR4 sticks we will have soon 4x32GB in desktop PCs, what about using 1GB blocks as well?!

But first, I need a new compile, not complaining about decompression size.

Also, my intention is to add:

XZ uses 64MB, by default, in its max level i.e. 9.

Can't wait to see also the Planet-Spitting Performance of LZSSE2 level 16, have to wait for its console version... yet, thanks to Hamid's TurboBench I made one quite interesting benchmark of 80 DSLs (dictionaries with minimalistic html-like tagging), the very same that form the:

03/05/2016  11:04 PM     7,717,431,808 _DSL_(Dictionary_Specification_Language_(ABBYY_Software_House))_corpus.tar

For the full 80 testdatafiles log: http://www.sanmayce.com/Downloads/log-Laptop-Toshiba_DSLs.txt

Just one, the Merriam brothers' 4 inches thick one with 2,662 pages defining 450,000+ words:

D:\zzz\_DSL_(Dictionary_Specification_Language_(ABBYY_Software_House))_corpus>turbobenchs.exe Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl -ezlib,9/fastlz,2/lzturbo,19,29,39/bzip2/chameleon,2/snappy_c/zstd,1,20,21/density,3/lz4,16/lz5,15/lzham,4/brieflz/brotli,11/crush,2/lzma,9/zpaq,2/lzf/yappy/trle/memcpy/lzsse2,1,16 -g -k0 -B2G
...
TurboBench:  - Sat Feb 27 03:21:01 2016

      C Size  ratio%     C MB/s     D MB/s   Name            File
    19519434    15.6       0.87     100.14   lzma 9          Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    19736547    15.8       0.68     475.76   lzturbo 39      Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    20004388    16.0       0.77     199.20   lzham 4         Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    20200910    16.1       1.11     352.36   zstd 21         Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    20364153    16.3       1.16     355.51   zstd 20         Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    20459680    16.3       0.34     290.88   brotli 11       Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    21473387    17.1       8.82      25.94   bzip2           Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    24798599    19.8       0.71     713.40   lzturbo 29      Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    25518771    20.4       3.94      73.30   zpaq 2          Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    25814446    20.6       0.24     312.29   crush 2         Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    27405010    21.9       1.46     571.12   lz5 15          Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    30559193    24.4      19.07     274.88   zlib 9          Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    31648561    25.3       4.96    2333.98   lzsse2 16       Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    34835640    27.8       0.81    2079.93   lzturbo 19      Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    34964699    27.9     178.92     483.54   zstd 1          Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    35208058    28.1      21.57    1758.15   lz4 16          Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    39035549    31.2      95.29     185.53   brieflz         Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    39274452    31.4     255.07     237.49   density 3       Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    39417344    31.5      11.19    2035.24   lzsse2 1        Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    47927011    38.3      70.62    1620.57   yappy           Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    50696387    40.5     230.31     366.75   fastlz 2        Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    50980477    40.7     232.57     471.36   lzf             Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    51775589    41.3     313.17     841.53   snappy_c        Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
    70877719    56.6    1045.33    2090.83   chameleon 2     Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
   125223987   100.0     134.64    1312.37   trle            Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb
   125234556   100.0    6289.09    5577.38   memcpy          Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.tbb

Just look what LZSSE2 is doing, I AM SHOCKED! Where is the law? 64KB vs 4MB vs 16MB vs 256MB vs 1GB...

Guru Josh - Whose Law (Is It Anyway) https://www.youtube.com/watch?v=x7ytf5T1fY8

“I’m living in a future. A future of law. I want to know. Whose law can it be? A future for you and I. I’m living in a world. In a world of law. Whose law can it be?”

Paraphrasing the awesome Guru's question, whose dictionary length can it be?

Sanmayce commented 8 years ago

Should I had had more computational power I would have been exploring deeply, anyway:

03/10/2016  08:51 AM       179,736,048 book_serie_RUSSIAN_BESTSELLER_885_novels_(Russian).tar.15_256MB.lz5
03/09/2016  08:45 PM       114,245,441 book_serie_RUSSIAN_BESTSELLER_885_novels_(Russian).tar.256MB.lzturbo12-39.lzt
03/08/2016  04:34 AM       118,648,815 book_serie_RUSSIAN_BESTSELLER_885_novels_(Russian).tar.L21.zst
03/05/2016  11:18 PM       141,549,516 book_serie_RUSSIAN_BESTSELLER_885_novels_(Russian).tar.method28.zpaq
03/06/2016  08:25 AM        99,143,807 book_serie_RUSSIAN_BESTSELLER_885_novels_(Russian).tar.method58.zpaq
03/10/2016  09:28 PM       122,193,254 book_serie_RUSSIAN_BESTSELLER_885_novels_(Russian).tar.ST6Block16.bsc
03/05/2016  12:00 PM       119,531,414 book_serie_RUSSIAN_BESTSELLER_885_novels_(Russian).tar.ST6Block256.bsc

03/10/2016  09:00 AM       173,702,643 book_serie_SPETSNAZ_803_novels_(Russian).tar.15_256MB.lz5
03/09/2016  09:05 PM       105,739,516 book_serie_SPETSNAZ_803_novels_(Russian).tar.256MB.lzturbo12-39.lzt
03/08/2016  04:46 AM       111,915,817 book_serie_SPETSNAZ_803_novels_(Russian).tar.L21.zst
03/05/2016  11:22 PM       124,286,661 book_serie_SPETSNAZ_803_novels_(Russian).tar.method28.zpaq
03/06/2016  08:58 AM        87,398,226 book_serie_SPETSNAZ_803_novels_(Russian).tar.method58.zpaq
03/10/2016  09:28 PM       119,176,438 book_serie_SPETSNAZ_803_novels_(Russian).tar.ST6Block16.bsc
03/05/2016  12:00 PM       115,613,238 book_serie_SPETSNAZ_803_novels_(Russian).tar.ST6Block256.bsc

03/10/2016  10:43 AM     1,623,198,054 chitanka.info.epub.name-2015.12-cumulative_(36046_books)_(Bulgarian)_UTF-8.tar.15_256MB.lz5
03/10/2016  12:55 AM     1,052,429,918 chitanka.info.epub.name-2015.12-cumulative_(36046_books)_(Bulgarian)_UTF-8.tar.256MB.lzturbo12-39.lzt
03/08/2016  06:56 AM     1,090,603,355 chitanka.info.epub.name-2015.12-cumulative_(36046_books)_(Bulgarian)_UTF-8.tar.L21.zst
03/06/2016  12:07 AM     1,317,366,597 chitanka.info.epub.name-2015.12-cumulative_(36046_books)_(Bulgarian)_UTF-8.tar.method28.zpaq
03/06/2016  03:09 PM       913,500,648 chitanka.info.epub.name-2015.12-cumulative_(36046_books)_(Bulgarian)_UTF-8.tar.method58.zpaq
03/10/2016  09:36 PM     1,116,793,936 chitanka.info.epub.name-2015.12-cumulative_(36046_books)_(Bulgarian)_UTF-8.tar.ST6Block16.bsc
03/05/2016  12:00 PM     1,093,309,220 chitanka.info.epub.name-2015.12-cumulative_(36046_books)_(Bulgarian)_UTF-8.tar.ST6Block256.bsc

03/10/2016  11:50 AM       639,156,091 enwiktionary-20160111-pages-articles.xml.15_256MB.lz5
03/10/2016  02:51 AM       446,433,293 enwiktionary-20160111-pages-articles.xml.256MB.lzturbo12-39.lzt
03/08/2016  07:51 AM       472,855,545 enwiktionary-20160111-pages-articles.xml.L21.zst
03/06/2016  12:26 AM       607,352,607 enwiktionary-20160111-pages-articles.xml.method28.zpaq
03/06/2016  06:52 PM       317,020,308 enwiktionary-20160111-pages-articles.xml.method58.zpaq
03/10/2016  09:40 PM       405,663,946 enwiktionary-20160111-pages-articles.xml.ST6Block16.bsc
03/05/2016  12:00 PM       372,389,584 enwiktionary-20160111-pages-articles.xml.ST6Block256.bsc

03/10/2016  12:11 PM       334,005,334 GoldenDict_DSLs_(En-Ru)_plus_(Ru-En).tar.15_256MB.lz5
03/10/2016  03:33 AM       229,328,828 GoldenDict_DSLs_(En-Ru)_plus_(Ru-En).tar.256MB.lzturbo12-39.lzt
03/08/2016  08:18 AM       238,122,680 GoldenDict_DSLs_(En-Ru)_plus_(Ru-En).tar.L21.zst
03/06/2016  12:34 AM       279,087,696 GoldenDict_DSLs_(En-Ru)_plus_(Ru-En).tar.method28.zpaq
03/06/2016  08:26 PM       161,557,969 GoldenDict_DSLs_(En-Ru)_plus_(Ru-En).tar.method58.zpaq
03/10/2016  09:41 PM       213,021,272 GoldenDict_DSLs_(En-Ru)_plus_(Ru-En).tar.ST6Block16.bsc
03/05/2016  12:00 PM       204,276,176 GoldenDict_DSLs_(En-Ru)_plus_(Ru-En).tar.ST6Block256.bsc

03/10/2016  01:22 PM       115,283,934 https_wikileaks.org_War_Diary_-_Afghanistan_War_Logs_afg.tar.15_256MB.lz5
03/10/2016  03:46 AM        68,846,521 https_wikileaks.org_War_Diary_-_Afghanistan_War_Logs_afg.tar.256MB.lzturbo12-39.lzt
03/08/2016  08:52 AM        83,879,310 https_wikileaks.org_War_Diary_-_Afghanistan_War_Logs_afg.tar.L21.zst
03/06/2016  12:46 AM        80,457,217 https_wikileaks.org_War_Diary_-_Afghanistan_War_Logs_afg.tar.method28.zpaq
03/07/2016  12:19 AM        52,840,005 https_wikileaks.org_War_Diary_-_Afghanistan_War_Logs_afg.tar.method58.zpaq
03/10/2016  09:42 PM        78,904,402 https_wikileaks.org_War_Diary_-_Afghanistan_War_Logs_afg.tar.ST6Block16.bsc
03/05/2016  12:00 PM        64,327,890 https_wikileaks.org_War_Diary_-_Afghanistan_War_Logs_afg.tar.ST6Block256.bsc

03/10/2016  01:57 PM       291,804,723 Sacred_Texts_7_(97830_htm_files).tar.15_256MB.lz5
03/10/2016  04:17 AM       208,014,989 Sacred_Texts_7_(97830_htm_files).tar.256MB.lzturbo12-39.lzt
03/08/2016  09:20 AM       213,134,910 Sacred_Texts_7_(97830_htm_files).tar.L21.zst
03/06/2016  12:53 AM       253,738,906 Sacred_Texts_7_(97830_htm_files).tar.method28.zpaq
03/07/2016  01:36 AM       151,113,394 Sacred_Texts_7_(97830_htm_files).tar.method58.zpaq
03/10/2016  09:44 PM       183,236,454 Sacred_Texts_7_(97830_htm_files).tar.ST6Block16.bsc
03/05/2016  12:00 PM       176,729,590 Sacred_Texts_7_(97830_htm_files).tar.ST6Block256.bsc

03/10/2016  02:00 PM       127,847,779 vxheaven.org.tar.15_256MB.lz5
03/10/2016  04:20 AM        71,482,377 vxheaven.org.tar.256MB.lzturbo12-39.lzt
03/08/2016  09:23 AM        71,138,408 vxheaven.org.tar.L21.zst
03/06/2016  12:55 AM        76,660,909 vxheaven.org.tar.method28.zpaq
03/07/2016  02:25 AM        64,309,018 vxheaven.org.tar.method58.zpaq
03/10/2016  09:46 PM       162,482,014 vxheaven.org.tar.ST6Block16.bsc
03/05/2016  12:00 PM        77,156,314 vxheaven.org.tar.ST6Block256.bsc

03/10/2016  04:12 PM     2,454,580,699 _Deathship_textual_corpus.tar.15_256MB.lz5
03/10/2016  07:28 AM     1,802,142,279 _Deathship_textual_corpus.tar.256MB.lzturbo12-39.lzt
03/08/2016  11:42 AM     1,849,442,370 _Deathship_textual_corpus.tar.L21.zst
03/06/2016  01:43 AM     2,132,027,578 _Deathship_textual_corpus.tar.method28.zpaq
03/07/2016  11:03 AM     1,433,135,510 _Deathship_textual_corpus.tar.method58.zpaq
03/10/2016  09:58 PM     1,748,595,476 _Deathship_textual_corpus.tar.ST6Block16.bsc
03/05/2016  12:00 PM     1,684,445,506 _Deathship_textual_corpus.tar.ST6Block256.bsc

03/10/2016  05:29 PM       754,471,906 _dict2txt_En_Fr_Ru_Ge_corpus.tar.15_256MB.lz5
03/10/2016  09:27 AM       530,473,578 _dict2txt_En_Fr_Ru_Ge_corpus.tar.256MB.lzturbo12-39.lzt
03/08/2016  12:55 PM       552,303,795 _dict2txt_En_Fr_Ru_Ge_corpus.tar.L21.zst
03/06/2016  02:07 AM       633,428,268 _dict2txt_En_Fr_Ru_Ge_corpus.tar.method28.zpaq
03/07/2016  05:00 PM       375,300,415 _dict2txt_En_Fr_Ru_Ge_corpus.tar.method58.zpaq
03/10/2016  10:02 PM       482,386,772 _dict2txt_En_Fr_Ru_Ge_corpus.tar.ST6Block16.bsc
03/05/2016  12:00 PM       448,139,006 _dict2txt_En_Fr_Ru_Ge_corpus.tar.ST6Block256.bsc

03/10/2016  07:22 PM     1,634,795,785 _DSL_(Dictionary_Specification_Language_(ABBYY_Software_House))_corpus.tar.15_256MB.lz5
03/10/2016  01:02 PM     1,131,510,386 _DSL_(Dictionary_Specification_Language_(ABBYY_Software_House))_corpus.tar.256MB.lzturbo12-39.lzt
03/08/2016  03:06 PM     1,171,777,463 _DSL_(Dictionary_Specification_Language_(ABBYY_Software_House))_corpus.tar.L21.zst
03/06/2016  02:55 AM     1,367,007,877 _DSL_(Dictionary_Specification_Language_(ABBYY_Software_House))_corpus.tar.method28.zpaq
03/08/2016  12:09 AM       865,834,590 _DSL_(Dictionary_Specification_Language_(ABBYY_Software_House))_corpus.tar.method58.zpaq
03/10/2016  10:10 PM     1,069,179,174 _DSL_(Dictionary_Specification_Language_(ABBYY_Software_House))_corpus.tar.ST6Block16.bsc
03/05/2016  12:00 PM     1,016,428,006 _DSL_(Dictionary_Specification_Language_(ABBYY_Software_House))_corpus.tar.ST6Block256.bsc

That's how the DSL looks like:

D:\z>type "Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl"
...
ac{┬╖}qui{┬╖}si{┬╖}tion
 [*]\\╦Мakw╔Щ╦Иzish╔Щn\\[/*] [p]noun[/p]
 [com]([b]-s[/b])[/com]
 [m1][com][*][b]Etymology:[/b] Middle English [i]acquisicioun,[/i] from Middle French or Latin; Middle French [i]acquisition,[/i] from Latin [i]acquisition-, acquisitio,[/i] from [i]acquisitus[/i] (past participle of [i]acquirere[/i]
to acquire) + [i]-ion-, -io[/i] -ion тАФ more at <<acquire>>[/*][/com][/m]
 [m1][b][c navy]1.[/c][/b] [p]тАв[/p] [trn]the act or action of acquiring[/trn][/m]
 [m2][*][ex]power resulting from the [i]acquisition[/i] of wealth[/ex][/*][/m]
 [m2][*][ex]the early [i]acquisition[/i] of self-control in the matter of fear тАФ Bertrand Russell[/ex][/*][/m]
 [m1][b][c navy]2.[/c][/b] [p]тАв[/p] [trn]a thing acquired or gained [p]тАв[/p] <<acquirement>>, <<gain>>[/trn][/m]
 [m1][p]тЦ║[/p] [b]ac┬╖qui┬╖si┬╖tion┬╖ist[/b] [*]\\-╔Щ╠Зst\\[/*] [p]noun[/p] [b]-s[/b][/m]

acquisition cost
 [p]noun[/p]
 [m1][p]тАв[/p] [trn]commissions and other selling expenses in insurance production[/trn][/m]

acquisititious
 [p]adjective[/p]
 [m1][com][*][b]Etymology:[/b] Latin [i]acquisit[/i]us + English [i]-itious[/i][/*][/com][/m]
 [m1][p]obsolete[/p] [p]тАв[/p] <<acquired>>[/m]
...

The add-on roster:

03/10/2016  03:30 PM       125,234,552 Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl
03/10/2016  10:07 PM        27,784,013 Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.15_16MB.lz5
03/10/2016  10:09 PM        27,405,025 Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.15_256MB.lz5
03/10/2016  09:51 PM        26,858,577 Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.16MB.lzturbo12-29.lzt
03/10/2016  09:49 PM        21,606,878 Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.16MB.lzturbo12-39.lzt
03/10/2016  09:47 PM        24,798,626 Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.256MB.lzturbo12-29.lzt
03/10/2016  09:43 PM        19,674,622 Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.256MB.lzturbo12-39.lzt
03/10/2016  03:41 PM        20,537,732 Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.L11_W24.brotli
03/10/2016  09:28 PM        29,197,281 Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.L9.zip
03/10/2016  03:31 PM        35,316,137 Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.lz4
03/10/2016  10:12 PM        23,004,572 Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.LZX21.cab
03/10/2016  03:47 PM        25,587,133 Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.method24.zpaq
03/10/2016  03:46 PM        23,578,720 Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.method28.zpaq
03/10/2016  04:05 PM        14,841,130 Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.method54.zpaq
03/10/2016  03:56 PM        13,444,347 Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.method58.zpaq
03/10/2016  10:12 PM        30,824,703 Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.MSZIP.cab
02/26/2016  07:38 PM        27,787,197 Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.Nakamichi
03/10/2016  09:24 PM        16,801,658 Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.ST6Block16.bsc
03/10/2016  09:24 PM        15,569,950 Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.ST6Block256.bsc
03/10/2016  03:44 PM        19,623,736 Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.xz

The respective command line invocations:

D:\z>LZ5.exe -15 -B5 Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.15_16MB.lz5
D:\z>LZ5.exe -15 -B7 Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.15_256MB.lz5
D:\z>lzturbo.exe -29 -b16 -p0 Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.16MB.lzturbo12-29.lzt
D:\z>lzturbo.exe -39 -b16 -p0 Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.16MB.lzturbo12-39.lzt
D:\z>lzturbo.exe -29 -b256 -p0 Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.256MB.lzturbo12-29.lzt
D:\z>lzturbo.exe -39 -b256 -p0 Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.256MB.lzturbo12-39.lzt
D:\z>bro_Feb-10-2016_mingw64_GCC_5.3.0 -i Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl -o Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.L11_W24.brotli -v --quality 11 --window 24
D:\z>7za a -tgzip -mx9 Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.L9.zip Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl
D:\z>LZ4.exe -16 "Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl"
D:\z>cabarc.exe -m LZX:21 N Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.LZX21.cab Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl
D:\z>zpaq64 add Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.method24.zpaq Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl -method 24 -threads 1
D:\z>zpaq64 add Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.method28.zpaq Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl -method 28 -threads 1
D:\z>zpaq64 add Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.method54.zpaq Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl -method 54 -threads 1
D:\z>zpaq64 add Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.method58.zpaq Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl -method 58 -threads 1
D:\z>cabarc.exe -m MSZIP N Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.MSZIP.cab Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl
D:\z>bsc e "Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl" "Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.ST6Block16.bsc" -b16 -m6 -cp -Tt
D:\z>bsc e "Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl" "Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl.ST6Block256.bsc" -b256 -m6 -cp -Tt
D:\z>xz -z -k -f -9 -e -v -v --threads=1 Webster's_Third_New_International_Unabridged_(1961)_(En-En).dsl

In UFC (Ultimate-Fighting-Championship), they use the term P4P to describe most balanced and skilled Mixed Martial Artist. Ilya Grebnov's BSC excels here, however, in my view the best pound-for-pound decompressor is the Conor Stokes' LZSSE2 level 17, still not available within a console tool... Second best, to me, is Hamid's LzTurbo mode 39, 475MB/s on a weak laptop with 2nd gen mobile i5 and RAM @666MHz - tell me, what compressor can trump it, someone? Third best, Zstd is closing in fast. Nakamichi 'Goldenboy' decompresses at 576MB/s but is fatty - with 7MB overweight. LZSSE2 smokes Goldenboy being 4x faster, PURE CRAZINESS!

Bottom-line: My current TOP 4 of speedy decompressors: