The upper limit for -B option?

[Sanmayce]

Thanks Joergen for your LZSS superb implementation, some feedback.

Two of my testdatafiles, on my i5-7200U 8GB DDR4 2133MHz:

C:\Nakamichi_2020-Jan-01_Defragmented_boosted>timer64 BriefLZ_120_Intel_v15_64bit.exe -x -b300m "Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar" "Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar".x_b300.blz
Current priority class is REALTIME_PRIORITY_CLASS.
blz_pack_ssparse ...
1

Kernel  Time =   635.609 =    0%
User    Time =172412.125 =   96%
Process Time =173047.734 =   96%    Virtual  Memory =   4247 MB
Global  Time =179191.172 =  100%    Physical Memory =   1569 MB

C:\Nakamichi_2020-Jan-01_Defragmented_boosted>timer64 BriefLZ_120_Intel_v15_64bit.exe -x -b350m Star_Trek_-_737_Ebooks.tar Star_Trek_-_737_Ebooks.tar.x_b350.blz
Current priority class is HIGH_PRIORITY_CLASS.
blz_pack_ssparse ...

Exit code: 255

Kernel  Time =     0.203 =    1%
User    Time =     0.000 =    0%
Process Time =     0.203 =    1%    Virtual  Memory =    850 MB
Global  Time =    18.373 =  100%    Physical Memory =    353 MB

C:\Nakamichi_2020-Jan-01_Defragmented_boosted>timer64 BriefLZ_120_Intel_v15_64bit.exe -x -b330m Star_Trek_-_737_Ebooks.tar Star_Trek_-_737_Ebooks.tar.x_b330.blz
Current priority class is HIGH_PRIORITY_CLASS.
blz_pack_ssparse ...
1

Kernel  Time =  2214.031 =    0%
User    Time =2061188.515 =   98%
Process Time =2063402.546 =   98%    Virtual  Memory =   4671 MB
Global  Time =2090984.304 =  100%    Physical Memory =   4141 MB

C:\Nakamichi_2020-Jan-01_Defragmented_boosted>dir

107,784,192 Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar
 35,968,472 Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.Nakamichi
 35,831,237 Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.9.crushxezz
 34,820,909 Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.x_b300.blz

325,071,872 Star_Trek_-_737_Ebooks.tar
100,553,919 Star_Trek_-_737_Ebooks.tar.9.crushxezz
 93,091,642 Star_Trek_-_737_Ebooks.tar.x_b330.blz
 92,975,501 Star_Trek_-_737_Ebooks.tar.Nakamichi

By chance '-b330' was enough to house 'Star-Trek', is it possible to use bigger numbers, ... in next revisions, maybe?

'StarTrek-_737_Ebooks.tar.x_b330.blz' is downloadable at: https://drive.google.com/file/d/12L8poFVBRNYP5kERdBI5TZHSWYwl4Iph/view?usp=sharing

[jibsen]

Thanks for the feedback and interesting results!

The problem is that blz_ssparse_workmem_size() returns an unsigned long, which is 32-bit on Windows, and the calculation overflows at 4096/(3*4) ~= 341MiB.

A quick fix would be to make all the workmem size functions use size_t instead, but you might want to hold off on testing data this size with ssparse, since I am working on a new parser that should be faster for this type of data.

[Sanmayce]

Thanks for the feedback and interesting results!

Always.

The problem is that blz_ssparse_workmem_size() returns an unsigned long, which is 32-bit on Windows, and the calculation overflows at 4096/(3*4) ~= 341MiB.

Good to know, thought that it might be something of that sort, I myself still use randomly both uint64_t or unsigned long long (aiming to have 32bit executables with 64bit variables), yet some such remnants are left here and there.

A quick fix would be to make all the workmem size functions use size_t instead, but you might want to hold off on testing data this size with ssparse, since I am working on a new parser that should be faster for this type of data.

Nice, will wait as long as it takes, Dr. Mahoney's ZPAQ 211 and your incoming BriefLZ (?) will be used as mainstay tools - in order to draw the baseline of non-entropy compressors in my future TEXTORAMIC SHOWDOWN. Not in a hurry, at all, a perennial benchmarker, here. Looking forward testing your latest-n-best.

The thing that stops me (or rather delays me), to enrich the TEXTORAMIC SHOWDOWN, is the lack of enough computational power :(

My observations are that BriefLZ's ssparse beats Nakamichi almost everywhere, with few exceptions (some English language .TXT corpora).

These are the ones which I tried so far:

E:\_TEXTUAL_MADNESS_bare-minimum_2020-Jan-05\BLZ>dir/on

03/26/2019  10:45 PM        42,920,232 (Dictionary_Specification_Language_(ABBYY_Software_House))_Hanyu_Cihai_new_Sea-of-Words_(Zho-Zho).dsl
03/26/2019  10:45 PM        14,336,148 (Dictionary_Specification_Language_(ABBYY_Software_House))_Hanyu_Cihai_new_Sea-of-Words_(Zho-Zho).dsl.Nakamichi
03/26/2019  10:45 PM        13,401,238 (Dictionary_Specification_Language_(ABBYY_Software_House))_Hanyu_Cihai_new_Sea-of-Words_(Zho-Zho).dsl.x_b300.blz

03/26/2019  10:45 PM        52,078,592 Agatha_Christie_108-ebooks_TXT.tar
03/26/2019  10:45 PM        12,283,940 Agatha_Christie_108-ebooks_TXT.tar.Nakamichi
03/26/2019  10:45 PM        11,818,902 Agatha_Christie_108-ebooks_TXT.tar.x_b300.blz

03/26/2019  10:45 PM        15,583,440 Arabian_Nights_complete.html
03/26/2019  10:45 PM         5,039,053 Arabian_Nights_complete.html.Nakamichi
03/26/2019  10:45 PM         4,834,828 Arabian_Nights_complete.html.x_b300.blz

03/26/2019  10:45 PM       193,367,552 Big_Soviet_Encyclopedia_in_30_volumes_(1239-HTMs).tar
03/26/2019  10:45 PM        60,929,805 Big_Soviet_Encyclopedia_in_30_volumes_(1239-HTMs).tar.Nakamichi
03/26/2019  10:45 PM        56,356,825 Big_Soviet_Encyclopedia_in_30_volumes_(1239-HTMs).tar.x_b300.blz

03/26/2019  10:45 PM        55,886,848 Charles_Dickens_in_30_volumes_(Russian_UTF-8).tar
03/26/2019  10:45 PM        12,042,476 Charles_Dickens_in_30_volumes_(Russian_UTF-8).tar.Nakamichi
03/26/2019  10:45 PM        11,973,291 Charles_Dickens_in_30_volumes_(Russian_UTF-8).tar.x_b300.blz

03/26/2019  10:45 PM        42,935,960 Complete_Works_of_Charles_Dickens.txt
03/26/2019  10:45 PM        13,448,970 Complete_Works_of_Charles_Dickens.txt.Nakamichi
03/26/2019  10:45 PM        13,490,362 Complete_Works_of_Charles_Dickens.txt.x_b300.blz

03/26/2019  10:45 PM        40,303,104 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
03/26/2019  10:45 PM         9,077,197 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.Nakamichi
03/26/2019  10:45 PM         8,851,573 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.x_b300.blz

03/26/2019  10:45 PM        11,520,000 ftp.gnu.org_grep-3.3.tar
03/26/2019  10:45 PM         2,163,309 ftp.gnu.org_grep-3.3.tar.Nakamichi
03/26/2019  10:45 PM         1,722,565 ftp.gnu.org_grep-3.3.tar.x_b300.blz

03/26/2019  10:45 PM         9,378,116 Hitler_-_Speeches_and_Proclamations_-_Max_Domarus.txt
03/26/2019  10:45 PM         2,939,813 Hitler_-_Speeches_and_Proclamations_-_Max_Domarus.txt.Nakamichi
03/26/2019  10:45 PM         2,807,518 Hitler_-_Speeches_and_Proclamations_-_Max_Domarus.txt.x_b300.blz

03/26/2019  10:45 PM         3,535,360 LEXX_subtitles.tar
03/26/2019  10:45 PM         1,173,012 LEXX_subtitles.tar.Nakamichi
03/26/2019  10:45 PM         1,085,000 LEXX_subtitles.tar.x_b300.blz

03/26/2019  10:45 PM       206,908,949 OSHO.TXT
03/26/2019  10:45 PM        51,502,498 OSHO.TXT.Nakamichi
03/26/2019  10:45 PM        51,101,509 OSHO.TXT.x_b300.blz

03/26/2019  10:45 PM       167,235,072 Star_Wars_321-books.tar
03/26/2019  10:45 PM        40,420,503 Star_Wars_321-books.tar.Nakamichi
03/26/2019  10:45 PM        39,755,386 Star_Wars_321-books.tar.x_b300.blz

03/26/2019  10:45 PM        69,000,704 Stephen_King_81_novels_TXT.tar
03/26/2019  10:45 PM        21,818,393 Stephen_King_81_novels_TXT.tar.Nakamichi
03/26/2019  10:45 PM        21,730,474 Stephen_King_81_novels_TXT.tar.x_b300.blz

03/26/2019  10:45 PM        33,106,082 Taishukan's_Genius_English-Japanese_Dictionary_(En-Jp).dsl
03/26/2019  10:45 PM         7,021,323 Taishukan's_Genius_English-Japanese_Dictionary_(En-Jp).dsl.Nakamichi
03/26/2019  10:45 PM         6,543,791 Taishukan's_Genius_English-Japanese_Dictionary_(En-Jp).dsl.x_b300.blz

03/26/2019  10:45 PM         3,714,387 The_Complete_Sherlock_Holmes_-_Doyle_Arthur_Conan.txt
03/26/2019  10:45 PM         1,286,110 The_Complete_Sherlock_Holmes_-_Doyle_Arthur_Conan.txt.Nakamichi
03/26/2019  10:45 PM         1,229,620 The_Complete_Sherlock_Holmes_-_Doyle_Arthur_Conan.txt.x_b300.blz

E:\_TEXTUAL_MADNESS_bare-minimum_2020-Jan-05\BLZ>

By the way, you can see my latest-and-finest .ZIP package (.C source and how to compile) here: http://www.sanmayce.com/Nakamichi/index.html#DOWNLOAD

Nakamichi_Vanilla_ICL150_64bit_2020-Feb-02.exe "%1" "%1.Nakamichi" 24 12000 i

In case of 16GB RAM available, above line can compress files less than 200MB at 50KB/s, roughly speaking.

After some 400h will share what RAZOR, bwtturbo, Nakamichi, ZPAQ 511 ... can do to the 'SPETSNAZ' ALL-RUSSIAN corpus which will enrich the picture of Textual Compression: https://pbs.twimg.com/media/EP0OyvjXsAIvMDe?format=jpg&name=large

[jibsen]

By the way, I noticed crushxezz in your results above, you could try bcrush as well.

[Sanmayce]

As always, my intent is all benchmarks to be reproducible, so I uploaded it to my Internet drive: https://drive.google.com/file/d/1YKKVTxqxf8jwl16ay9GftJL6wxVT0Lex/view?usp=sharing

Testmachine: i7-3630QM 3.4GHz, 16GB DDR3 2x800MHz. Currently I played with 4 corpora:

C.Time |        Size | Filename                                                           | Command Line
-------|-------------|--------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------
       | 832,290,816 | book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar                 |
  405s | 166,179,937 | book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar.9.crush_xezz    | crush_xezz.exe -9 "%1" "%1.9.crush_xezz"
  677s | 101,121,392 | book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar.L22W2GB.zst     | zstd-v1.4.4-win64.exe --single-thread --ultra -22 --zstd=wlog=31,clog=30,hlog=30,slog=26 "%1" -o "%1.L22W2GB.zst"
  258s | 121,901,223 | book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar.method211.zpaq  | "zpaq_v7.05_x64.exe" add "%1.method211.zpaq" "%1" -method 211 -threads 1
 1881s |  87,527,794 | book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar.method511.zpaq  | "zpaq_v7.05_x64.exe" add "%1.method511.zpaq" "%1" -method 511 -threads 1
  684s |  97,805,186 | book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar.MX9Dict1024.7z  | "7za_x64_v1900.exe" a -t7z -mx9 -md=30 "%1.MX9Dict1024.7z" "%1"
  N.A. | ???,???,??? | book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar.Nakamichi       | Nakamichi_Vanilla_ICL150_64bit_2020-Feb-02.exe "%1" "%1.Nakamichi" 24 92000 e
  784s | 154,632,025 | book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar.optimal.bcrush  | bcrush_ICL150_64bit.exe --optimal "%1" "%1.optimal.bcrush"
 2812s |  82,503,337 | book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar.rz              | rz.exe a -d 1023M "%1.rz" "%1"
   93s |  76,516,343 | book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar.zv              | bwtturbo -59 -t0 -b1024 "%1" "%1.zv"

C.Time |        Size | Filename                                                           | Command Line
-------|-------------|--------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------
       | 107,784,192 | Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar                      |
   46s |  35,831,237 | Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.9.crush_xezz         | crush_xezz.exe -9 "%1" "%1.9.crush_xezz"
  103s |  27,968,085 | Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.L22W2GB.zst          | zstd-v1.4.4-win64.exe --single-thread --ultra -22 --zstd=wlog=31,clog=30,hlog=30,slog=26 "%1" -o "%1.L22W2GB.zst"
   39s |  33,141,734 | Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.method211.zpaq       | "zpaq_v7.05_x64.exe" add "%1.method211.zpaq" "%1" -method 211 -threads 1
  360s |  20,539,721 | Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.method511.zpaq       | "zpaq_v7.05_x64.exe" add "%1.method511.zpaq" "%1" -method 511 -threads 1
   77s |  27,535,075 | Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.MX9Dict1024.7z       | "7za_x64_v1900.exe" a -t7z -mx9 -md=30 "%1.MX9Dict1024.7z" "%1"
  N.A. |  35,968,472 | Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.Nakamichi            | Nakamichi_Vanilla_ICL150_64bit_2020-Feb-02.exe "%1" "%1.Nakamichi" 24 92000 e
   53s |  33,838,607 | Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.optimal.bcrush       | bcrush_ICL150_64bit.exe --optimal "%1" "%1.optimal.bcrush"
  293s |  24,345,365 | Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.rz                   | rz.exe a -d 1023M "%1.rz" "%1"
   14s |  22,279,538 | Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.zv                   | bwtturbo -59 -t0 -b1024 "%1" "%1.zv"

C.Time |        Size | Filename                                                           | Command Line
-------|-------------|--------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------
       | 325,071,872 | Star_Trek_-_737_Ebooks.tar                                         |
  144s | 100,553,919 | Star_Trek_-_737_Ebooks.tar.9.crush_xezz                            | crush_xezz.exe -9 "%1" "%1.9.crush_xezz"
  377s |  72,762,430 | Star_Trek_-_737_Ebooks.tar.L22W2GB.zst                             | zstd-v1.4.4-win64.exe --single-thread --ultra -22 --zstd=wlog=31,clog=30,hlog=30,slog=26 "%1" -o "%1.L22W2GB.zst"
  119s |  87,925,703 | Star_Trek_-_737_Ebooks.tar.method211.zpaq                          | "zpaq_v7.05_x64.exe" add "%1.method211.zpaq" "%1" -method 211 -threads 1
 1097s |  56,030,908 | Star_Trek_-_737_Ebooks.tar.method511.zpaq                          | "zpaq_v7.05_x64.exe" add "%1.method511.zpaq" "%1" -method 511 -threads 1
  300s |  72,018,734 | Star_Trek_-_737_Ebooks.tar.MX9Dict1024.7z                          | "7za_x64_v1900.exe" a -t7z -mx9 -md=30 "%1.MX9Dict1024.7z" "%1"
  N.A. |  92,975,501 | Star_Trek_-_737_Ebooks.tar.Nakamichi                               | Nakamichi_Vanilla_ICL150_64bit_2020-Feb-02.exe "%1" "%1.Nakamichi" 24 92000 e
  171s |  94,122,899 | Star_Trek_-_737_Ebooks.tar.optimal.bcrush                          | bcrush_ICL150_64bit.exe --optimal "%1" "%1.optimal.bcrush"
 1124s |  63,070,759 | Star_Trek_-_737_Ebooks.tar.rz                                      | rz.exe a -d 1023M "%1.rz" "%1"
   45s |  59,411,519 | Star_Trek_-_737_Ebooks.tar.zv                                      | bwtturbo -59 -t0 -b1024 "%1" "%1.zv"

C.Time |        Size | Filename                                                           | Command Line
-------|-------------|--------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------
       | 630,339,584 | Machine-Learning_..._(over_500000_song_lyrics).tar                 |
  184s | 165,173,195 | Machine-Learning_..._(over_500000_song_lyrics).tar.9.crush_xezz    | crush_xezz.exe -9 "%1" "%1.9.crush_xezz"
  667s | 108,281,760 | Machine-Learning_..._(over_500000_song_lyrics).tar.L22W2GB.zst     | zstd-v1.4.4-win64.exe --single-thread --ultra -22 --zstd=wlog=31,clog=30,hlog=30,slog=26 "%1" -o "%1.L22W2GB.zst"
  206s | 125,295,628 | Machine-Learning_..._(over_500000_song_lyrics).tar.method211.zpaq  | "zpaq_v7.05_x64.exe" add "%1.method211.zpaq" "%1" -method 211 -threads 1
 2043s |  86,008,151 | Machine-Learning_..._(over_500000_song_lyrics).tar.method511.zpaq  | "zpaq_v7.05_x64.exe" add "%1.method511.zpaq" "%1" -method 511 -threads 1
  514s | 103,355,628 | Machine-Learning_..._(over_500000_song_lyrics).tar.MX9Dict1024.7z  | "7za_x64_v1900.exe" a -t7z -mx9 -md=30 "%1.MX9Dict1024.7z" "%1"
  N.A. | 145,750,630 | Machine-Learning_..._(over_500000_song_lyrics).tar.Nakamichi       | Nakamichi_Vanilla_ICL150_64bit_2020-Feb-02.exe "%1" "%1.Nakamichi" 24 92000 e
  392s | 150,924,735 | Machine-Learning_..._(over_500000_song_lyrics).tar.optimal.bcrush  | bcrush_ICL150_64bit.exe --optimal "%1" "%1.optimal.bcrush"
 1748s |  91,722,822 | Machine-Learning_..._(over_500000_song_lyrics).tar.rz              | rz.exe a -d 1023M "%1.rz" "%1"
   68s |  95,432,407 | Machine-Learning_..._(over_500000_song_lyrics).tar.zv              | bwtturbo -59 -t0 -b1024 "%1" "%1.zv"

Note: The 'SPETSNAZ' is currently being crunched by Nakamichi on the same laptop:

Compressing 832,290,816 bytes ...
-; Each rotation means 64KB are encoded; Speed: 0,000,387 B/s; Done 28%; Compression Ratio: 8.03:1; Matches(16/24/48): 1,954,450/1,508,433/82,855; 128[+] long matches: 174,568; ETA: 425.12 hours

Joergen , I have a question, BLOCKWISE, what will be the difference between BriefLZ and BCRUSH? What is the short description of these two LZSS?

For example, my Nakamichi's DIZ: LZSS with 1TB sliding window with greedy parsing.

[jibsen]

Crush is LZSS using a 2MiB sliding window on 64MiB blocks. Under the same restrictions, the encoding used in crush is more efficient than that of BriefLZ, Also, crush has a minimum match length of 3, which works better on some types of data.

BriefLZ is LZSS with a block size up to 4GiB (default 1MiB), and the window size is equal to the block size for most of the algorithms included.

bcrush --optimal         enwik9 -> 264.504.187
blzpack -b64m --optimal  enwik9 -> 271.277.397
blzpack -b1g --optimal   enwik9 -> 252.991.647

[Sanmayce]

Thanks. Looking forward seeing BriefLZ (with -b1024) results for the 10GB wikis...

[jibsen]

BriefLZ 1.3.0 is up, including a fix for this.

[Sanmayce]

Thank you Jørgen for your nifty new performer, surely will play with it ... exhaustively ... but after ~340 hours - the 'SPETSNAZ' has to complete.

makeEXE.bat:

cd example
icl /Qstd=c99 /O3 /I..\include blzpack.c parg.c ..\src\brieflz.c ..\src\depack.c ..\src\depacks.c
copy blzpack.exe ..\BriefLZ_130_Intel_v19_64bit.exe /y

_MakeEXE_Nakamichi_ICL.bat:

icl /TP /O3 /arch:SSE4.1 Nakamichi_Ryuugan-ditto-1TB_btree.c -D_N_XMM -D_N_prefetch_4096 -D_N_HIGH_PRIORITY -D_icl_mumbo_jumbo_ /FAcs -DHashInBITS=24 -DHashChunkSizeInBITS=24 -DRAMpoolInKB=5120 -DBtreeHEURISTIC -D_WIN32_ENVIRONMENT_ -DLongestLineInclusive=64 -DSpeedUpBuilding=32 -DLITE -D_N_alone
copy Nakamichi_Ryuugan-ditto-1TB_btree.exe Nakamichi_Vanilla_LITE_ICL190_64bit.exe /y

A quick showdown:

Satanichi (a.k.a. Nakamichi_2020-Feb-02_Defragmented_boosted_STDOUT_LITE) vs BriefLZ_130

The bottomline first, 2895/106=27.3x slower compression rate :(

Note1: The latest-and-finest revision of Nakamichi is already called Satanichi - it uses only B-trees and discards all memmem() invocations. It is the fastest and most pure Nakamichi, the defragmented external B-trees will show how much IOPS modern SSDs with NAND and Xpoint deliver. Note2: If there is physical RAM in spades then Satanichi is gonna scream when building all the B-trees in one pass only, simply the compile option '-DSpeedUpBuilding=32' has to be increased e.g. to 14000.

The corpus 'Judaica' on i7-3630QM @3.4GHz, 16GB DDR3 @1600MHz:

F:\brieflz-1.3.0_Intel_Bench>timer64 BriefLZ_130_Intel_v19_64bit.exe --optimal -b500m "Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar" "Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.blz"

Kernel  Time =     0.953 =    0%
User    Time =   105.093 =   99%
Process Time =   106.046 =   99%    Virtual  Memory =  11086 MB
Global  Time =   106.142 =  100%    Physical Memory =   2593 MB

F:\brieflz-1.3.0_Intel_Bench>timer64 Nakamichi_Vanilla_LITE_Intel_v19_64bit.exe "Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar" "Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.Nakamichi" 24 9000 i

Nakamichi 'Ryuugan-ditto-1TB', written by Kaze, inspired by Haruhiko Okumura sharing, based on Nobuo Ito's LZSS source, muffinesque tips by m^2, Jim Dempsey and Kirill Kryukov enforced.
Downloadable at: https://twitter.com/Sanmayce/status/1211005691873439744
Note0: Nakamichi 'Dragoneye', (AUTO-RESETTING PASSES and DEFRAGMENTED B-trees revision) is 100% FREE, licenseless that is.
Note1: Hamid Buzidi's LzTurbo ([a] FASTEST [Textual] Decompressor, Levels 19/29/39) retains kingship, his TurboBench (2017-Apr-07) proves the supremacy of LzTurbo, Turbo-Amazing!
Note2: Conor Stokes' LZSSE2 ([a] FASTEST Textual Decompressor, Level 17) is embedded, all credits along with many thanks go to him.
Note3: The matchfinder is either 'Railgun_Trolldom' (matches longer than 18, except 36 and 64) or Leprechaun's B-tree order 3.
Note4: Instead of '_mm_loadu_si128' '_mm_lddqu_si128' is used.
Note5: Maximum compression ratio is 44:1, for 704 bytes long matches within 1TB Sliding Window.
Note6: Please send me (at sanmayce@sanmayce.com) decompression results obtained on machines with fast CPU-RAM subsystems.
Note7: In this compile, clock() was replaced with time() - to counter bigtime stats misreporting.
Note8: Multi-way hashing allows each KeySize to occupy its own HASH pool, thus less RAM is in use - the LEAF is smaller.
Note9: In this revision, B-tree heuristics are in use, allowing skipping many unnecessary memmem() invocations.
NoteA: The file being compressed should be 4 bytes or longer due to Building-Blocks being in range 4..18, 24,28,36,48,64.
NoteB: In this compile, the keysizes in the LEAF are not HEXed i.e. not doubled. Also, the hash in use: FNV1A_Pippip_Yurii, the fastest one known to me.
NoteC: In this latest (2020-Feb-02) compile, SHA3-224 is enabled at compile time, it is optional, that is.
Current priority class is REALTIME_PRIORITY_CLASS.
This compile uses B-trees only, no memmem() invocations - it compresses worse but much faster.
Allocating Source-Buffer 102 MB ...
Allocating Source-Buffer 102 MB (REVERSED) ...
Allocating Target-Buffer 134 MB ...
Leprechaun: Memory pool for B-trees is 9,000 MB.
Leprechaun: In this revision 128MB 13-way hash is used which results in 13 x 16,777,216 internal B-Trees of order 3.
Leprechaun: Allocating HASH memory 1,879,048,257 bytes ... OK
Leprechaun: Allocating memory for B-trees 9001 MB ... OK
Leprechaun: Size of input file: 107,784,192

Leprechaun: Inserting keys/BBs of order 004 into B-trees, free RAM in B-tree pool is 00,009,000 MB; Pass #000,001 of 000,001 ... DONE; 00,000,388,979 B-trees have been rooted so far.
Leprechaun: Inserting keys/BBs of order 006 into B-trees, free RAM in B-tree pool is 00,008,981 MB; Pass #000,001 of 000,001 ...
Leprechaun: Failure! Increment 'Memory for B-trees'!
Automatically increasing number of passes in order to fit B-trees into Target Buffer.
Leprechaun: Inserting keys/BBs of order 006 into B-trees, free RAM in B-tree pool is 00,008,914 MB; Pass #000,002 of 000,002 ... DONE; 00,002,953,386 B-trees have been rooted so far.
Leprechaun: Inserting keys/BBs of order 008 into B-trees, free RAM in B-tree pool is 00,008,848 MB; Pass #000,001 of 000,002 ...
Leprechaun: Failure! Increment 'Memory for B-trees'!
Automatically increasing number of passes in order to fit B-trees into Target Buffer.
Leprechaun: Inserting keys/BBs of order 008 into B-trees, free RAM in B-tree pool is 00,008,848 MB; Pass #000,001 of 000,004 ...
Leprechaun: Failure! Increment 'Memory for B-trees'!
Automatically increasing number of passes in order to fit B-trees into Target Buffer.
Leprechaun: Inserting keys/BBs of order 008 into B-trees, free RAM in B-tree pool is 00,008,579 MB; Pass #000,008 of 000,008 ... DONE; 00,008,259,270 B-trees have been rooted so far.
Leprechaun: Inserting keys/BBs of order 010 into B-trees, free RAM in B-tree pool is 00,008,541 MB; Pass #000,001 of 000,008 ...
Leprechaun: Failure! Increment 'Memory for B-trees'!
Automatically increasing number of passes in order to fit B-trees into Target Buffer.
Leprechaun: Inserting keys/BBs of order 010 into B-trees, free RAM in B-tree pool is 00,008,104 MB; Pass #000,016 of 000,016 ... DONE; 00,015,485,628 B-trees have been rooted so far.
Leprechaun: Inserting keys/BBs of order 012 into B-trees, free RAM in B-tree pool is 00,008,075 MB; Pass #000,001 of 000,016 ...
Leprechaun: Failure! Increment 'Memory for B-trees'!
Automatically increasing number of passes in order to fit B-trees into Target Buffer.
Leprechaun: Inserting keys/BBs of order 012 into B-trees, free RAM in B-tree pool is 00,007,535 MB; Pass #000,032 of 000,032 ... DONE; 00,023,418,126 B-trees have been rooted so far.
Leprechaun: Inserting keys/BBs of order 014 into B-trees, free RAM in B-tree pool is 00,006,965 MB; Pass #000,032 of 000,032 ... DONE; 00,031,128,052 B-trees have been rooted so far.
Leprechaun: Inserting keys/BBs of order 016 into B-trees, free RAM in B-tree pool is 00,006,439 MB; Pass #000,032 of 000,032 ... DONE; 00,038,000,585 B-trees have been rooted so far.
Leprechaun: Inserting keys/BBs of order 018 into B-trees, free RAM in B-tree pool is 00,006,422 MB; Pass #000,001 of 000,032 ...
Leprechaun: Failure! Increment 'Memory for B-trees'!
Automatically increasing number of passes in order to fit B-trees into Target Buffer.
Leprechaun: Inserting keys/BBs of order 018 into B-trees, free RAM in B-tree pool is 00,005,980 MB; Pass #000,064 of 000,064 ... DONE; 00,043,741,138 B-trees have been rooted so far.
Leprechaun: Inserting keys/BBs of order 036 into B-trees, free RAM in B-tree pool is 00,005,973 MB; Pass #000,001 of 000,064 ...
Leprechaun: Failure! Increment 'Memory for B-trees'!
Automatically increasing number of passes in order to fit B-trees into Target Buffer.
Leprechaun: Inserting keys/BBs of order 036 into B-trees, free RAM in B-tree pool is 00,005,908 MB; Pass #000,128 of 000,128 ... DONE; 00,044,346,052 B-trees have been rooted so far.
Leprechaun: Inserting keys/BBs of order 064 into B-trees, free RAM in B-tree pool is 00,005,888 MB; Pass #000,128 of 000,128 ... DONE; 00,044,464,928 B-trees have been rooted so far.
Leprechaun: Inserting keys/BBs of order 024 into B-trees, free RAM in B-tree pool is 00,005,658 MB; Pass #000,128 of 000,128 ... DONE; 00,047,139,473 B-trees have been rooted so far.
Leprechaun: Inserting keys/BBs of order 028 into B-trees, free RAM in B-tree pool is 00,005,514 MB; Pass #000,128 of 000,128 ... DONE; 00,048,670,461 B-trees have been rooted so far.
Leprechaun: Inserting keys/BBs of order 048 into B-trees, free RAM in B-tree pool is 00,005,479 MB; Pass #000,128 of 000,128 ... DONE; 00,048,925,115 B-trees have been rooted so far.

Leprechaun: Total Searches-n-Inserts Per Second: 36,471,808 SNIPS = 90,486,557,125 keys in 2,481 seconds
Leprechaun: RAM needed to house B-trees (relative to the file being ripped): 34N = 3,521MB

Compressing 107,784,192 bytes ...
|; Each rotation means 64KB are encoded; Speed: 0,292,098 B/s; Done 100%; Compression Ratio: 2.99:1; Matches(16/24/48): 1,194,147/371,659/29,877; 128[+] long matches: 0; ETA: 0.00 hours
NumberOfFullLiterals (lower-the-better): 4712
Tsuyo_HEURISTIC_APPLIED_thrice_back-to-back: 0
NumberOf(Tiny)Matches[Micro]Window (4)[16B]: 0
NumberOfMatches[Bheema]Window [128GB window]: 4406
RAM-to-RAM performance: 292098 B/s.
Compressed to 36,035,319 bytes.
Source-file-Hash(FNV1A_YoshimitsuTRIAD) = 0x5bd3,767b
Target-file-Hash(FNV1A_YoshimitsuTRIAD) = 0x23db,b784
Decompressing 36,035,319 (being the compressed stream) bytes ...
RAM-to-RAM performance: 590 MB/s.
Verification (input and output sizes match) OK.
Verification (input and output blocks match) OK.

Kernel  Time =     5.625 =    0%
User    Time =  2888.140 =   99%
Process Time =  2893.765 =   99%    Virtual  Memory =  11156 MB
Global  Time =  2895.675 =  100%    Physical Memory =  11136 MB

F:\brieflz-1.3.0_Intel_Bench>dir

04/06/2017  21:44       107,784,192 Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar
09/15/2018  20:09        34,820,909 Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.blz
09/15/2018  20:57        36,035,319 Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.Nakamichi

F:\brieflz-1.3.0_Intel_Bench>

After finishing 'SPETSNAZ' will run the new BriefLZ with many corpora...

The .C source and ICL v19.0 and GCC v7.3.0 binaries are here: http://www.sanmayce.com/Nakamichi/Satanichi_aka_Nakamichi_2020-Feb-02_Defragmented_boosted_STDOUT_LITE.zip

[jibsen]

Glad to hear it appears to be working!

I don't think it will make much of a difference, but you can compile with /DNDEBUG to disable the assertions around the BriefLZ code.

[Sanmayce]

Two more testfiles are done, on i5-7200U 3.1GHz, 8GB DDR4 2x1066MHz:

A Russian prose UTF-8 .TXT tarred corpus:

D:\TEXTORAMIC_part>lzbench18 -c4 -i1,15 -o3 -elzham,4/slz_deflate,3/slz_gzip,3/slz_zlib,3/tornado,1,16/bzip2,1,9/zstd,1,22/zstdLDM,1,22/fastlzma2,1,10/brotli,0,1,3,5,11/blosclz,1,9/crush,2/csc,5/density,1,3/fastlz,2/gipfeli/lzo1b,999/libdeflate,1,12/lz4,1,12/lizard,11,19,21,29,31,39,41,49/lzf,1/lzfse/lzg,9/lzjb/lzlib,9/lzma,0,1,9/lzrw,5/lzsse2,17/lzsse4,17/lzsse8,17/lzvn/pithy,9/quicklz,3/snappy/slz_zlib,3/ucl_nrv2b,9/ucl_nrv2d,9/ucl_nrv2e,9/xpack,1,9/xz,9/yalz77,12/yappy,99/zlib,1,5,9/zling,4/shrinker/wflz/lzmat ""book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar""
lzbench 1.8 (64-bit Windows)   Assembled by P.Skibinski
The results sorted by column number 4:
Compressor name         Compress. Decompress.  Orig. size  Compr. size  Ratio Filename
zstd 1.4.3 -22           1.36 MB/s   892 MB/s   832290816    106285862  12.77 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
zstdLDM 1.4.3 -22        1.37 MB/s   893 MB/s   832290816    106556756  12.80 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
xz 5.2.4 -9              1.13 MB/s   140 MB/s   832290816    107796983  12.95 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lzma 19.00 -9            1.10 MB/s   139 MB/s   832290816    107849433  12.96 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
tornado 0.6a -16         1.48 MB/s   288 MB/s   832290816    108105061  12.99 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lzham 1.0 -d26 -4        0.80 MB/s   313 MB/s   832290816    108470968  13.03 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
fastlzma2 1.0.1 -10      2.36 MB/s   120 MB/s   832290816    112251678  13.49 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lzlib 1.11 -9            1.05 MB/s    92 MB/s   832290816    113146693  13.59 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
csc 2016-10-13 -5        1.39 MB/s   125 MB/s   832290816    116667138  14.02 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar

BriefLZ_130_Intel_v19_64bit --optimal -b2g                   131370690        ! outside lzbench !

brotli 2019-10-01 -11    0.45 MB/s   551 MB/s   832290816    132070667  15.87 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lizard 1.0 -49           1.50 MB/s  1044 MB/s   832290816    134883363  16.21 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
bzip2 1.0.8 -9             10 MB/s    26 MB/s   832290816    145658823  17.50 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar

Satanichi_GCC730_64bit.exe          1124 MB/s                149041296        ! outside lzbench !

lizard 1.0 -29           1.55 MB/s  1312 MB/s   832290816    158642651  19.06 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
crush 1.0 -2             0.21 MB/s   407 MB/s   832290816    167891024  20.17 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
zling 2018-10-12 -4        31 MB/s   194 MB/s   832290816    172424277  20.72 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
bzip2 1.0.8 -1             11 MB/s    39 MB/s   832290816    177659512  21.35 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
brotli 2019-10-01 -5       28 MB/s   523 MB/s   832290816    179403587  21.56 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
xpack 2016-06-02 -9      4.38 MB/s   948 MB/s   832290816    207604165  24.94 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lizard 1.0 -39           4.31 MB/s  1832 MB/s   832290816    209834821  25.21 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
libdeflate 1.3 -12       3.67 MB/s  1038 MB/s   832290816    213213594  25.62 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lzsse2 2019-04-18 -17    2.43 MB/s  3926 MB/s   832290816    216869808  26.06 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lzsse4 2019-04-18 -17    2.53 MB/s  3909 MB/s   832290816    217770793  26.17 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
brotli 2019-10-01 -3       78 MB/s   436 MB/s   832290816    219485981  26.37 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lzsse8 2019-04-18 -17    2.45 MB/s  3527 MB/s   832290816    224389638  26.96 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
zlib 1.2.11 -9           4.17 MB/s   299 MB/s   832290816    227825566  27.37 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
fastlzma2 1.0.1 -1         13 MB/s    60 MB/s   832290816    232444725  27.93 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lzma 19.00 -1              16 MB/s    62 MB/s   832290816    233717316  28.08 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
yalz77 2015-09-19 -12      30 MB/s   376 MB/s   832290816    237645327  28.55 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
ucl_nrv2e 1.03 -9        0.36 MB/s   311 MB/s   832290816    243105097  29.21 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lzma 19.00 -0              17 MB/s    57 MB/s   832290816    244108780  29.33 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
ucl_nrv2d 1.03 -9        0.30 MB/s   315 MB/s   832290816    245933936  29.55 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lzo1b 2.10 -999          4.20 MB/s   578 MB/s   832290816    246693493  29.64 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lizard 1.0 -19           4.35 MB/s  2617 MB/s   832290816    248001307  29.80 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
zlib 1.2.11 -5             21 MB/s   275 MB/s   832290816    251891158  30.26 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
ucl_nrv2b 1.03 -9        0.34 MB/s   300 MB/s   832290816    252275771  30.31 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lzg 1.0.10 -9            0.18 MB/s   663 MB/s   832290816    254874747  30.62 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lizard 1.0 -41            152 MB/s  1148 MB/s   832290816    255405016  30.69 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
zstdLDM 1.4.3 -1          134 MB/s   413 MB/s   832290816    257066314  30.89 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
zstd 1.4.3 -1             210 MB/s   413 MB/s   832290816    257209725  30.90 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lzfse 2017-03-08           50 MB/s   531 MB/s   832290816    259743898  31.21 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
brotli 2019-10-01 -1      174 MB/s   356 MB/s   832290816    266460227  32.02 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lzmat 1.01                 11 MB/s   299 MB/s   832290816    266952375  32.07 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
libdeflate 1.3 -1         135 MB/s   756 MB/s   832290816    285713451  34.33 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
brotli 2019-10-01 -0      217 MB/s   343 MB/s   832290816    287973706  34.60 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
xpack 2016-06-02 -1       119 MB/s   499 MB/s   832290816    298137386  35.82 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lizard 1.0 -21            176 MB/s  1742 MB/s   832290816    310381370  37.29 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lizard 1.0 -31            198 MB/s  1459 MB/s   832290816    315336786  37.89 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
zlib 1.2.11 -1             66 MB/s   244 MB/s   832290816    321781027  38.66 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
yappy 2014-03-22 -99       55 MB/s  1887 MB/s   832290816    322920818  38.80 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lzvn 2017-03-08            39 MB/s   773 MB/s   832290816    331212729  39.80 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
pithy 2011-12-24 -9       289 MB/s  1437 MB/s   832290816    338709477  40.70 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
gipfeli 2016-07-13        231 MB/s   444 MB/s   832290816    344610201  41.41 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lzrw 15-Jul-1991 -5        87 MB/s   405 MB/s   832290816    346324028  41.61 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
quicklz 1.5.0 -3           36 MB/s   557 MB/s   832290816    348569403  41.88 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
density 0.14.2 -3         420 MB/s   380 MB/s   832290816    370942860  44.57 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lizard 1.0 -11            245 MB/s  2063 MB/s   832290816    387848544  46.60 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lz4 1.9.2                 310 MB/s  2283 MB/s   832290816    408975647  49.14 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lz4 1.9.2                 309 MB/s  2282 MB/s   832290816    408975647  49.14 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
snappy 2019-09-30         193 MB/s   829 MB/s   832290816    423828918  50.92 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
tornado 0.6a -1           220 MB/s   269 MB/s   832290816    435175239  52.29 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
fastlz 0.1 -2             246 MB/s   385 MB/s   832290816    436415456  52.44 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lzf 3.6 -1                229 MB/s   470 MB/s   832290816    443098437  53.24 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
density 0.14.2 -1        1714 MB/s  1809 MB/s   832290816    445806192  53.56 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
slz_deflate 1.0.0 -3      203 MB/s   244 MB/s   832290816    453783256  54.52 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
slz_gzip 1.0.0 -3         184 MB/s   223 MB/s   832290816    453783262  54.52 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
slz_zlib 1.0.0 -3         169 MB/s   203 MB/s   832290816    453783274  54.52 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
slz_zlib 1.0.0 -3         175 MB/s   203 MB/s   832290816    453783274  54.52 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
blosclz 2.0.0 -9          215 MB/s   360 MB/s   832290816    497813982  59.81 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lzjb 2010                 174 MB/s   285 MB/s   832290816    539617346  64.84 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
wflz 2015-09-16           185 MB/s   463 MB/s   832290816    557630036  67.00 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
shrinker 0.1              228 MB/s  6557 MB/s   832290816    820229674  98.55 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
blosclz 2.0.0 -1         7352 MB/s  7534 MB/s   832290816    832290816 100.00 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
memcpy                  10742 MB/s 11009 MB/s   832290816    832290816 100.00 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar

D:\TEXTORAMIC_part>"turbobench_v18.05_-_build_04_May_2018" ""book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar"" -esnappy_c/yappy/bzip2/lzma,9d30:fb273:mf=bt4/oodle,89,91,95,99,111,115,119,129,139/lzturbo,19,12,10,29,22,20,39,32,30,59/brotli,11d30/zstd,1,5,12,22,22d30/lizard,11,19,21,29,31,39,41,49/trle -I3 -J31 -k1 -B2G
      C Size  ratio%     C MB/s     D MB/s   Name            File
    86229509    10.4       4.35      28.08   lzturbo 59                       book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
    97744466    11.7       0.86     148.98   lzma 9d30:fb273:mf=bt4           book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
    97853381    11.8       0.07     940.83   lzturbo 39                       book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
    99153037    11.9       0.40     358.11   brotli 11d30                     book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
   101289846    12.2       1.28     813.45   zstd 22d30                       book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
   104062328    12.5       0.18     736.02   oodle 139                        book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
   104380801    12.5       0.23     789.25   oodle 89                         book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
   104402322    12.5       0.17     784.01   oodle 129                        book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
   106451962    12.8       1.45    1608.63   zstd 22                          book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
   110920918    13.3       0.07    1409.53   lzturbo 29                       book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
   126495932    15.2       0.26    1532.43   oodle 99                         book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar

   131370690                                 BriefLZ_130_Intel_v19_64bit --optimal -b2g                   ! outside turbobench !

   134883367    16.2       1.39    1004.33   lizard 49                        book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
   136202664    16.4       0.49    1838.40   oodle 95                         book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
   145658827    17.5      10.54      24.71   bzip2                            book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
   148673910    17.9       0.27    2484.08   oodle 119                        book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
   148929102    17.9      52.47    1202.12   lzturbo 32                       book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar

   149041296                       1124      Satanichi_GCC730_64bit.exe       ! outside turbobench !

   158150115    19.0       0.53    2004.32   oodle 115                        book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
   158642655    19.1       1.41    1241.57   lizard 29                        book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
   167316896    20.1       7.59    1654.77   zstd 12                          book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
   209834825    25.2       3.95    1696.33   lizard 39                        book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
   224390328    27.0      83.35     679.06   zstd 5                           book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
   231399459    27.8      48.79    1392.72   lzturbo 22                       book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
   233761178    28.1     277.87    1009.25   lzturbo 30                       book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
   247855856    29.8       0.07    3179.57   lzturbo 19                       book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
   248001311    29.8       4.37    2362.27   lizard 19                        book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
   253908003    30.5     196.22    2479.17   oodle 91                         book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
   255405020    30.7     129.25    1102.80   lizard 41                        book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
   257953411    31.0     166.80     767.78   zstd 1                           book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
   277477721    33.3      64.90    3352.01   lzturbo 12                       book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
   310381374    37.3     142.54    1659.84   lizard 21                        book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
   315336790    37.9     154.72    1350.42   lizard 31                        book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
   326323817    39.2      60.69    1804.80   yappy                            book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
   343518192    41.3     408.19    1659.96   lzturbo 20                       book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
   381026902    45.8     235.37    2868.88   oodle 111                        book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
   387848548    46.6     182.29    2414.03   lizard 11                        book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
   423797463    50.9     326.63    1653.17   snappy_c                         book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
   478153400    57.5     378.77    2116.37   lzturbo 10                       book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
   830022523    99.7     196.24    2016.97   trle                             book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar

And the 'usual suspects':

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| Corpus Name                                                     Corpus Size |   Global Time |  Virtual Memory | Command Line                                                                                                      |
|                                                             Compressed Size |  Process Time | Physical Memory | Testmachine                                                                                                       |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar        832,290,816 bytes |    1,282,007s |        9,601 MB | Satanichi_GCC730_64bit.exe "%1" "%1.Nakamichi" 26 80000 e                                                         |
|                                                           149,041,296 bytes |      248,676s |        9,552 MB | i7-3630QM 3.4GHz, 16GB DDR3 2x800MHz, SSD 256GB Samsung 860 PRO (2bit MLC, LPDDR4-1866 512 MB)                    |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar        832,290,816 bytes |          260s |        5,387 MB | "zpaq_v7.05_x64.exe" add "%1.method211.zpaq" "%1" -method 211 -threads 1                                          |
|                                                           121,901,223 bytes |          260s |        3,922 MB | i7-3630QM 3.4GHz, 16GB DDR3 2x800MHz, SSD 256GB Samsung 860 PRO (2bit MLC, LPDDR4-1866 512 MB)                    |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar        832,290,816 bytes |        1,845s |       11,177 MB | "zpaq_v7.05_x64.exe" add "%1.method511.zpaq" "%1" -method 511 -threads 1                                          |
|                                                            87,527,794 bytes |        1,845s |        6,937 MB | i7-3630QM 3.4GHz, 16GB DDR3 2x800MHz, SSD 256GB Samsung 860 PRO (2bit MLC, LPDDR4-1866 512 MB)                    |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar        832,290,816 bytes |          650s |        9,006 MB | zstd-v1.4.4-win64.exe --single-thread --ultra -22 --zstd=wlog=31,clog=30,hlog=30,slog=26 "%1" -o "%1.L22W2GB.zst" |
|                                                           101,121,392 bytes |          645s |        8,990 MB | i7-3630QM 3.4GHz, 16GB DDR3 2x800MHz, SSD 256GB Samsung 860 PRO (2bit MLC, LPDDR4-1866 512 MB)                    |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar        832,290,816 bytes |           92s |        5,651 MB | bwtturbo -59 -t0 -b1024 "%1" "%1.zv"                                                                              |
|                                                            76,516,343 bytes |           91s |        4,127 MB | i7-3630QM 3.4GHz, 16GB DDR3 2x800MHz, SSD 256GB Samsung 860 PRO (2bit MLC, LPDDR4-1866 512 MB)                    |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar        832,290,816 bytes |        2,764s |        9,531 MB | rz.exe a -d 1023M "%1.rz" "%1"                                                                                    |
|                                                            82,503,337 bytes |        3,469s |        6,400 MB | i7-3630QM 3.4GHz, 16GB DDR3 2x800MHz, SSD 256GB Samsung 860 PRO (2bit MLC, LPDDR4-1866 512 MB)                    |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar        832,290,816 bytes |          672s |       10,781 MB | "7za_x64_v1900.exe" a -t7z -mx9 -md=30 "%1.MX9Dict1024.7z" "%1"                                                   |
|                                                            97,805,186 bytes |          880s |        8,178 MB | i7-3630QM 3.4GHz, 16GB DDR3 2x800MHz, SSD 256GB Samsung 860 PRO (2bit MLC, LPDDR4-1866 512 MB)                    |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar        832,290,816 bytes |          775s |        1,420 MB | bcrush_ICL150_64bit.exe --optimal "%1" "%1.optimal.bcrush"                                                        |
|                                                           154,632,025 bytes |          775s |        1,360 MB | i7-3630QM 3.4GHz, 16GB DDR3 2x800MHz, SSD 256GB Samsung 860 PRO (2bit MLC, LPDDR4-1866 512 MB)                    |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar        832,290,816 bytes |          396s |          851 MB | crush_xezz.exe -9 "%1" "%1.9.crush_xezz"                                                                          |
|                                                           166,179,937 bytes |          396s |          852 MB | i7-3630QM 3.4GHz, 16GB DDR3 2x800MHz, SSD 256GB Samsung 860 PRO (2bit MLC, LPDDR4-1866 512 MB)                    |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar        832,290,816 bytes |      221,703s |       45,402 MB | BriefLZ_130_Intel_v19_64bit.exe --optimal -b2g "%1" "%1.blz"                                                      |
|                                                           131,370,690 bytes |      221,699s |       16,383 MB | i7-3630QM 3.4GHz, 16GB DDR3 2x800MHz, SSD 256GB Samsung 860 PRO (2bit MLC, LPDDR4-1866 512 MB)                    |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar        832,290,816 bytes |        2,105s |       10,338 MB | brotli_ic19.exe -q 11 --large_window=30 "%1" -o "%1.bro"                                                          |
|                                                            99,152,526 bytes |        2,105s |        6,094 MB | i7-3630QM 3.4GHz, 16GB DDR3 2x800MHz, SSD 256GB Samsung 860 PRO (2bit MLC, LPDDR4-1866 512 MB)                    |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

An English lyrics .TXT tarred corpus:

D:\TEXTORAMIC_part>lzbench18 -c4 -i1,15 -o3 -elzham,4/slz_deflate,3/slz_gzip,3/slz_zlib,3/tornado,1,16/bzip2,1,9/zstd,1,22/zstdLDM,1,22/fastlzma2,1,10/brotli,0,1,3,5,11/blosclz,1,9/crush,2/csc,5/density,1,3/fastlz,2/gipfeli/lzo1b,999/libdeflate,1,12/lz4,1,12/lizard,11,19,21,29,31,39,41,49/lzf,1/lzfse/lzg,9/lzjb/lzlib,9/lzma,0,1,9/lzrw,5/lzsse2,17/lzsse4,17/lzsse8,17/lzvn/pithy,9/quicklz,3/snappy/slz_zlib,3/ucl_nrv2b,9/ucl_nrv2d,9/ucl_nrv2e,9/xpack,1,9/xz,9/yalz77,12/yappy,99/zlib,1,5,9/zling,4/shrinker/wflz/lzmat ""Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar""
lzbench 1.8 (64-bit Windows)   Assembled by P.Skibinski
The results sorted by column number 4:
Compressor name         Compress. Decompress.  Orig. size  Compr. size  Ratio Filename
lzma 19.00 -9            1.19 MB/s    99 MB/s   630339584    115019802  18.25 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
xz 5.2.4 -9              1.21 MB/s    94 MB/s   630339584    115020496  18.25 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
csc 2016-10-13 -5        2.70 MB/s    64 MB/s   630339584    115904942  18.39 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
zstd 1.4.3 -22           1.03 MB/s   675 MB/s   630339584    116108562  18.42 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
tornado 0.6a -16         1.40 MB/s   209 MB/s   630339584    116657946  18.51 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
lzham 1.0 -d26 -4        0.84 MB/s   222 MB/s   630339584    116724512  18.52 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
fastlzma2 1.0.1 -10      2.29 MB/s    89 MB/s   630339584    117172953  18.59 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
zstdLDM 1.4.3 -22        1.07 MB/s   673 MB/s   630339584    117382083  18.62 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
lzlib 1.11 -9            1.05 MB/s    69 MB/s   630339584    119213183  18.91 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar

BriefLZ_130_Intel_v19_64bit --optimal -b2g                   129595542        ! outside lzbench !

brotli 2019-10-01 -11    0.45 MB/s   369 MB/s   630339584    133801391  21.23 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
zling 2018-10-12 -4        33 MB/s   168 MB/s   630339584    149896637  23.78 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
lizard 1.0 -49           1.20 MB/s   935 MB/s   630339584    152783164  24.24 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar

Satanichi_GCC730_64bit.exe           942 MB/s                155402007        ! outside lzbench !

brotli 2019-10-01 -5       25 MB/s   397 MB/s   630339584    159025077  25.23 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
bzip2 1.0.8 -9             10 MB/s    26 MB/s   630339584    159111060  25.24 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
crush 1.0 -2             0.24 MB/s   311 MB/s   630339584    159884165  25.36 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
fastlzma2 1.0.1 -1         12 MB/s    63 MB/s   630339584    169549257  26.90 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
xpack 2016-06-02 -9      8.15 MB/s   868 MB/s   630339584    171789474  27.25 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
libdeflate 1.3 -12       5.88 MB/s   687 MB/s   630339584    174665802  27.71 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
bzip2 1.0.8 -1             11 MB/s    34 MB/s   630339584    176353273  27.98 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
lzma 19.00 -1              19 MB/s    61 MB/s   630339584    177675597  28.19 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
brotli 2019-10-01 -3       75 MB/s   380 MB/s   630339584    180750423  28.68 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
lizard 1.0 -29           1.27 MB/s  1479 MB/s   630339584    181099446  28.73 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
lzfse 2017-03-08           49 MB/s   647 MB/s   630339584    182618202  28.97 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
zlib 1.2.11 -9             14 MB/s   279 MB/s   630339584    182978127  29.03 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
lizard 1.0 -39           3.87 MB/s  1735 MB/s   630339584    186023054  29.51 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
zlib 1.2.11 -5             28 MB/s   273 MB/s   630339584    186950753  29.66 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
lzma 19.00 -0              22 MB/s    54 MB/s   630339584    188157873  29.85 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
ucl_nrv2e 1.03 -9        0.97 MB/s   269 MB/s   630339584    192253614  30.50 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
brotli 2019-10-01 -1      159 MB/s   339 MB/s   630339584    193676957  30.73 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
ucl_nrv2d 1.03 -9        0.94 MB/s   275 MB/s   630339584    194174184  30.80 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
lzsse2 2019-04-18 -17    2.76 MB/s  2836 MB/s   630339584    194627075  30.88 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
lzsse4 2019-04-18 -17    2.80 MB/s  3205 MB/s   630339584    196223860  31.13 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
zstdLDM 1.4.3 -1          155 MB/s   570 MB/s   630339584    197357493  31.31 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
lzsse8 2019-04-18 -17    2.75 MB/s  3142 MB/s   630339584    197510652  31.33 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
ucl_nrv2b 1.03 -9        1.06 MB/s   267 MB/s   630339584    198130244  31.43 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
zstd 1.4.3 -1             278 MB/s   563 MB/s   630339584    199130517  31.59 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
xpack 2016-06-02 -1       131 MB/s   648 MB/s   630339584    200250214  31.77 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
libdeflate 1.3 -1         142 MB/s   755 MB/s   630339584    200440695  31.80 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
lizard 1.0 -41            136 MB/s  1046 MB/s   630339584    205067726  32.53 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
lzo1b 2.10 -999            10 MB/s   506 MB/s   630339584    206761274  32.80 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
brotli 2019-10-01 -0      214 MB/s   316 MB/s   630339584    207643666  32.94 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
lzmat 1.01                 31 MB/s   270 MB/s   630339584    211176426  33.50 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
lizard 1.0 -19           4.14 MB/s  2623 MB/s   630339584    213237120  33.83 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
zlib 1.2.11 -1             78 MB/s   277 MB/s   630339584    215765399  34.23 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
yalz77 2015-09-19 -12      20 MB/s   289 MB/s   630339584    222530955  35.30 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
lzg 1.0.10 -9            0.59 MB/s   485 MB/s   630339584    227738922  36.13 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
lzvn 2017-03-08            41 MB/s   790 MB/s   630339584    230181093  36.52 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
quicklz 1.5.0 -3           44 MB/s   682 MB/s   630339584    235091903  37.30 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
lizard 1.0 -31            182 MB/s  1769 MB/s   630339584    235304777  37.33 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
gipfeli 2016-07-13        215 MB/s   385 MB/s   630339584    236217347  37.47 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
pithy 2011-12-24 -9       245 MB/s  1344 MB/s   630339584    249961466  39.66 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
lizard 1.0 -21            161 MB/s  1825 MB/s   630339584    255415901  40.52 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
lizard 1.0 -11            214 MB/s  2575 MB/s   630339584    261140979  41.43 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
lzrw 15-Jul-1991 -5       100 MB/s   382 MB/s   630339584    266188186  42.23 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
fastlz 0.1 -2             249 MB/s   421 MB/s   630339584    273296035  43.36 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
lzf 3.6 -1                244 MB/s   513 MB/s   630339584    273507057  43.39 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
density 0.14.2 -3         319 MB/s   313 MB/s   630339584    276123004  43.81 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
yappy 2014-03-22 -99       76 MB/s  1580 MB/s   630339584    278881061  44.24 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
slz_deflate 1.0.0 -3      198 MB/s   275 MB/s   630339584    281261549  44.62 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
slz_gzip 1.0.0 -3         180 MB/s   248 MB/s   630339584    281261555  44.62 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
slz_zlib 1.0.0 -3         171 MB/s   224 MB/s   630339584    281261567  44.62 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
slz_zlib 1.0.0 -3         171 MB/s   223 MB/s   630339584    281261567  44.62 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
lz4 1.9.2                 412 MB/s  2963 MB/s   630339584    289439405  45.92 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
lz4 1.9.2                 415 MB/s  2940 MB/s   630339584    289439405  45.92 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
snappy 2019-09-30         234 MB/s   878 MB/s   630339584    299969197  47.59 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
tornado 0.6a -1           207 MB/s   301 MB/s   630339584    303416172  48.14 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
wflz 2015-09-16           181 MB/s   601 MB/s   630339584    336375445  53.36 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
lzjb 2010                 197 MB/s   345 MB/s   630339584    359561457  57.04 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
density 0.14.2 -1        1070 MB/s  1496 MB/s   630339584    363440712  57.66 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
blosclz 2.0.0 -9          229 MB/s   458 MB/s   630339584    406780399  64.53 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
shrinker 0.1              228 MB/s  3496 MB/s   630339584    577034189  91.54 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
blosclz 2.0.0 -1         7509 MB/s  7710 MB/s   630339584    630339584 100.00 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
memcpy                  11163 MB/s 10761 MB/s   630339584    630339584 100.00 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar

D:\TEXTORAMIC_part>"turbobench_v18.05_-_build_04_May_2018" ""Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar"" -esnappy_c/yappy/bzip2/lzma,9d30:fb273:mf=bt4/oodle,89,91,95,99,111,115,119,129,139/lzturbo,19,12,10,29,22,20,39,32,30,59/brotli,11d30/zstd,1,5,22d30/lizard,11,19,21,29,31,39,41,49/trle -I3 -J31 -k1 -B2G
      C Size  ratio%     C MB/s     D MB/s   Name            File
   102237996    16.2       0.39     264.44   brotli 11d30                     Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
   102669101    16.3       6.54      33.42   lzturbo 59                       Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
   103163583    16.4       0.83     108.35   lzma 9d30:fb273:mf=bt4           Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
   105172793    16.7       0.60     779.94   lzturbo 39                       Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
   107805227    17.1       0.19     613.36   oodle 139                        Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
   108042846    17.1       1.12     704.78   zstd 22d30                       Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
   108053499    17.1       0.24     689.24   oodle 89                         Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
   108069844    17.1       0.16     690.75   oodle 129                        Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
   119598213    19.0       0.58    1133.88   lzturbo 29                       Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar

   129595542                                 BriefLZ_130_Intel_v19_64bit --optimal -b2g                                                               ! outside turbobench !

   129784894    20.6      41.55     756.45   lzturbo 32                       Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
   134870724    21.4       0.34    1869.38   oodle 99                         Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
   137326345    21.8       0.74    1874.64   oodle 95                         Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
   152783168    24.2       1.23     939.20   lizard 49                        Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar

   155402007                        942      Satanichi_GCC730_64bit.exe       ! outside turbobench !

   159111064    25.2      10.64      25.78   bzip2                            Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
   168353489    26.7       0.36    2870.07   oodle 119                        Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
   172048724    27.3       0.80    2946.52   oodle 115                        Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
   174938936    27.8      57.94     721.46   zstd 5                           Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
   181099450    28.7       1.30    1477.80   lizard 29                        Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
   182005612    28.9      37.41    1109.71   lzturbo 22                       Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
   186023058    29.5       4.13    1651.84   lizard 39                        Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
   194179350    30.8     223.97     969.77   lzturbo 30                       Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
   199248465    31.6     131.93    1741.08   oodle 91                         Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
   199521959    31.7     277.58     914.52   zstd 1                           Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
   205067730    32.5     136.22    1023.51   lizard 41                        Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
   212944026    33.8       0.43    3062.50   lzturbo 19                       Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
   213237124    33.8       4.66    2479.30   lizard 19                        Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
   227939040    36.2      52.69    3010.22   lzturbo 12                       Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
   235304781    37.3     181.33    1686.59   lizard 31                        Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
   255415905    40.5     162.06    1819.06   lizard 21                        Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
   261140983    41.4     215.01    2429.45   lizard 11                        Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
   266282792    42.2     339.58    1086.26   lzturbo 20                       Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
   266307002    42.2     148.95    3135.91   oodle 111                        Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
   279864603    44.4      81.24    1521.44   yappy                            Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
   290977860    46.2     368.64    2673.59   lzturbo 10                       Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
   299900275    47.6     333.39     946.33   snappy_c                         Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
   629881032    99.9     191.84    1998.16   trle                             Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar

And the 'usual suspects':

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| Corpus Name                                                     Corpus Size |   Global Time |  Virtual Memory | Command Line                                                                                                      |
|                                                             Compressed Size |  Process Time | Physical Memory | Testmachine                                                                                                       |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| Machine-Learning_..._(over_500000_song_lyrics).tar        630,339,584 bytes |    1,185,386s |        3,636 MB | Satanichi_GCC730_64bit.exe "%1" "%1.Nakamichi" 24 99000 e                                                         |
|                                                           155,402,007 bytes |       58,847s |        3,029 MB | i5-7200U 3.1GHz, 8GB DDR4 2x1066MHz, SSD 500GB Samsung 860 EVO (3bit MLC, LPDDR4-1866 512 MB)                     |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| Machine-Learning_..._(over_500000_song_lyrics).tar        630,339,584 bytes |          193s |        5,144 MB | "zpaq_v7.05_x64.exe" add "%1.method211.zpaq" "%1" -method 211 -threads 1                                          |
|                                                           125,295,629 bytes |          193s |        3,619 MB | i5-7200U 3.1GHz, 8GB DDR4 2x1066MHz, SSD 500GB Samsung 860 EVO (3bit MLC, LPDDR4-1866 512 MB)                     |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| Machine-Learning_..._(over_500000_song_lyrics).tar        630,339,584 bytes |        3,036s |       11,698 MB | "zpaq_v7.05_x64.exe" add "%1.method511.zpaq" "%1" -method 511 -threads 1                                          |
|                                                            86,008,152 bytes |        2,859s |        7,174 MB | i5-7200U 3.1GHz, 8GB DDR4 2x1066MHz, SSD 500GB Samsung 860 EVO (3bit MLC, LPDDR4-1866 512 MB)                     |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| Machine-Learning_..._(over_500000_song_lyrics).tar        630,339,584 bytes |          664s |        8,813 MB | zstd-v1.4.4-win64.exe --single-thread --ultra -22 --zstd=wlog=31,clog=30,hlog=30,slog=26 "%1" -o "%1.L22W2GB.zst" |
|                                                           108,281,760 bytes |          659s |        7,344 MB | i5-7200U 3.1GHz, 8GB DDR4 2x1066MHz, SSD 500GB Samsung 860 EVO (3bit MLC, LPDDR4-1866 512 MB)                     |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| Machine-Learning_..._(over_500000_song_lyrics).tar        630,339,584 bytes |           69s |        4,833 MB | bwtturbo -59 -t0 -b1024 "%1" "%1.zv"                                                                              |
|                                                            95,432,407 bytes |           67s |        3,346 MB | i5-7200U 3.1GHz, 8GB DDR4 2x1066MHz, SSD 500GB Samsung 860 EVO (3bit MLC, LPDDR4-1866 512 MB)                     |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| Machine-Learning_..._(over_500000_song_lyrics).tar        630,339,584 bytes |        1,839s |        7,455 MB | rz.exe a -d 1023M "%1.rz" "%1"                                                                                    |
|                                                            91,722,824 bytes |        2,403s |        5,539 MB | i5-7200U 3.1GHz, 8GB DDR4 2x1066MHz, SSD 500GB Samsung 860 EVO (3bit MLC, LPDDR4-1866 512 MB)                     |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| Machine-Learning_..._(over_500000_song_lyrics).tar        630,339,584 bytes |          496s |        8,345 MB | "7za_x64_v1900.exe" a -t7z -mx9 -md=30 "%1.MX9Dict1024.7z" "%1"                                                   |
|                                                           103,355,628 bytes |          607s |        6,441 MB | i5-7200U 3.1GHz, 8GB DDR4 2x1066MHz, SSD 500GB Samsung 860 EVO (3bit MLC, LPDDR4-1866 512 MB)                     |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| Machine-Learning_..._(over_500000_song_lyrics).tar        630,339,584 bytes |          417s |        1,420 MB | bcrush_ICL150_64bit.exe --optimal "%1" "%1.optimal.bcrush"                                                        |
|                                                           150,924,735 bytes |          416s |        1,362 MB | i5-7200U 3.1GHz, 8GB DDR4 2x1066MHz, SSD 500GB Samsung 860 EVO (3bit MLC, LPDDR4-1866 512 MB)                     |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| Machine-Learning_..._(over_500000_song_lyrics).tar        630,339,584 bytes |          203s |          852 MB | crush_xezz.exe -9 "%1" "%1.9.crush_xezz"                                                                          |
|                                                           165,173,195 bytes |          203s |          851 MB | i5-7200U 3.1GHz, 8GB DDR4 2x1066MHz, SSD 500GB Samsung 860 EVO (3bit MLC, LPDDR4-1866 512 MB)                     |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| Machine-Learning_..._(over_500000_song_lyrics).tar        630,339,584 bytes |        8,054s |       45,402 MB | BriefLZ_130_Intel_v19_64bit.exe --optimal -b2g "%1" "%1.blz"                                                      |
|                                                           129,595,542 bytes |        8,031s |        7,167 MB | i5-7200U 3.1GHz, 8GB DDR4 2x1066MHz, SSD 500GB Samsung 860 EVO (3bit MLC, LPDDR4-1866 512 MB)                     |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| Machine-Learning_..._(over_500000_song_lyrics).tar        630,339,584 bytes |        1,567s |       10,344 MB | brotli_ic19.exe -q 11 --large_window=30 "%1" -o "%1.bro"                                                          |
|                                                           102,237,706 bytes |        1,567s |        5,398 MB | i5-7200U 3.1GHz, 8GB DDR4 2x1066MHz, SSD 500GB Samsung 860 EVO (3bit MLC, LPDDR4-1866 512 MB)                     |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

And print-friendly 6 PDF pages: http://www.satanichi.net/The_Zennish_Microdeduplicator_9_pages.pdf

Really strong textual compressor BriefLZ 1.3.0 proves to be, really nice!

[jibsen]

Thanks once again, some interesting results. lzturbo 29 is at an interesting performance point, shame it isn't open source.

[Sanmayce]

Indeed, LZTurbo simply amazes, it is a diamant - my favorite decompressor; Hamid shared that (having profiled it) only uncached accesses prevent it from exploding, which makes me think that on future CPU-RAM subsystems with lot of cache it will reach new unseen speeds.

Oodle 'Mermaid' also is amazing, kinda these two performers have left the rest in the dust.

The duo lzbench and turbobench brings very informative roster and serves as a ... wake up call.

Will continue to enrich the benchmarks with: PGN: www.pgnmentor.com_242_players.tar 247 MB PDF: Jewish_Review_ofBooks-_2010-2018_completecollection(36_issues_TRUE-PDF).tar 446 MB DNA: www.ncbi.nlm.nih.govDragonfly(Ladona_fulva)_whole_genome_shotgun.tar 929 MB UTF8: http_unbound.biola.edu103-bibles(40+languages).tar 601 MB

[Sanmayce]

In order one to be able to play, temporarily (my Internet drive is full) 3 of above files are here: https://drive.google.com/file/d/1lT5QgRgTKi5sGY6LjpwRlo0yfrZntgVe/view?usp=sharing

[Sanmayce]

Hi, BriefLZ v1.3.0 is indeed very good, finally was able to benchmark the 2GB block with two popular 1GB testfiles: http://www.sanmayce.com/Nakamichi/index.html#2020Dec07