Closed Sanmayce closed 3 years ago
Thanks for the feedback and interesting results!
The problem is that blz_ssparse_workmem_size()
returns an unsigned long
, which is 32-bit on Windows, and the calculation overflows at 4096/(3*4) ~= 341MiB.
A quick fix would be to make all the workmem size functions use size_t
instead, but you might want to hold off on testing data this size with ssparse, since I am working on a new parser that should be faster for this type of data.
Thanks for the feedback and interesting results!
Always.
The problem is that blz_ssparse_workmem_size() returns an unsigned long, which is 32-bit on Windows, and the calculation overflows at 4096/(3*4) ~= 341MiB.
Good to know, thought that it might be something of that sort, I myself still use randomly both uint64_t or unsigned long long (aiming to have 32bit executables with 64bit variables), yet some such remnants are left here and there.
A quick fix would be to make all the workmem size functions use size_t instead, but you might want to hold off on testing data this size with ssparse, since I am working on a new parser that should be faster for this type of data.
Nice, will wait as long as it takes, Dr. Mahoney's ZPAQ 211 and your incoming BriefLZ (?) will be used as mainstay tools - in order to draw the baseline of non-entropy compressors in my future TEXTORAMIC SHOWDOWN. Not in a hurry, at all, a perennial benchmarker, here. Looking forward testing your latest-n-best.
The thing that stops me (or rather delays me), to enrich the TEXTORAMIC SHOWDOWN, is the lack of enough computational power :(
My observations are that BriefLZ's ssparse beats Nakamichi almost everywhere, with few exceptions (some English language .TXT corpora).
These are the ones which I tried so far:
E:\_TEXTUAL_MADNESS_bare-minimum_2020-Jan-05\BLZ>dir/on
03/26/2019 10:45 PM 42,920,232 (Dictionary_Specification_Language_(ABBYY_Software_House))_Hanyu_Cihai_new_Sea-of-Words_(Zho-Zho).dsl
03/26/2019 10:45 PM 14,336,148 (Dictionary_Specification_Language_(ABBYY_Software_House))_Hanyu_Cihai_new_Sea-of-Words_(Zho-Zho).dsl.Nakamichi
03/26/2019 10:45 PM 13,401,238 (Dictionary_Specification_Language_(ABBYY_Software_House))_Hanyu_Cihai_new_Sea-of-Words_(Zho-Zho).dsl.x_b300.blz
03/26/2019 10:45 PM 52,078,592 Agatha_Christie_108-ebooks_TXT.tar
03/26/2019 10:45 PM 12,283,940 Agatha_Christie_108-ebooks_TXT.tar.Nakamichi
03/26/2019 10:45 PM 11,818,902 Agatha_Christie_108-ebooks_TXT.tar.x_b300.blz
03/26/2019 10:45 PM 15,583,440 Arabian_Nights_complete.html
03/26/2019 10:45 PM 5,039,053 Arabian_Nights_complete.html.Nakamichi
03/26/2019 10:45 PM 4,834,828 Arabian_Nights_complete.html.x_b300.blz
03/26/2019 10:45 PM 193,367,552 Big_Soviet_Encyclopedia_in_30_volumes_(1239-HTMs).tar
03/26/2019 10:45 PM 60,929,805 Big_Soviet_Encyclopedia_in_30_volumes_(1239-HTMs).tar.Nakamichi
03/26/2019 10:45 PM 56,356,825 Big_Soviet_Encyclopedia_in_30_volumes_(1239-HTMs).tar.x_b300.blz
03/26/2019 10:45 PM 55,886,848 Charles_Dickens_in_30_volumes_(Russian_UTF-8).tar
03/26/2019 10:45 PM 12,042,476 Charles_Dickens_in_30_volumes_(Russian_UTF-8).tar.Nakamichi
03/26/2019 10:45 PM 11,973,291 Charles_Dickens_in_30_volumes_(Russian_UTF-8).tar.x_b300.blz
03/26/2019 10:45 PM 42,935,960 Complete_Works_of_Charles_Dickens.txt
03/26/2019 10:45 PM 13,448,970 Complete_Works_of_Charles_Dickens.txt.Nakamichi
03/26/2019 10:45 PM 13,490,362 Complete_Works_of_Charles_Dickens.txt.x_b300.blz
03/26/2019 10:45 PM 40,303,104 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
03/26/2019 10:45 PM 9,077,197 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.Nakamichi
03/26/2019 10:45 PM 8,851,573 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.x_b300.blz
03/26/2019 10:45 PM 11,520,000 ftp.gnu.org_grep-3.3.tar
03/26/2019 10:45 PM 2,163,309 ftp.gnu.org_grep-3.3.tar.Nakamichi
03/26/2019 10:45 PM 1,722,565 ftp.gnu.org_grep-3.3.tar.x_b300.blz
03/26/2019 10:45 PM 9,378,116 Hitler_-_Speeches_and_Proclamations_-_Max_Domarus.txt
03/26/2019 10:45 PM 2,939,813 Hitler_-_Speeches_and_Proclamations_-_Max_Domarus.txt.Nakamichi
03/26/2019 10:45 PM 2,807,518 Hitler_-_Speeches_and_Proclamations_-_Max_Domarus.txt.x_b300.blz
03/26/2019 10:45 PM 3,535,360 LEXX_subtitles.tar
03/26/2019 10:45 PM 1,173,012 LEXX_subtitles.tar.Nakamichi
03/26/2019 10:45 PM 1,085,000 LEXX_subtitles.tar.x_b300.blz
03/26/2019 10:45 PM 206,908,949 OSHO.TXT
03/26/2019 10:45 PM 51,502,498 OSHO.TXT.Nakamichi
03/26/2019 10:45 PM 51,101,509 OSHO.TXT.x_b300.blz
03/26/2019 10:45 PM 167,235,072 Star_Wars_321-books.tar
03/26/2019 10:45 PM 40,420,503 Star_Wars_321-books.tar.Nakamichi
03/26/2019 10:45 PM 39,755,386 Star_Wars_321-books.tar.x_b300.blz
03/26/2019 10:45 PM 69,000,704 Stephen_King_81_novels_TXT.tar
03/26/2019 10:45 PM 21,818,393 Stephen_King_81_novels_TXT.tar.Nakamichi
03/26/2019 10:45 PM 21,730,474 Stephen_King_81_novels_TXT.tar.x_b300.blz
03/26/2019 10:45 PM 33,106,082 Taishukan's_Genius_English-Japanese_Dictionary_(En-Jp).dsl
03/26/2019 10:45 PM 7,021,323 Taishukan's_Genius_English-Japanese_Dictionary_(En-Jp).dsl.Nakamichi
03/26/2019 10:45 PM 6,543,791 Taishukan's_Genius_English-Japanese_Dictionary_(En-Jp).dsl.x_b300.blz
03/26/2019 10:45 PM 3,714,387 The_Complete_Sherlock_Holmes_-_Doyle_Arthur_Conan.txt
03/26/2019 10:45 PM 1,286,110 The_Complete_Sherlock_Holmes_-_Doyle_Arthur_Conan.txt.Nakamichi
03/26/2019 10:45 PM 1,229,620 The_Complete_Sherlock_Holmes_-_Doyle_Arthur_Conan.txt.x_b300.blz
E:\_TEXTUAL_MADNESS_bare-minimum_2020-Jan-05\BLZ>
By the way, you can see my latest-and-finest .ZIP package (.C source and how to compile) here: http://www.sanmayce.com/Nakamichi/index.html#DOWNLOAD
Nakamichi_Vanilla_ICL150_64bit_2020-Feb-02.exe "%1" "%1.Nakamichi" 24 12000 i
In case of 16GB RAM available, above line can compress files less than 200MB at 50KB/s, roughly speaking.
After some 400h will share what RAZOR, bwtturbo, Nakamichi, ZPAQ 511 ... can do to the 'SPETSNAZ' ALL-RUSSIAN corpus which will enrich the picture of Textual Compression: https://pbs.twimg.com/media/EP0OyvjXsAIvMDe?format=jpg&name=large
By the way, I noticed crushxezz in your results above, you could try bcrush as well.
As always, my intent is all benchmarks to be reproducible, so I uploaded it to my Internet drive: https://drive.google.com/file/d/1YKKVTxqxf8jwl16ay9GftJL6wxVT0Lex/view?usp=sharing
Testmachine: i7-3630QM 3.4GHz, 16GB DDR3 2x800MHz. Currently I played with 4 corpora:
C.Time | Size | Filename | Command Line
-------|-------------|--------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------
| 832,290,816 | book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar |
405s | 166,179,937 | book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar.9.crush_xezz | crush_xezz.exe -9 "%1" "%1.9.crush_xezz"
677s | 101,121,392 | book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar.L22W2GB.zst | zstd-v1.4.4-win64.exe --single-thread --ultra -22 --zstd=wlog=31,clog=30,hlog=30,slog=26 "%1" -o "%1.L22W2GB.zst"
258s | 121,901,223 | book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar.method211.zpaq | "zpaq_v7.05_x64.exe" add "%1.method211.zpaq" "%1" -method 211 -threads 1
1881s | 87,527,794 | book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar.method511.zpaq | "zpaq_v7.05_x64.exe" add "%1.method511.zpaq" "%1" -method 511 -threads 1
684s | 97,805,186 | book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar.MX9Dict1024.7z | "7za_x64_v1900.exe" a -t7z -mx9 -md=30 "%1.MX9Dict1024.7z" "%1"
N.A. | ???,???,??? | book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar.Nakamichi | Nakamichi_Vanilla_ICL150_64bit_2020-Feb-02.exe "%1" "%1.Nakamichi" 24 92000 e
784s | 154,632,025 | book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar.optimal.bcrush | bcrush_ICL150_64bit.exe --optimal "%1" "%1.optimal.bcrush"
2812s | 82,503,337 | book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar.rz | rz.exe a -d 1023M "%1.rz" "%1"
93s | 76,516,343 | book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar.zv | bwtturbo -59 -t0 -b1024 "%1" "%1.zv"
C.Time | Size | Filename | Command Line
-------|-------------|--------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------
| 107,784,192 | Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar |
46s | 35,831,237 | Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.9.crush_xezz | crush_xezz.exe -9 "%1" "%1.9.crush_xezz"
103s | 27,968,085 | Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.L22W2GB.zst | zstd-v1.4.4-win64.exe --single-thread --ultra -22 --zstd=wlog=31,clog=30,hlog=30,slog=26 "%1" -o "%1.L22W2GB.zst"
39s | 33,141,734 | Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.method211.zpaq | "zpaq_v7.05_x64.exe" add "%1.method211.zpaq" "%1" -method 211 -threads 1
360s | 20,539,721 | Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.method511.zpaq | "zpaq_v7.05_x64.exe" add "%1.method511.zpaq" "%1" -method 511 -threads 1
77s | 27,535,075 | Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.MX9Dict1024.7z | "7za_x64_v1900.exe" a -t7z -mx9 -md=30 "%1.MX9Dict1024.7z" "%1"
N.A. | 35,968,472 | Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.Nakamichi | Nakamichi_Vanilla_ICL150_64bit_2020-Feb-02.exe "%1" "%1.Nakamichi" 24 92000 e
53s | 33,838,607 | Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.optimal.bcrush | bcrush_ICL150_64bit.exe --optimal "%1" "%1.optimal.bcrush"
293s | 24,345,365 | Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.rz | rz.exe a -d 1023M "%1.rz" "%1"
14s | 22,279,538 | Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.zv | bwtturbo -59 -t0 -b1024 "%1" "%1.zv"
C.Time | Size | Filename | Command Line
-------|-------------|--------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------
| 325,071,872 | Star_Trek_-_737_Ebooks.tar |
144s | 100,553,919 | Star_Trek_-_737_Ebooks.tar.9.crush_xezz | crush_xezz.exe -9 "%1" "%1.9.crush_xezz"
377s | 72,762,430 | Star_Trek_-_737_Ebooks.tar.L22W2GB.zst | zstd-v1.4.4-win64.exe --single-thread --ultra -22 --zstd=wlog=31,clog=30,hlog=30,slog=26 "%1" -o "%1.L22W2GB.zst"
119s | 87,925,703 | Star_Trek_-_737_Ebooks.tar.method211.zpaq | "zpaq_v7.05_x64.exe" add "%1.method211.zpaq" "%1" -method 211 -threads 1
1097s | 56,030,908 | Star_Trek_-_737_Ebooks.tar.method511.zpaq | "zpaq_v7.05_x64.exe" add "%1.method511.zpaq" "%1" -method 511 -threads 1
300s | 72,018,734 | Star_Trek_-_737_Ebooks.tar.MX9Dict1024.7z | "7za_x64_v1900.exe" a -t7z -mx9 -md=30 "%1.MX9Dict1024.7z" "%1"
N.A. | 92,975,501 | Star_Trek_-_737_Ebooks.tar.Nakamichi | Nakamichi_Vanilla_ICL150_64bit_2020-Feb-02.exe "%1" "%1.Nakamichi" 24 92000 e
171s | 94,122,899 | Star_Trek_-_737_Ebooks.tar.optimal.bcrush | bcrush_ICL150_64bit.exe --optimal "%1" "%1.optimal.bcrush"
1124s | 63,070,759 | Star_Trek_-_737_Ebooks.tar.rz | rz.exe a -d 1023M "%1.rz" "%1"
45s | 59,411,519 | Star_Trek_-_737_Ebooks.tar.zv | bwtturbo -59 -t0 -b1024 "%1" "%1.zv"
C.Time | Size | Filename | Command Line
-------|-------------|--------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------
| 630,339,584 | Machine-Learning_..._(over_500000_song_lyrics).tar |
184s | 165,173,195 | Machine-Learning_..._(over_500000_song_lyrics).tar.9.crush_xezz | crush_xezz.exe -9 "%1" "%1.9.crush_xezz"
667s | 108,281,760 | Machine-Learning_..._(over_500000_song_lyrics).tar.L22W2GB.zst | zstd-v1.4.4-win64.exe --single-thread --ultra -22 --zstd=wlog=31,clog=30,hlog=30,slog=26 "%1" -o "%1.L22W2GB.zst"
206s | 125,295,628 | Machine-Learning_..._(over_500000_song_lyrics).tar.method211.zpaq | "zpaq_v7.05_x64.exe" add "%1.method211.zpaq" "%1" -method 211 -threads 1
2043s | 86,008,151 | Machine-Learning_..._(over_500000_song_lyrics).tar.method511.zpaq | "zpaq_v7.05_x64.exe" add "%1.method511.zpaq" "%1" -method 511 -threads 1
514s | 103,355,628 | Machine-Learning_..._(over_500000_song_lyrics).tar.MX9Dict1024.7z | "7za_x64_v1900.exe" a -t7z -mx9 -md=30 "%1.MX9Dict1024.7z" "%1"
N.A. | 145,750,630 | Machine-Learning_..._(over_500000_song_lyrics).tar.Nakamichi | Nakamichi_Vanilla_ICL150_64bit_2020-Feb-02.exe "%1" "%1.Nakamichi" 24 92000 e
392s | 150,924,735 | Machine-Learning_..._(over_500000_song_lyrics).tar.optimal.bcrush | bcrush_ICL150_64bit.exe --optimal "%1" "%1.optimal.bcrush"
1748s | 91,722,822 | Machine-Learning_..._(over_500000_song_lyrics).tar.rz | rz.exe a -d 1023M "%1.rz" "%1"
68s | 95,432,407 | Machine-Learning_..._(over_500000_song_lyrics).tar.zv | bwtturbo -59 -t0 -b1024 "%1" "%1.zv"
Note: The 'SPETSNAZ' is currently being crunched by Nakamichi on the same laptop:
Compressing 832,290,816 bytes ...
-; Each rotation means 64KB are encoded; Speed: 0,000,387 B/s; Done 28%; Compression Ratio: 8.03:1; Matches(16/24/48): 1,954,450/1,508,433/82,855; 128[+] long matches: 174,568; ETA: 425.12 hours
Joergen , I have a question, BLOCKWISE, what will be the difference between BriefLZ and BCRUSH? What is the short description of these two LZSS?
For example, my Nakamichi's DIZ: LZSS with 1TB sliding window with greedy parsing.
Crush is LZSS using a 2MiB sliding window on 64MiB blocks. Under the same restrictions, the encoding used in crush is more efficient than that of BriefLZ, Also, crush has a minimum match length of 3, which works better on some types of data.
BriefLZ is LZSS with a block size up to 4GiB (default 1MiB), and the window size is equal to the block size for most of the algorithms included.
bcrush --optimal enwik9 -> 264.504.187
blzpack -b64m --optimal enwik9 -> 271.277.397
blzpack -b1g --optimal enwik9 -> 252.991.647
Thanks. Looking forward seeing BriefLZ (with -b1024) results for the 10GB wikis...
BriefLZ 1.3.0 is up, including a fix for this.
Thank you Jørgen for your nifty new performer, surely will play with it ... exhaustively ... but after ~340 hours - the 'SPETSNAZ' has to complete.
makeEXE.bat:
cd example
icl /Qstd=c99 /O3 /I..\include blzpack.c parg.c ..\src\brieflz.c ..\src\depack.c ..\src\depacks.c
copy blzpack.exe ..\BriefLZ_130_Intel_v19_64bit.exe /y
_MakeEXE_Nakamichi_ICL.bat:
icl /TP /O3 /arch:SSE4.1 Nakamichi_Ryuugan-ditto-1TB_btree.c -D_N_XMM -D_N_prefetch_4096 -D_N_HIGH_PRIORITY -D_icl_mumbo_jumbo_ /FAcs -DHashInBITS=24 -DHashChunkSizeInBITS=24 -DRAMpoolInKB=5120 -DBtreeHEURISTIC -D_WIN32_ENVIRONMENT_ -DLongestLineInclusive=64 -DSpeedUpBuilding=32 -DLITE -D_N_alone
copy Nakamichi_Ryuugan-ditto-1TB_btree.exe Nakamichi_Vanilla_LITE_ICL190_64bit.exe /y
A quick showdown:
Satanichi (a.k.a. Nakamichi_2020-Feb-02_Defragmented_boosted_STDOUT_LITE) vs BriefLZ_130
The bottomline first, 2895/106=27.3x slower compression rate :(
Note1: The latest-and-finest revision of Nakamichi is already called Satanichi - it uses only B-trees and discards all memmem() invocations. It is the fastest and most pure Nakamichi, the defragmented external B-trees will show how much IOPS modern SSDs with NAND and Xpoint deliver. Note2: If there is physical RAM in spades then Satanichi is gonna scream when building all the B-trees in one pass only, simply the compile option '-DSpeedUpBuilding=32' has to be increased e.g. to 14000.
The corpus 'Judaica' on i7-3630QM @3.4GHz, 16GB DDR3 @1600MHz:
F:\brieflz-1.3.0_Intel_Bench>timer64 BriefLZ_130_Intel_v19_64bit.exe --optimal -b500m "Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar" "Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.blz"
Kernel Time = 0.953 = 0%
User Time = 105.093 = 99%
Process Time = 106.046 = 99% Virtual Memory = 11086 MB
Global Time = 106.142 = 100% Physical Memory = 2593 MB
F:\brieflz-1.3.0_Intel_Bench>timer64 Nakamichi_Vanilla_LITE_Intel_v19_64bit.exe "Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar" "Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.Nakamichi" 24 9000 i
Nakamichi 'Ryuugan-ditto-1TB', written by Kaze, inspired by Haruhiko Okumura sharing, based on Nobuo Ito's LZSS source, muffinesque tips by m^2, Jim Dempsey and Kirill Kryukov enforced.
Downloadable at: https://twitter.com/Sanmayce/status/1211005691873439744
Note0: Nakamichi 'Dragoneye', (AUTO-RESETTING PASSES and DEFRAGMENTED B-trees revision) is 100% FREE, licenseless that is.
Note1: Hamid Buzidi's LzTurbo ([a] FASTEST [Textual] Decompressor, Levels 19/29/39) retains kingship, his TurboBench (2017-Apr-07) proves the supremacy of LzTurbo, Turbo-Amazing!
Note2: Conor Stokes' LZSSE2 ([a] FASTEST Textual Decompressor, Level 17) is embedded, all credits along with many thanks go to him.
Note3: The matchfinder is either 'Railgun_Trolldom' (matches longer than 18, except 36 and 64) or Leprechaun's B-tree order 3.
Note4: Instead of '_mm_loadu_si128' '_mm_lddqu_si128' is used.
Note5: Maximum compression ratio is 44:1, for 704 bytes long matches within 1TB Sliding Window.
Note6: Please send me (at sanmayce@sanmayce.com) decompression results obtained on machines with fast CPU-RAM subsystems.
Note7: In this compile, clock() was replaced with time() - to counter bigtime stats misreporting.
Note8: Multi-way hashing allows each KeySize to occupy its own HASH pool, thus less RAM is in use - the LEAF is smaller.
Note9: In this revision, B-tree heuristics are in use, allowing skipping many unnecessary memmem() invocations.
NoteA: The file being compressed should be 4 bytes or longer due to Building-Blocks being in range 4..18, 24,28,36,48,64.
NoteB: In this compile, the keysizes in the LEAF are not HEXed i.e. not doubled. Also, the hash in use: FNV1A_Pippip_Yurii, the fastest one known to me.
NoteC: In this latest (2020-Feb-02) compile, SHA3-224 is enabled at compile time, it is optional, that is.
Current priority class is REALTIME_PRIORITY_CLASS.
This compile uses B-trees only, no memmem() invocations - it compresses worse but much faster.
Allocating Source-Buffer 102 MB ...
Allocating Source-Buffer 102 MB (REVERSED) ...
Allocating Target-Buffer 134 MB ...
Leprechaun: Memory pool for B-trees is 9,000 MB.
Leprechaun: In this revision 128MB 13-way hash is used which results in 13 x 16,777,216 internal B-Trees of order 3.
Leprechaun: Allocating HASH memory 1,879,048,257 bytes ... OK
Leprechaun: Allocating memory for B-trees 9001 MB ... OK
Leprechaun: Size of input file: 107,784,192
Leprechaun: Inserting keys/BBs of order 004 into B-trees, free RAM in B-tree pool is 00,009,000 MB; Pass #000,001 of 000,001 ... DONE; 00,000,388,979 B-trees have been rooted so far.
Leprechaun: Inserting keys/BBs of order 006 into B-trees, free RAM in B-tree pool is 00,008,981 MB; Pass #000,001 of 000,001 ...
Leprechaun: Failure! Increment 'Memory for B-trees'!
Automatically increasing number of passes in order to fit B-trees into Target Buffer.
Leprechaun: Inserting keys/BBs of order 006 into B-trees, free RAM in B-tree pool is 00,008,914 MB; Pass #000,002 of 000,002 ... DONE; 00,002,953,386 B-trees have been rooted so far.
Leprechaun: Inserting keys/BBs of order 008 into B-trees, free RAM in B-tree pool is 00,008,848 MB; Pass #000,001 of 000,002 ...
Leprechaun: Failure! Increment 'Memory for B-trees'!
Automatically increasing number of passes in order to fit B-trees into Target Buffer.
Leprechaun: Inserting keys/BBs of order 008 into B-trees, free RAM in B-tree pool is 00,008,848 MB; Pass #000,001 of 000,004 ...
Leprechaun: Failure! Increment 'Memory for B-trees'!
Automatically increasing number of passes in order to fit B-trees into Target Buffer.
Leprechaun: Inserting keys/BBs of order 008 into B-trees, free RAM in B-tree pool is 00,008,579 MB; Pass #000,008 of 000,008 ... DONE; 00,008,259,270 B-trees have been rooted so far.
Leprechaun: Inserting keys/BBs of order 010 into B-trees, free RAM in B-tree pool is 00,008,541 MB; Pass #000,001 of 000,008 ...
Leprechaun: Failure! Increment 'Memory for B-trees'!
Automatically increasing number of passes in order to fit B-trees into Target Buffer.
Leprechaun: Inserting keys/BBs of order 010 into B-trees, free RAM in B-tree pool is 00,008,104 MB; Pass #000,016 of 000,016 ... DONE; 00,015,485,628 B-trees have been rooted so far.
Leprechaun: Inserting keys/BBs of order 012 into B-trees, free RAM in B-tree pool is 00,008,075 MB; Pass #000,001 of 000,016 ...
Leprechaun: Failure! Increment 'Memory for B-trees'!
Automatically increasing number of passes in order to fit B-trees into Target Buffer.
Leprechaun: Inserting keys/BBs of order 012 into B-trees, free RAM in B-tree pool is 00,007,535 MB; Pass #000,032 of 000,032 ... DONE; 00,023,418,126 B-trees have been rooted so far.
Leprechaun: Inserting keys/BBs of order 014 into B-trees, free RAM in B-tree pool is 00,006,965 MB; Pass #000,032 of 000,032 ... DONE; 00,031,128,052 B-trees have been rooted so far.
Leprechaun: Inserting keys/BBs of order 016 into B-trees, free RAM in B-tree pool is 00,006,439 MB; Pass #000,032 of 000,032 ... DONE; 00,038,000,585 B-trees have been rooted so far.
Leprechaun: Inserting keys/BBs of order 018 into B-trees, free RAM in B-tree pool is 00,006,422 MB; Pass #000,001 of 000,032 ...
Leprechaun: Failure! Increment 'Memory for B-trees'!
Automatically increasing number of passes in order to fit B-trees into Target Buffer.
Leprechaun: Inserting keys/BBs of order 018 into B-trees, free RAM in B-tree pool is 00,005,980 MB; Pass #000,064 of 000,064 ... DONE; 00,043,741,138 B-trees have been rooted so far.
Leprechaun: Inserting keys/BBs of order 036 into B-trees, free RAM in B-tree pool is 00,005,973 MB; Pass #000,001 of 000,064 ...
Leprechaun: Failure! Increment 'Memory for B-trees'!
Automatically increasing number of passes in order to fit B-trees into Target Buffer.
Leprechaun: Inserting keys/BBs of order 036 into B-trees, free RAM in B-tree pool is 00,005,908 MB; Pass #000,128 of 000,128 ... DONE; 00,044,346,052 B-trees have been rooted so far.
Leprechaun: Inserting keys/BBs of order 064 into B-trees, free RAM in B-tree pool is 00,005,888 MB; Pass #000,128 of 000,128 ... DONE; 00,044,464,928 B-trees have been rooted so far.
Leprechaun: Inserting keys/BBs of order 024 into B-trees, free RAM in B-tree pool is 00,005,658 MB; Pass #000,128 of 000,128 ... DONE; 00,047,139,473 B-trees have been rooted so far.
Leprechaun: Inserting keys/BBs of order 028 into B-trees, free RAM in B-tree pool is 00,005,514 MB; Pass #000,128 of 000,128 ... DONE; 00,048,670,461 B-trees have been rooted so far.
Leprechaun: Inserting keys/BBs of order 048 into B-trees, free RAM in B-tree pool is 00,005,479 MB; Pass #000,128 of 000,128 ... DONE; 00,048,925,115 B-trees have been rooted so far.
Leprechaun: Total Searches-n-Inserts Per Second: 36,471,808 SNIPS = 90,486,557,125 keys in 2,481 seconds
Leprechaun: RAM needed to house B-trees (relative to the file being ripped): 34N = 3,521MB
Compressing 107,784,192 bytes ...
|; Each rotation means 64KB are encoded; Speed: 0,292,098 B/s; Done 100%; Compression Ratio: 2.99:1; Matches(16/24/48): 1,194,147/371,659/29,877; 128[+] long matches: 0; ETA: 0.00 hours
NumberOfFullLiterals (lower-the-better): 4712
Tsuyo_HEURISTIC_APPLIED_thrice_back-to-back: 0
NumberOf(Tiny)Matches[Micro]Window (4)[16B]: 0
NumberOfMatches[Bheema]Window [128GB window]: 4406
RAM-to-RAM performance: 292098 B/s.
Compressed to 36,035,319 bytes.
Source-file-Hash(FNV1A_YoshimitsuTRIAD) = 0x5bd3,767b
Target-file-Hash(FNV1A_YoshimitsuTRIAD) = 0x23db,b784
Decompressing 36,035,319 (being the compressed stream) bytes ...
RAM-to-RAM performance: 590 MB/s.
Verification (input and output sizes match) OK.
Verification (input and output blocks match) OK.
Kernel Time = 5.625 = 0%
User Time = 2888.140 = 99%
Process Time = 2893.765 = 99% Virtual Memory = 11156 MB
Global Time = 2895.675 = 100% Physical Memory = 11136 MB
F:\brieflz-1.3.0_Intel_Bench>dir
04/06/2017 21:44 107,784,192 Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar
09/15/2018 20:09 34,820,909 Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.blz
09/15/2018 20:57 36,035,319 Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar.Nakamichi
F:\brieflz-1.3.0_Intel_Bench>
After finishing 'SPETSNAZ' will run the new BriefLZ with many corpora...
The .C source and ICL v19.0 and GCC v7.3.0 binaries are here: http://www.sanmayce.com/Nakamichi/Satanichi_aka_Nakamichi_2020-Feb-02_Defragmented_boosted_STDOUT_LITE.zip
Glad to hear it appears to be working!
I don't think it will make much of a difference, but you can compile with /DNDEBUG
to disable the assertions around the BriefLZ code.
Two more testfiles are done, on i5-7200U 3.1GHz, 8GB DDR4 2x1066MHz:
A Russian prose UTF-8 .TXT tarred corpus:
D:\TEXTORAMIC_part>lzbench18 -c4 -i1,15 -o3 -elzham,4/slz_deflate,3/slz_gzip,3/slz_zlib,3/tornado,1,16/bzip2,1,9/zstd,1,22/zstdLDM,1,22/fastlzma2,1,10/brotli,0,1,3,5,11/blosclz,1,9/crush,2/csc,5/density,1,3/fastlz,2/gipfeli/lzo1b,999/libdeflate,1,12/lz4,1,12/lizard,11,19,21,29,31,39,41,49/lzf,1/lzfse/lzg,9/lzjb/lzlib,9/lzma,0,1,9/lzrw,5/lzsse2,17/lzsse4,17/lzsse8,17/lzvn/pithy,9/quicklz,3/snappy/slz_zlib,3/ucl_nrv2b,9/ucl_nrv2d,9/ucl_nrv2e,9/xpack,1,9/xz,9/yalz77,12/yappy,99/zlib,1,5,9/zling,4/shrinker/wflz/lzmat ""book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar""
lzbench 1.8 (64-bit Windows) Assembled by P.Skibinski
The results sorted by column number 4:
Compressor name Compress. Decompress. Orig. size Compr. size Ratio Filename
zstd 1.4.3 -22 1.36 MB/s 892 MB/s 832290816 106285862 12.77 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
zstdLDM 1.4.3 -22 1.37 MB/s 893 MB/s 832290816 106556756 12.80 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
xz 5.2.4 -9 1.13 MB/s 140 MB/s 832290816 107796983 12.95 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lzma 19.00 -9 1.10 MB/s 139 MB/s 832290816 107849433 12.96 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
tornado 0.6a -16 1.48 MB/s 288 MB/s 832290816 108105061 12.99 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lzham 1.0 -d26 -4 0.80 MB/s 313 MB/s 832290816 108470968 13.03 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
fastlzma2 1.0.1 -10 2.36 MB/s 120 MB/s 832290816 112251678 13.49 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lzlib 1.11 -9 1.05 MB/s 92 MB/s 832290816 113146693 13.59 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
csc 2016-10-13 -5 1.39 MB/s 125 MB/s 832290816 116667138 14.02 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
BriefLZ_130_Intel_v19_64bit --optimal -b2g 131370690 ! outside lzbench !
brotli 2019-10-01 -11 0.45 MB/s 551 MB/s 832290816 132070667 15.87 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lizard 1.0 -49 1.50 MB/s 1044 MB/s 832290816 134883363 16.21 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
bzip2 1.0.8 -9 10 MB/s 26 MB/s 832290816 145658823 17.50 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
Satanichi_GCC730_64bit.exe 1124 MB/s 149041296 ! outside lzbench !
lizard 1.0 -29 1.55 MB/s 1312 MB/s 832290816 158642651 19.06 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
crush 1.0 -2 0.21 MB/s 407 MB/s 832290816 167891024 20.17 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
zling 2018-10-12 -4 31 MB/s 194 MB/s 832290816 172424277 20.72 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
bzip2 1.0.8 -1 11 MB/s 39 MB/s 832290816 177659512 21.35 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
brotli 2019-10-01 -5 28 MB/s 523 MB/s 832290816 179403587 21.56 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
xpack 2016-06-02 -9 4.38 MB/s 948 MB/s 832290816 207604165 24.94 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lizard 1.0 -39 4.31 MB/s 1832 MB/s 832290816 209834821 25.21 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
libdeflate 1.3 -12 3.67 MB/s 1038 MB/s 832290816 213213594 25.62 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lzsse2 2019-04-18 -17 2.43 MB/s 3926 MB/s 832290816 216869808 26.06 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lzsse4 2019-04-18 -17 2.53 MB/s 3909 MB/s 832290816 217770793 26.17 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
brotli 2019-10-01 -3 78 MB/s 436 MB/s 832290816 219485981 26.37 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lzsse8 2019-04-18 -17 2.45 MB/s 3527 MB/s 832290816 224389638 26.96 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
zlib 1.2.11 -9 4.17 MB/s 299 MB/s 832290816 227825566 27.37 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
fastlzma2 1.0.1 -1 13 MB/s 60 MB/s 832290816 232444725 27.93 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lzma 19.00 -1 16 MB/s 62 MB/s 832290816 233717316 28.08 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
yalz77 2015-09-19 -12 30 MB/s 376 MB/s 832290816 237645327 28.55 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
ucl_nrv2e 1.03 -9 0.36 MB/s 311 MB/s 832290816 243105097 29.21 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lzma 19.00 -0 17 MB/s 57 MB/s 832290816 244108780 29.33 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
ucl_nrv2d 1.03 -9 0.30 MB/s 315 MB/s 832290816 245933936 29.55 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lzo1b 2.10 -999 4.20 MB/s 578 MB/s 832290816 246693493 29.64 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lizard 1.0 -19 4.35 MB/s 2617 MB/s 832290816 248001307 29.80 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
zlib 1.2.11 -5 21 MB/s 275 MB/s 832290816 251891158 30.26 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
ucl_nrv2b 1.03 -9 0.34 MB/s 300 MB/s 832290816 252275771 30.31 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lzg 1.0.10 -9 0.18 MB/s 663 MB/s 832290816 254874747 30.62 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lizard 1.0 -41 152 MB/s 1148 MB/s 832290816 255405016 30.69 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
zstdLDM 1.4.3 -1 134 MB/s 413 MB/s 832290816 257066314 30.89 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
zstd 1.4.3 -1 210 MB/s 413 MB/s 832290816 257209725 30.90 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lzfse 2017-03-08 50 MB/s 531 MB/s 832290816 259743898 31.21 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
brotli 2019-10-01 -1 174 MB/s 356 MB/s 832290816 266460227 32.02 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lzmat 1.01 11 MB/s 299 MB/s 832290816 266952375 32.07 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
libdeflate 1.3 -1 135 MB/s 756 MB/s 832290816 285713451 34.33 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
brotli 2019-10-01 -0 217 MB/s 343 MB/s 832290816 287973706 34.60 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
xpack 2016-06-02 -1 119 MB/s 499 MB/s 832290816 298137386 35.82 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lizard 1.0 -21 176 MB/s 1742 MB/s 832290816 310381370 37.29 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lizard 1.0 -31 198 MB/s 1459 MB/s 832290816 315336786 37.89 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
zlib 1.2.11 -1 66 MB/s 244 MB/s 832290816 321781027 38.66 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
yappy 2014-03-22 -99 55 MB/s 1887 MB/s 832290816 322920818 38.80 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lzvn 2017-03-08 39 MB/s 773 MB/s 832290816 331212729 39.80 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
pithy 2011-12-24 -9 289 MB/s 1437 MB/s 832290816 338709477 40.70 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
gipfeli 2016-07-13 231 MB/s 444 MB/s 832290816 344610201 41.41 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lzrw 15-Jul-1991 -5 87 MB/s 405 MB/s 832290816 346324028 41.61 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
quicklz 1.5.0 -3 36 MB/s 557 MB/s 832290816 348569403 41.88 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
density 0.14.2 -3 420 MB/s 380 MB/s 832290816 370942860 44.57 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lizard 1.0 -11 245 MB/s 2063 MB/s 832290816 387848544 46.60 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lz4 1.9.2 310 MB/s 2283 MB/s 832290816 408975647 49.14 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lz4 1.9.2 309 MB/s 2282 MB/s 832290816 408975647 49.14 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
snappy 2019-09-30 193 MB/s 829 MB/s 832290816 423828918 50.92 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
tornado 0.6a -1 220 MB/s 269 MB/s 832290816 435175239 52.29 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
fastlz 0.1 -2 246 MB/s 385 MB/s 832290816 436415456 52.44 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lzf 3.6 -1 229 MB/s 470 MB/s 832290816 443098437 53.24 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
density 0.14.2 -1 1714 MB/s 1809 MB/s 832290816 445806192 53.56 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
slz_deflate 1.0.0 -3 203 MB/s 244 MB/s 832290816 453783256 54.52 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
slz_gzip 1.0.0 -3 184 MB/s 223 MB/s 832290816 453783262 54.52 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
slz_zlib 1.0.0 -3 169 MB/s 203 MB/s 832290816 453783274 54.52 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
slz_zlib 1.0.0 -3 175 MB/s 203 MB/s 832290816 453783274 54.52 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
blosclz 2.0.0 -9 215 MB/s 360 MB/s 832290816 497813982 59.81 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
lzjb 2010 174 MB/s 285 MB/s 832290816 539617346 64.84 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
wflz 2015-09-16 185 MB/s 463 MB/s 832290816 557630036 67.00 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
shrinker 0.1 228 MB/s 6557 MB/s 832290816 820229674 98.55 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
blosclz 2.0.0 -1 7352 MB/s 7534 MB/s 832290816 832290816 100.00 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
memcpy 10742 MB/s 11009 MB/s 832290816 832290816 100.00 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
D:\TEXTORAMIC_part>"turbobench_v18.05_-_build_04_May_2018" ""book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar"" -esnappy_c/yappy/bzip2/lzma,9d30:fb273:mf=bt4/oodle,89,91,95,99,111,115,119,129,139/lzturbo,19,12,10,29,22,20,39,32,30,59/brotli,11d30/zstd,1,5,12,22,22d30/lizard,11,19,21,29,31,39,41,49/trle -I3 -J31 -k1 -B2G
C Size ratio% C MB/s D MB/s Name File
86229509 10.4 4.35 28.08 lzturbo 59 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
97744466 11.7 0.86 148.98 lzma 9d30:fb273:mf=bt4 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
97853381 11.8 0.07 940.83 lzturbo 39 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
99153037 11.9 0.40 358.11 brotli 11d30 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
101289846 12.2 1.28 813.45 zstd 22d30 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
104062328 12.5 0.18 736.02 oodle 139 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
104380801 12.5 0.23 789.25 oodle 89 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
104402322 12.5 0.17 784.01 oodle 129 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
106451962 12.8 1.45 1608.63 zstd 22 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
110920918 13.3 0.07 1409.53 lzturbo 29 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
126495932 15.2 0.26 1532.43 oodle 99 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
131370690 BriefLZ_130_Intel_v19_64bit --optimal -b2g ! outside turbobench !
134883367 16.2 1.39 1004.33 lizard 49 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
136202664 16.4 0.49 1838.40 oodle 95 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
145658827 17.5 10.54 24.71 bzip2 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
148673910 17.9 0.27 2484.08 oodle 119 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
148929102 17.9 52.47 1202.12 lzturbo 32 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
149041296 1124 Satanichi_GCC730_64bit.exe ! outside turbobench !
158150115 19.0 0.53 2004.32 oodle 115 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
158642655 19.1 1.41 1241.57 lizard 29 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
167316896 20.1 7.59 1654.77 zstd 12 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
209834825 25.2 3.95 1696.33 lizard 39 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
224390328 27.0 83.35 679.06 zstd 5 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
231399459 27.8 48.79 1392.72 lzturbo 22 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
233761178 28.1 277.87 1009.25 lzturbo 30 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
247855856 29.8 0.07 3179.57 lzturbo 19 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
248001311 29.8 4.37 2362.27 lizard 19 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
253908003 30.5 196.22 2479.17 oodle 91 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
255405020 30.7 129.25 1102.80 lizard 41 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
257953411 31.0 166.80 767.78 zstd 1 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
277477721 33.3 64.90 3352.01 lzturbo 12 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
310381374 37.3 142.54 1659.84 lizard 21 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
315336790 37.9 154.72 1350.42 lizard 31 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
326323817 39.2 60.69 1804.80 yappy book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
343518192 41.3 408.19 1659.96 lzturbo 20 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
381026902 45.8 235.37 2868.88 oodle 111 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
387848548 46.6 182.29 2414.03 lizard 11 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
423797463 50.9 326.63 1653.17 snappy_c book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
478153400 57.5 378.77 2116.37 lzturbo 10 book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
830022523 99.7 196.24 2016.97 trle book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar
And the 'usual suspects':
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| Corpus Name Corpus Size | Global Time | Virtual Memory | Command Line |
| Compressed Size | Process Time | Physical Memory | Testmachine |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar 832,290,816 bytes | 1,282,007s | 9,601 MB | Satanichi_GCC730_64bit.exe "%1" "%1.Nakamichi" 26 80000 e |
| 149,041,296 bytes | 248,676s | 9,552 MB | i7-3630QM 3.4GHz, 16GB DDR3 2x800MHz, SSD 256GB Samsung 860 PRO (2bit MLC, LPDDR4-1866 512 MB) |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar 832,290,816 bytes | 260s | 5,387 MB | "zpaq_v7.05_x64.exe" add "%1.method211.zpaq" "%1" -method 211 -threads 1 |
| 121,901,223 bytes | 260s | 3,922 MB | i7-3630QM 3.4GHz, 16GB DDR3 2x800MHz, SSD 256GB Samsung 860 PRO (2bit MLC, LPDDR4-1866 512 MB) |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar 832,290,816 bytes | 1,845s | 11,177 MB | "zpaq_v7.05_x64.exe" add "%1.method511.zpaq" "%1" -method 511 -threads 1 |
| 87,527,794 bytes | 1,845s | 6,937 MB | i7-3630QM 3.4GHz, 16GB DDR3 2x800MHz, SSD 256GB Samsung 860 PRO (2bit MLC, LPDDR4-1866 512 MB) |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar 832,290,816 bytes | 650s | 9,006 MB | zstd-v1.4.4-win64.exe --single-thread --ultra -22 --zstd=wlog=31,clog=30,hlog=30,slog=26 "%1" -o "%1.L22W2GB.zst" |
| 101,121,392 bytes | 645s | 8,990 MB | i7-3630QM 3.4GHz, 16GB DDR3 2x800MHz, SSD 256GB Samsung 860 PRO (2bit MLC, LPDDR4-1866 512 MB) |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar 832,290,816 bytes | 92s | 5,651 MB | bwtturbo -59 -t0 -b1024 "%1" "%1.zv" |
| 76,516,343 bytes | 91s | 4,127 MB | i7-3630QM 3.4GHz, 16GB DDR3 2x800MHz, SSD 256GB Samsung 860 PRO (2bit MLC, LPDDR4-1866 512 MB) |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar 832,290,816 bytes | 2,764s | 9,531 MB | rz.exe a -d 1023M "%1.rz" "%1" |
| 82,503,337 bytes | 3,469s | 6,400 MB | i7-3630QM 3.4GHz, 16GB DDR3 2x800MHz, SSD 256GB Samsung 860 PRO (2bit MLC, LPDDR4-1866 512 MB) |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar 832,290,816 bytes | 672s | 10,781 MB | "7za_x64_v1900.exe" a -t7z -mx9 -md=30 "%1.MX9Dict1024.7z" "%1" |
| 97,805,186 bytes | 880s | 8,178 MB | i7-3630QM 3.4GHz, 16GB DDR3 2x800MHz, SSD 256GB Samsung 860 PRO (2bit MLC, LPDDR4-1866 512 MB) |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar 832,290,816 bytes | 775s | 1,420 MB | bcrush_ICL150_64bit.exe --optimal "%1" "%1.optimal.bcrush" |
| 154,632,025 bytes | 775s | 1,360 MB | i7-3630QM 3.4GHz, 16GB DDR3 2x800MHz, SSD 256GB Samsung 860 PRO (2bit MLC, LPDDR4-1866 512 MB) |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar 832,290,816 bytes | 396s | 851 MB | crush_xezz.exe -9 "%1" "%1.9.crush_xezz" |
| 166,179,937 bytes | 396s | 852 MB | i7-3630QM 3.4GHz, 16GB DDR3 2x800MHz, SSD 256GB Samsung 860 PRO (2bit MLC, LPDDR4-1866 512 MB) |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar 832,290,816 bytes | 221,703s | 45,402 MB | BriefLZ_130_Intel_v19_64bit.exe --optimal -b2g "%1" "%1.blz" |
| 131,370,690 bytes | 221,699s | 16,383 MB | i7-3630QM 3.4GHz, 16GB DDR3 2x800MHz, SSD 256GB Samsung 860 PRO (2bit MLC, LPDDR4-1866 512 MB) |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| book_serie_SPETSNAZ_(981_UTF-8_novels_Russian).tar 832,290,816 bytes | 2,105s | 10,338 MB | brotli_ic19.exe -q 11 --large_window=30 "%1" -o "%1.bro" |
| 99,152,526 bytes | 2,105s | 6,094 MB | i7-3630QM 3.4GHz, 16GB DDR3 2x800MHz, SSD 256GB Samsung 860 PRO (2bit MLC, LPDDR4-1866 512 MB) |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
An English lyrics .TXT tarred corpus:
D:\TEXTORAMIC_part>lzbench18 -c4 -i1,15 -o3 -elzham,4/slz_deflate,3/slz_gzip,3/slz_zlib,3/tornado,1,16/bzip2,1,9/zstd,1,22/zstdLDM,1,22/fastlzma2,1,10/brotli,0,1,3,5,11/blosclz,1,9/crush,2/csc,5/density,1,3/fastlz,2/gipfeli/lzo1b,999/libdeflate,1,12/lz4,1,12/lizard,11,19,21,29,31,39,41,49/lzf,1/lzfse/lzg,9/lzjb/lzlib,9/lzma,0,1,9/lzrw,5/lzsse2,17/lzsse4,17/lzsse8,17/lzvn/pithy,9/quicklz,3/snappy/slz_zlib,3/ucl_nrv2b,9/ucl_nrv2d,9/ucl_nrv2e,9/xpack,1,9/xz,9/yalz77,12/yappy,99/zlib,1,5,9/zling,4/shrinker/wflz/lzmat ""Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar""
lzbench 1.8 (64-bit Windows) Assembled by P.Skibinski
The results sorted by column number 4:
Compressor name Compress. Decompress. Orig. size Compr. size Ratio Filename
lzma 19.00 -9 1.19 MB/s 99 MB/s 630339584 115019802 18.25 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
xz 5.2.4 -9 1.21 MB/s 94 MB/s 630339584 115020496 18.25 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
csc 2016-10-13 -5 2.70 MB/s 64 MB/s 630339584 115904942 18.39 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
zstd 1.4.3 -22 1.03 MB/s 675 MB/s 630339584 116108562 18.42 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
tornado 0.6a -16 1.40 MB/s 209 MB/s 630339584 116657946 18.51 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
lzham 1.0 -d26 -4 0.84 MB/s 222 MB/s 630339584 116724512 18.52 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
fastlzma2 1.0.1 -10 2.29 MB/s 89 MB/s 630339584 117172953 18.59 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
zstdLDM 1.4.3 -22 1.07 MB/s 673 MB/s 630339584 117382083 18.62 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
lzlib 1.11 -9 1.05 MB/s 69 MB/s 630339584 119213183 18.91 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
BriefLZ_130_Intel_v19_64bit --optimal -b2g 129595542 ! outside lzbench !
brotli 2019-10-01 -11 0.45 MB/s 369 MB/s 630339584 133801391 21.23 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
zling 2018-10-12 -4 33 MB/s 168 MB/s 630339584 149896637 23.78 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
lizard 1.0 -49 1.20 MB/s 935 MB/s 630339584 152783164 24.24 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
Satanichi_GCC730_64bit.exe 942 MB/s 155402007 ! outside lzbench !
brotli 2019-10-01 -5 25 MB/s 397 MB/s 630339584 159025077 25.23 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
bzip2 1.0.8 -9 10 MB/s 26 MB/s 630339584 159111060 25.24 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
crush 1.0 -2 0.24 MB/s 311 MB/s 630339584 159884165 25.36 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
fastlzma2 1.0.1 -1 12 MB/s 63 MB/s 630339584 169549257 26.90 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
xpack 2016-06-02 -9 8.15 MB/s 868 MB/s 630339584 171789474 27.25 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
libdeflate 1.3 -12 5.88 MB/s 687 MB/s 630339584 174665802 27.71 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
bzip2 1.0.8 -1 11 MB/s 34 MB/s 630339584 176353273 27.98 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
lzma 19.00 -1 19 MB/s 61 MB/s 630339584 177675597 28.19 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
brotli 2019-10-01 -3 75 MB/s 380 MB/s 630339584 180750423 28.68 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
lizard 1.0 -29 1.27 MB/s 1479 MB/s 630339584 181099446 28.73 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
lzfse 2017-03-08 49 MB/s 647 MB/s 630339584 182618202 28.97 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
zlib 1.2.11 -9 14 MB/s 279 MB/s 630339584 182978127 29.03 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
lizard 1.0 -39 3.87 MB/s 1735 MB/s 630339584 186023054 29.51 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
zlib 1.2.11 -5 28 MB/s 273 MB/s 630339584 186950753 29.66 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
lzma 19.00 -0 22 MB/s 54 MB/s 630339584 188157873 29.85 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
ucl_nrv2e 1.03 -9 0.97 MB/s 269 MB/s 630339584 192253614 30.50 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
brotli 2019-10-01 -1 159 MB/s 339 MB/s 630339584 193676957 30.73 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
ucl_nrv2d 1.03 -9 0.94 MB/s 275 MB/s 630339584 194174184 30.80 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
lzsse2 2019-04-18 -17 2.76 MB/s 2836 MB/s 630339584 194627075 30.88 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
lzsse4 2019-04-18 -17 2.80 MB/s 3205 MB/s 630339584 196223860 31.13 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
zstdLDM 1.4.3 -1 155 MB/s 570 MB/s 630339584 197357493 31.31 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
lzsse8 2019-04-18 -17 2.75 MB/s 3142 MB/s 630339584 197510652 31.33 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
ucl_nrv2b 1.03 -9 1.06 MB/s 267 MB/s 630339584 198130244 31.43 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
zstd 1.4.3 -1 278 MB/s 563 MB/s 630339584 199130517 31.59 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
xpack 2016-06-02 -1 131 MB/s 648 MB/s 630339584 200250214 31.77 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
libdeflate 1.3 -1 142 MB/s 755 MB/s 630339584 200440695 31.80 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
lizard 1.0 -41 136 MB/s 1046 MB/s 630339584 205067726 32.53 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
lzo1b 2.10 -999 10 MB/s 506 MB/s 630339584 206761274 32.80 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
brotli 2019-10-01 -0 214 MB/s 316 MB/s 630339584 207643666 32.94 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
lzmat 1.01 31 MB/s 270 MB/s 630339584 211176426 33.50 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
lizard 1.0 -19 4.14 MB/s 2623 MB/s 630339584 213237120 33.83 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
zlib 1.2.11 -1 78 MB/s 277 MB/s 630339584 215765399 34.23 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
yalz77 2015-09-19 -12 20 MB/s 289 MB/s 630339584 222530955 35.30 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
lzg 1.0.10 -9 0.59 MB/s 485 MB/s 630339584 227738922 36.13 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
lzvn 2017-03-08 41 MB/s 790 MB/s 630339584 230181093 36.52 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
quicklz 1.5.0 -3 44 MB/s 682 MB/s 630339584 235091903 37.30 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
lizard 1.0 -31 182 MB/s 1769 MB/s 630339584 235304777 37.33 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
gipfeli 2016-07-13 215 MB/s 385 MB/s 630339584 236217347 37.47 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
pithy 2011-12-24 -9 245 MB/s 1344 MB/s 630339584 249961466 39.66 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
lizard 1.0 -21 161 MB/s 1825 MB/s 630339584 255415901 40.52 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
lizard 1.0 -11 214 MB/s 2575 MB/s 630339584 261140979 41.43 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
lzrw 15-Jul-1991 -5 100 MB/s 382 MB/s 630339584 266188186 42.23 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
fastlz 0.1 -2 249 MB/s 421 MB/s 630339584 273296035 43.36 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
lzf 3.6 -1 244 MB/s 513 MB/s 630339584 273507057 43.39 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
density 0.14.2 -3 319 MB/s 313 MB/s 630339584 276123004 43.81 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
yappy 2014-03-22 -99 76 MB/s 1580 MB/s 630339584 278881061 44.24 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
slz_deflate 1.0.0 -3 198 MB/s 275 MB/s 630339584 281261549 44.62 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
slz_gzip 1.0.0 -3 180 MB/s 248 MB/s 630339584 281261555 44.62 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
slz_zlib 1.0.0 -3 171 MB/s 224 MB/s 630339584 281261567 44.62 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
slz_zlib 1.0.0 -3 171 MB/s 223 MB/s 630339584 281261567 44.62 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
lz4 1.9.2 412 MB/s 2963 MB/s 630339584 289439405 45.92 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
lz4 1.9.2 415 MB/s 2940 MB/s 630339584 289439405 45.92 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
snappy 2019-09-30 234 MB/s 878 MB/s 630339584 299969197 47.59 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
tornado 0.6a -1 207 MB/s 301 MB/s 630339584 303416172 48.14 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
wflz 2015-09-16 181 MB/s 601 MB/s 630339584 336375445 53.36 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
lzjb 2010 197 MB/s 345 MB/s 630339584 359561457 57.04 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
density 0.14.2 -1 1070 MB/s 1496 MB/s 630339584 363440712 57.66 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
blosclz 2.0.0 -9 229 MB/s 458 MB/s 630339584 406780399 64.53 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
shrinker 0.1 228 MB/s 3496 MB/s 630339584 577034189 91.54 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
blosclz 2.0.0 -1 7509 MB/s 7710 MB/s 630339584 630339584 100.00 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
memcpy 11163 MB/s 10761 MB/s 630339584 630339584 100.00 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
D:\TEXTORAMIC_part>"turbobench_v18.05_-_build_04_May_2018" ""Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar"" -esnappy_c/yappy/bzip2/lzma,9d30:fb273:mf=bt4/oodle,89,91,95,99,111,115,119,129,139/lzturbo,19,12,10,29,22,20,39,32,30,59/brotli,11d30/zstd,1,5,22d30/lizard,11,19,21,29,31,39,41,49/trle -I3 -J31 -k1 -B2G
C Size ratio% C MB/s D MB/s Name File
102237996 16.2 0.39 264.44 brotli 11d30 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
102669101 16.3 6.54 33.42 lzturbo 59 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
103163583 16.4 0.83 108.35 lzma 9d30:fb273:mf=bt4 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
105172793 16.7 0.60 779.94 lzturbo 39 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
107805227 17.1 0.19 613.36 oodle 139 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
108042846 17.1 1.12 704.78 zstd 22d30 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
108053499 17.1 0.24 689.24 oodle 89 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
108069844 17.1 0.16 690.75 oodle 129 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
119598213 19.0 0.58 1133.88 lzturbo 29 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
129595542 BriefLZ_130_Intel_v19_64bit --optimal -b2g ! outside turbobench !
129784894 20.6 41.55 756.45 lzturbo 32 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
134870724 21.4 0.34 1869.38 oodle 99 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
137326345 21.8 0.74 1874.64 oodle 95 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
152783168 24.2 1.23 939.20 lizard 49 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
155402007 942 Satanichi_GCC730_64bit.exe ! outside turbobench !
159111064 25.2 10.64 25.78 bzip2 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
168353489 26.7 0.36 2870.07 oodle 119 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
172048724 27.3 0.80 2946.52 oodle 115 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
174938936 27.8 57.94 721.46 zstd 5 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
181099450 28.7 1.30 1477.80 lizard 29 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
182005612 28.9 37.41 1109.71 lzturbo 22 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
186023058 29.5 4.13 1651.84 lizard 39 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
194179350 30.8 223.97 969.77 lzturbo 30 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
199248465 31.6 131.93 1741.08 oodle 91 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
199521959 31.7 277.58 914.52 zstd 1 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
205067730 32.5 136.22 1023.51 lizard 41 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
212944026 33.8 0.43 3062.50 lzturbo 19 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
213237124 33.8 4.66 2479.30 lizard 19 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
227939040 36.2 52.69 3010.22 lzturbo 12 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
235304781 37.3 181.33 1686.59 lizard 31 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
255415905 40.5 162.06 1819.06 lizard 21 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
261140983 41.4 215.01 2429.45 lizard 11 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
266282792 42.2 339.58 1086.26 lzturbo 20 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
266307002 42.2 148.95 3135.91 oodle 111 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
279864603 44.4 81.24 1521.44 yappy Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
290977860 46.2 368.64 2673.59 lzturbo 10 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
299900275 47.6 333.39 946.33 snappy_c Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
629881032 99.9 191.84 1998.16 trle Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
And the 'usual suspects':
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| Corpus Name Corpus Size | Global Time | Virtual Memory | Command Line |
| Compressed Size | Process Time | Physical Memory | Testmachine |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| Machine-Learning_..._(over_500000_song_lyrics).tar 630,339,584 bytes | 1,185,386s | 3,636 MB | Satanichi_GCC730_64bit.exe "%1" "%1.Nakamichi" 24 99000 e |
| 155,402,007 bytes | 58,847s | 3,029 MB | i5-7200U 3.1GHz, 8GB DDR4 2x1066MHz, SSD 500GB Samsung 860 EVO (3bit MLC, LPDDR4-1866 512 MB) |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| Machine-Learning_..._(over_500000_song_lyrics).tar 630,339,584 bytes | 193s | 5,144 MB | "zpaq_v7.05_x64.exe" add "%1.method211.zpaq" "%1" -method 211 -threads 1 |
| 125,295,629 bytes | 193s | 3,619 MB | i5-7200U 3.1GHz, 8GB DDR4 2x1066MHz, SSD 500GB Samsung 860 EVO (3bit MLC, LPDDR4-1866 512 MB) |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| Machine-Learning_..._(over_500000_song_lyrics).tar 630,339,584 bytes | 3,036s | 11,698 MB | "zpaq_v7.05_x64.exe" add "%1.method511.zpaq" "%1" -method 511 -threads 1 |
| 86,008,152 bytes | 2,859s | 7,174 MB | i5-7200U 3.1GHz, 8GB DDR4 2x1066MHz, SSD 500GB Samsung 860 EVO (3bit MLC, LPDDR4-1866 512 MB) |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| Machine-Learning_..._(over_500000_song_lyrics).tar 630,339,584 bytes | 664s | 8,813 MB | zstd-v1.4.4-win64.exe --single-thread --ultra -22 --zstd=wlog=31,clog=30,hlog=30,slog=26 "%1" -o "%1.L22W2GB.zst" |
| 108,281,760 bytes | 659s | 7,344 MB | i5-7200U 3.1GHz, 8GB DDR4 2x1066MHz, SSD 500GB Samsung 860 EVO (3bit MLC, LPDDR4-1866 512 MB) |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| Machine-Learning_..._(over_500000_song_lyrics).tar 630,339,584 bytes | 69s | 4,833 MB | bwtturbo -59 -t0 -b1024 "%1" "%1.zv" |
| 95,432,407 bytes | 67s | 3,346 MB | i5-7200U 3.1GHz, 8GB DDR4 2x1066MHz, SSD 500GB Samsung 860 EVO (3bit MLC, LPDDR4-1866 512 MB) |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| Machine-Learning_..._(over_500000_song_lyrics).tar 630,339,584 bytes | 1,839s | 7,455 MB | rz.exe a -d 1023M "%1.rz" "%1" |
| 91,722,824 bytes | 2,403s | 5,539 MB | i5-7200U 3.1GHz, 8GB DDR4 2x1066MHz, SSD 500GB Samsung 860 EVO (3bit MLC, LPDDR4-1866 512 MB) |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| Machine-Learning_..._(over_500000_song_lyrics).tar 630,339,584 bytes | 496s | 8,345 MB | "7za_x64_v1900.exe" a -t7z -mx9 -md=30 "%1.MX9Dict1024.7z" "%1" |
| 103,355,628 bytes | 607s | 6,441 MB | i5-7200U 3.1GHz, 8GB DDR4 2x1066MHz, SSD 500GB Samsung 860 EVO (3bit MLC, LPDDR4-1866 512 MB) |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| Machine-Learning_..._(over_500000_song_lyrics).tar 630,339,584 bytes | 417s | 1,420 MB | bcrush_ICL150_64bit.exe --optimal "%1" "%1.optimal.bcrush" |
| 150,924,735 bytes | 416s | 1,362 MB | i5-7200U 3.1GHz, 8GB DDR4 2x1066MHz, SSD 500GB Samsung 860 EVO (3bit MLC, LPDDR4-1866 512 MB) |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| Machine-Learning_..._(over_500000_song_lyrics).tar 630,339,584 bytes | 203s | 852 MB | crush_xezz.exe -9 "%1" "%1.9.crush_xezz" |
| 165,173,195 bytes | 203s | 851 MB | i5-7200U 3.1GHz, 8GB DDR4 2x1066MHz, SSD 500GB Samsung 860 EVO (3bit MLC, LPDDR4-1866 512 MB) |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| Machine-Learning_..._(over_500000_song_lyrics).tar 630,339,584 bytes | 8,054s | 45,402 MB | BriefLZ_130_Intel_v19_64bit.exe --optimal -b2g "%1" "%1.blz" |
| 129,595,542 bytes | 8,031s | 7,167 MB | i5-7200U 3.1GHz, 8GB DDR4 2x1066MHz, SSD 500GB Samsung 860 EVO (3bit MLC, LPDDR4-1866 512 MB) |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| Machine-Learning_..._(over_500000_song_lyrics).tar 630,339,584 bytes | 1,567s | 10,344 MB | brotli_ic19.exe -q 11 --large_window=30 "%1" -o "%1.bro" |
| 102,237,706 bytes | 1,567s | 5,398 MB | i5-7200U 3.1GHz, 8GB DDR4 2x1066MHz, SSD 500GB Samsung 860 EVO (3bit MLC, LPDDR4-1866 512 MB) |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
And print-friendly 6 PDF pages: http://www.satanichi.net/The_Zennish_Microdeduplicator_9_pages.pdf
Really strong textual compressor BriefLZ 1.3.0 proves to be, really nice!
Thanks once again, some interesting results. lzturbo 29 is at an interesting performance point, shame it isn't open source.
Indeed, LZTurbo simply amazes, it is a diamant - my favorite decompressor; Hamid shared that (having profiled it) only uncached accesses prevent it from exploding, which makes me think that on future CPU-RAM subsystems with lot of cache it will reach new unseen speeds.
Oodle 'Mermaid' also is amazing, kinda these two performers have left the rest in the dust.
The duo lzbench and turbobench brings very informative roster and serves as a ... wake up call.
Will continue to enrich the benchmarks with: PGN: www.pgnmentor.com_242_players.tar 247 MB PDF: Jewish_Review_ofBooks-_2010-2018_completecollection(36_issues_TRUE-PDF).tar 446 MB DNA: www.ncbi.nlm.nih.govDragonfly(Ladona_fulva)_whole_genome_shotgun.tar 929 MB UTF8: http_unbound.biola.edu103-bibles(40+languages).tar 601 MB
In order one to be able to play, temporarily (my Internet drive is full) 3 of above files are here: https://drive.google.com/file/d/1lT5QgRgTKi5sGY6LjpwRlo0yfrZntgVe/view?usp=sharing
Hi, BriefLZ v1.3.0 is indeed very good, finally was able to benchmark the 2GB block with two popular 1GB testfiles: http://www.sanmayce.com/Nakamichi/index.html#2020Dec07
Thanks Joergen for your LZSS superb implementation, some feedback.
Two of my testdatafiles, on my i5-7200U 8GB DDR4 2133MHz:
By chance '-b330' was enough to house 'Star-Trek', is it possible to use bigger numbers, ... in next revisions, maybe?
'StarTrek-_737_Ebooks.tar.x_b330.blz' is downloadable at: https://drive.google.com/file/d/12L8poFVBRNYP5kERdBI5TZHSWYwl4Iph/view?usp=sharing