Benchmarking Zstd versus 14 decompressors

Another iteration of 'Fulg' [De]Compression Benchmark.

Bottomline, first: Zstd with its asynchronous write strongly nears the title Fastest Decompressor-Dumper, congratulations.

Speedwise, I hate not seeing (like missing the superawesome Oodle 'Mermaid' and LZTurbo 29) the complete picture, my attempt to shed some light on the decompression craft follows.

Included compressors:

RAR v.6.20 by Alexander Roshal, Russia; CRUSH v.1.00 by Ilya Muravyov, Russia; BR, Brotli v.1.0.9 by Jyrki Alakuijala, Finland; ZPAQ v.7.05 by Matt Mahoney, America; RZ, RAZOR archiver v.1.03.7 by Christian Martelock, Germany; BWTSATAN by Powturbo, worldwide; GZ, 7zip's GZ v.22.01 by Igor Pavlov, Russia; BZ2, 7zip's BZ2 v.22.01 by Igor Pavlov, Russia; LZ4 v.1.9.3 by Yann Collet aka Cyan, France; ZSTD v.1.5.4 by Yann Collet aka Cyan, France; BSC v.3.2.5 by Ilya Grebnov aka Gribok, Russia; LZSSE by Conor Stokes, Australia; libdeflate v.1.17 by Eric Biggers aka ebiggers, Earth; Satanichi, Sanmayce's texttoy, Bulgaria; BriefLZ v.1.3.0 by Joergen Ibsen, Denmark.

The corresponding lines for compression:

timer64 "rar.exe" a -m5 -md1g %1.rar %1
timer64 "crush_ICL_15.0_O3.exe" cx "%1" "%1.crush"
timer64 "brotli109_compiled_by_Shelwien_mingw_gcc10.2_SSE4.exe" -q 11 --large_window=30 "%1"
timer64 "zpaq_v7.05_x64.exe" add "%1.zpaq" "%1" -method 511 -threads 2
timer64 rz.exe a -d 1023M "%1.rz" "%1"
timer64 "bwtsatan.exe" -17 "%1" "%1"
timer64 "7za_2201-x64.exe" a -tgzip -mx=9 -mmt=8 "%1.gz" "%1"
timer64 "lz4_win64_v1_9_3.exe" -9 "%1"
timer64 "bsc-3.2.5-x64.exe" e "%1" "%1.bsc" -p -b2047 -m0 -e2
timer64 "zstd-v1.5.4-win64.exe" --ultra -22 --long=31 --zstd=wlog=31,clog=30,hlog=30,slog=26 "%1" -o "%1.zst"
timer64 lzsse.exe -2 -l17 "%1" "%1.lzsse2"
timer64 "7za_2201-x64.exe" a -tbzip2 -mx=9 "%1.bz2" "%1"
timer64 "gzip_libdeflate-1.17-windows-x86_64-bin.exe" -12 -f -k -S .LIBDEFLATE12 "%1"
timer64 Satanichi_Nakamichi_Vanilla_LITE_DD-128AES_CLANG_64bit.exe "%1" "%1.Nakamichi" 22 121000 i
timer64 BriefLZ_130_Intel_v15_64bit_indicator.exe --optimal -b3g "%1" "%1.blz"

Decompression lines:

timer64.exe "rar.exe" x %1.rar
timer64.exe "crush_ICL_15.0_O3.exe" d %1.crush %1.uncrush
timer64.exe brotli109_compiled_by_Shelwien_mingw_gcc10.2_SSE4.exe -d -k "%1.br"
timer64.exe "zpaq_v7.05_x64.exe" x "%1.zpaq" -threads 2
timer64.exe "rz.exe" e "%1.rz" "%1"
timer64.exe bwtsatan -d "%1.rc" "%1"
timer64.exe "7za_2201-x64.exe" e "%1.gz"
timer64.exe "lz4_win64_v1_9_3.exe" -d "%1.lz4"
timer64.exe bsc-3.2.5-x64.exe d "%1.bsc" "%1"
timer64.exe "zstd-v1.5.4-win64.exe" -f --priority=rt -d --long=31 "%1.zst"
timer64.exe "zstd-v1.5.4-win64.exe" -f --priority=rt --no-asyncio -d --long=31 "%1.zst"
timer64.exe lzsse.exe -d "%1.lzsse2" "%1" 
timer64.exe "7za_2201-x64.exe" e "%1.bz2"
timer64.exe "gzip_libdeflate-1.17-windows-x86_64-bin.exe" -d -f -k -S .LIBDEFLATE12 "%1.LIBDEFLATE12"
Satanichi_Nakamichi_Vanilla_LITE_DD-128AES_GCC_64bit.exe %1.Nakamichi>%1.NKMCH
timer64.exe "BriefLZ_130_Intel_v15_64bit_indicator" -d -b3g "%1.blz" "%1"

Testdatafile #1, sorted by decompression time (4th column): Size: 59,494,400 bytes Name: TERAPIG_EvgeniyGulyakovskiy(45-FB2-files).tar sha1: 721dd1544a77bc7bdcba4f0e054b72b9419f2468 Testmachine: laptop 'Brutalitto' AMD 4800H max turbo 4.3GHz 8/16 cores/threads, DDR4 64GB 3200MHz RAM, Windows 10

+---------------------------+---------------+-----------------+-----------------+-------------------+---------------------------+
| [De]compressor,           | Compression,  | Compression,    | Decompression,  | Memory Footprint, | CPU (Process Time)        |
| MAX settings              | size in bytes | time in seconds,| time in seconds,| Virtual/Physical  | Utilization               |
|                           |               | Global Time     | Global Time     | Memory            | compression/decompression |
+---------------------------+---------------+-----------------+-----------------+-------------------+---------------------------+
| ZSTD v.1.5.4              |    12,240,928 |         46.76 s |          0.07 s |      3700/1104 MB |                 100%/177% |
| LZ4 v.1.9.3               |    25,218,044 |          3.44 s |          0.07 s |           9/10 MB |                  98%/109% | ! Why 9% above 100? !
| LZSSE                     |    23,423,825 |          7.01 s |          0.08 s |      3340/1520 MB |                   99%/95% |
| Satanichi_GCC             |    17,557,771 |            N.A. |          0.08 s |              N.A. |                 N.A./100% |
| libdeflate v.1.17         |    20,349,168 |         11.48 s |          0.09 s |          67/87 MB |                  99%/110% | ! Why 10% above 100? !
| Satanichi_CLANG           |    17,557,771 |        933.65 s |          0.09 s |    122899/9553 MB |                 100%/100% |
| Satanichi_ICL             |    17,557,771 |            N.A. |          0.09 s |              N.A. |                 N.A./100% |
| ZSTD v.1.5.4 --no-asyncio |    12,240,928 |         46.76 s |          0.11 s |      3700/1104 MB |                 100%/124% |
| Brotli v.1.0.9            |    11,969,256 |         80.41 s |          0.15 s |      10334/530 MB |                   99%/98% |
| BriefLZ v.1.3.0           |    16,446,090 |       1766.29 s |          0.16 s |    68103/16181 MB |                   99%/93% |
| RAR v.6.20                |    13,462,342 |          2.98 s |          0.20 s |        519/517 MB |                 932%/153% |
| Crush                     |    20,179,200 |        445.41 s |          0.20 s |        145/147 MB |                  100%/99% |
| 7zip's GZ v.22.01         |    20,413,210 |         47.62 s |          0.29 s |            5/9 MB |                   99%/96% |
| RAZOR v.1.03.7            |    11,256,631 |         92.23 s |          0.51 s |        778/568 MB |                  125%/97% |
| BSC v.3.2.5               |    11,806,322 |          2.01 s |          0.72 s |        295/297 MB |               1347%/1196% |
| 7zip's BZ2 v.22.01        |    16,273,277 |          6.81 s |          0.86 s |        188/158 MB |                1371%/133% |
| BWTSATAN                  |    11,940,918 |          4.29 s |          1.67 s |      2369/1312 MB |                  100%/99% |
| ZPAQ v.7.05               |    11,525,951 |        128.00 s |        130.10 s |       2864/846 MB |                  99%/100% |
+---------------------------+---------------+-----------------+-----------------+-------------------+---------------------------+

Note1: The Decompression time includes LOAD-DECOMPRESS-DUMP times, that is, external-RAM -> internal-RAM -> external-RAM. Note2: The decompression is done on RamDisk of size 12GB, both the compressed and the decompressed files are on it. Note3: Should LZSSE be threaded, it would scream insanely. Note4: Both BriefLZ and Crush were compiled with Intel Compiler v15.0, /O3 was used. Note5: Satanichi monstrously devours physical RAM, like 3TB, in order to flex its muscles.

Testdatafile #2, sorted by decompression time (4th column): Size: 245,732,864 bytes Name: SUPRAPIG_Marx_Engels_Lenin_Stalin_CollectedWritings(2052_HTMs_in_Russian).tar sha1: 2196779fe2cc03718b02876262254ee40780dab6 Testmachine: laptop 'Brutalitto' AMD 4800H max turbo 4.3GHz 8/16 cores/threads, DDR4 64GB 3200MHz RAM, Windows 10

+---------------------------+---------------+-----------------+-----------------+-------------------+---------------------------+
| [De]compressor,           | Compression,  | Compression,    | Decompression,  | Memory Footprint, | CPU (Process Time)        |
| MAX settings              | size in bytes | time in seconds,| time in seconds,| Virtual/Physical  | Utilization               |
|                           |               | Global Time     | Global Time     | Memory            | compression/decompression |
+---------------------------+---------------+-----------------+-----------------+-------------------+---------------------------+
| LZSSE                     |    70,011,827 |         37.28 s |          0.21 s |      3340/3249 MB |                  99%/100% |
| LZ4 v.1.9.3               |    77,690,671 |          6.15 s |          0.24 s |            9/9 MB |                  99%/100% |
| ZSTD v.1.5.4              |    42,349,272 |        211.21 s |          0.33 s |      9453/4396 MB |                  99%/147% |
| libdeflate v.1.17         |    66,803,854 |         41.31 s |          0.37 s |        245/310 MB |                   99%/94% |
| Satanichi_GCC             |    57,046,870 |            N.A. |          0.39 s |              N.A. |                 N.A./100% |
| Satanichi_CLANG           |    57,046,870 |       4217.44 s |          0.40 s |   123255/11948 MB |                  99%/100% |
| Satanichi_ICL             |    57,046,870 |            N.A. |          0.42 s |              N.A. |                 N.A./100% |
| ZSTD v.1.5.4 --no-asyncio |    42,349,272 |        211.21 s |          0.47 s |      9453/4396 MB |                   99%/99% |
| Crush                     |    55,697,571 |        826.18 s |          0.66 s |        145/147 MB |                   99%/98% |
| Brotli v.1.0.9            |    41,877,772 |        437.70 s |          0.74 s |     10340/2150 MB |                 100%/100% |
| RAR v.6.20                |    46,525,825 |         18.52 s |          0.75 s |      1609/1595 MB |                1226%/252% |
| BriefLZ v.1.3.0           |    52,403,895 |        227.62 s |          0.89 s |     68103/9838 MB |                   99%/98% |
| 7zip's GZ v.22.01         |    66,932,970 |        178.20 s |          1.07 s |            5/9 MB |                   86%/98% |
| BSC v.3.2.5               |    33,211,258 |          6.89 s |          2.28 s |      1185/1185 MB |               1390%/1148% |
| RAZOR v.1.03.7            |    35,514,213 |        507.47 s |          2.49 s |      2870/2442 MB |                  135%/99% |
| 7zip's BZ2 v.22.01        |    48,703,720 |         24.61 s |          3.20 s |        188/151 MB |                1555%/134% |
| BWTSATAN                  |    33,642,530 |         14.90 s |          6.24 s |      3128/2075 MB |                  99%/100% |
| ZPAQ v.7.05               |    31,249,775 |        581.15 s |        585.11 s |      4620/2742 MB |                   99%/99% |
+---------------------------+---------------+-----------------+-----------------+-------------------+---------------------------+

Testdatafile #3, sorted by decompression time (4th column): Size: 606,683,136 bytes Name: Intel_Parallel_Studio_XE_2020_Product_Documentation_IPSXE2020u4_OfflineDocs2020.4.011(38516-files).tar sha1: e8ef98887362adb569baf8fb3bac4dfd33085ba1 Testmachine: laptop 'Brutalitto' AMD 4800H max turbo 4.3GHz 8/16 cores/threads, DDR4 64GB 3200MHz RAM, Windows 10

+---------------------------+---------------+-----------------+-----------------+-------------------+---------------------------+
| [De]compressor,           | Compression,  | Compression,    | Decompression,  | Memory Footprint, | CPU (Process Time)        |
| MAX settings              | size in bytes | time in seconds,| time in seconds,| Virtual/Physical  | Utilization               |
|                           |               | Global Time     | Global Time     | Memory            | compression/decompression |
+---------------------------+---------------+-----------------+-----------------+-------------------+---------------------------+
| ZSTD v.1.5.4              |   117,008,878 |        443.71 s |          0.43 s |     13606/8963 MB |                 100%/182% |
| LZSSE                     |   215,528,254 |        300.89 s |          0.49 s |      3340/3274 MB |                   99%/96% |
| LZ4 v.1.9.3               |   187,022,882 |          6.99 s |          0.53 s |           9/11 MB |                   98%/99% |
| Satanichi_GCC             |   147,421,238 |            N.A. |          0.58 s |              N.A. |                 N.A./100% |
| Satanichi_CLANG           |   147,421,238 |      10427.89 s |          0.61 s |   123945/44101 MB |                  99%/100% |
| Satanichi_ICL             |   147,421,238 |            N.A. |          0.62 s |              N.A. |                 N.A./100% |
| ZSTD v.1.5.4 --no-asyncio |   117,008,878 |        443.71 s |          0.68 s |     13606/8963 MB |                  100%/99% |
| libdeflate v.1.17         |   176,912,170 |        114.50 s |          0.89 s |        591/758 MB |                   99%/94% |
| BriefLZ v.1.3.0           |   133,674,583 |      16201.20 s |          1.02 s |    68103/30713 MB |                   99%/94% |
| Crush                     |   161,892,032 |        187.64 s |          1.11 s |        145/147 MB |                  100%/97% |
| RAR v.6.20                |   119,918,661 |         13.38 s |          1.22 s |      6226/6217 MB |                 869%/158% |
| Brotli v.1.0.9            |   116,379,429 |        774.02 s |          1.49 s |  10339/5157 MB MB |                  99%/100% |
| 7zip's GZ v.22.01         |   177,284,089 |        258.77 s |          2.08 s |            5/9 MB |                   99%/98% |
| RAZOR v.1.03.7            |   113,458,656 |        517.99 s |          3.18 s |      7220/3614 MB |                  120%/97% |
| 7zip's BZ2 v.22.01        |   162,560,647 |         62.97 s |          7.33 s |        188/175 MB |                1496%/134% |
| BSC v.3.2.5               |   124,519,806 |         18.24 s |          8.63 s |      2910/2906 MB |                1306%/814% |
| BWTSATAN                  |   126,197,358 |         28.75 s |         16.83 s |      3791/2472 MB |                   99%/99% |
| ZPAQ v.7.05               |   109,537,017 |       1438.59 s |       1452.80 s |     11176/8989 MB |                   99%/99% |
+---------------------------+---------------+-----------------+-----------------+-------------------+---------------------------+

Testdatafile #4, sorted by decompression time (4th column): Size: 998,337,024 bytes Name: TERAPIGSPETSNAZ(1012-FB2-books).tar sha1: 92e9886e3b09c749a90ae0e567218275a0c1140b Testmachine: laptop 'Brutalitto' AMD 4800H max turbo 4.3GHz 8/16 cores/threads, DDR4 64GB 3200MHz RAM, Windows 10

+---------------------------+---------------+-----------------+-----------------+-------------------+---------------------------+
| [De]compressor,           | Compression,  | Compression,    | Decompression,  | Memory Footprint, | CPU (Process Time)        |
| MAX settings              | size in bytes | time in seconds,| time in seconds,| Virtual/Physical  | Utilization               |
|                           |               | Global Time     | Global Time     | Memory            | compression/decompression |
+---------------------------+---------------+-----------------+-----------------+-------------------+---------------------------+
| LZSSE                     |   428,308,549 |        124.74 s |          0.81 s |      3340/3269 MB |                   99%/93% |
| LZ4 v.1.9.3               |   453,298,991 |         53.01 s |          0.97 s |           9/10 MB |                   99%/99% |
| Satanichi_GCC             |   361,785,996 |            N.A. |          1.56 s |              N.A. |                 N.A./100% |
| ZSTD v.1.5.4              |   262,887,821 |        837.83 s |          1.59 s |     13606/9469 MB |                 100%/136% |
| libdeflate v.1.17         |   363,952,523 |        185.13 s |          1.61 s |       966/1310 MB |                   99%/97% |
| Satanichi_CLANG           |   361,785,996 |     506703.98 s |          1.63 s |   124693/61407 MB |                  15%/100% | ! RAM needed to house B-trees: 74,667MB; RAM needed to build B-trees IN ONE PASS: 62,464MB !
| Satanichi_ICL             |   361,785,996 |            N.A. |          1.64 s |              N.A. |                 N.A./100% |
| ZSTD v.1.5.4 --no-asyncio |   262,887,821 |        837.83 s |          2.12 s |     13606/9469 MB |                  100%/99% |
| Crush                     |   389,066,140 |       7272.93 s |          3.16 s |        145/147 MB |                   99%/97% |
| Brotli v.1.0.9            |   252,519,695 |       2018.26 s |          3.58 s |     10348/8058 MB |                  99%/100% |
| BriefLZ v.1.3.0           |   349,799,748 |     110401.14 s |          3.99 s |    68103/34388 MB |                   99%/98% |
| RAR v.6.20                |   298,941,140 |         74.33 s |          4.41 s |      6225/6213 MB |                1090%/184% |
| 7zip's GZ v.22.01         |   364,996,238 |        761.81 s |          4.69 s |            5/9 MB |                  99%/100% |
| RAZOR v.1.03.7            |   235,256,567 |       2879.64 s |         12.06 s |     11249/8595 MB |                  134%/99% |
| 7zip's BZ2 v.22.01        |   300,191,216 |        103.44 s |         13.76 s |        188/165 MB |                1563%/139% |
| BSC v.3.2.5               |   236,002,728 |         37.38 s |         14.38 s |      4781/4774 MB |                1264%/997% |
| BWTSATAN                  |   237,498,822 |         98.01 s |         41.04 s |      6824/5788 MB |                   99%/99% |
| ZPAQ v.7.05               |   236,553,795 |       2273.49 s |       2319.31 s |     11369/9881 MB |                   99%/99% |
+---------------------------+---------------+-----------------+-----------------+-------------------+---------------------------+

Benchmarking bigger than 1GB files will clarify the picture more...

Finishing with 5 1+GB testdatafiles.

Testdatafile #5, sorted by decompression time (4th column): Size: 1,258,373,120 bytes Name: FULG3+llvm-project-llvmorg-15.0.7(118554_files).tar sha1: 22621fae541c8093c2a88be94350a60a3da77a46 Testmachine: laptop 'Brutalitto' AMD 4800H max turbo 4.3GHz 8/16 cores/threads, DDR4 64GB 3200MHz RAM, Windows 10

+---------------------------+---------------+-----------------+-----------------+-------------------+---------------------------+
| [De]compressor,           | Compression,  | Compression,    | Decompression,  | Memory Footprint, | CPU (Process Time)        |
| MAX settings              | size in bytes | time in seconds,| time in seconds,| Virtual/Physical  | Utilization               |
|                           |               | Global Time     | Global Time     | Memory            | compression/decompression |
+---------------------------+---------------+-----------------+-----------------+-------------------+---------------------------+
| LZSSE                     |   193,412,757 |      15070.89 s |          0.85 s |      3340/3240 MB |                   99%/85% |
| ZSTD v.1.5.4              |   103,971,207 |        938.69 s |          1.02 s |     14697/9622 MB |                 100%/162% |
| LZ4 v.1.9.3               |   192,926,894 |         23.03 s |          1.08 s |           9/11 MB |                  99%/100% |
| Satanichi_GCC             |   163,502,460 |            N.A. |          1.35 s |              N.A. |                 N.A./100% |
| Satanichi_ICL             |   163,502,460 |            N.A. |          1.42 s |              N.A. |                 N.A./100% |
| Satanichi_CLANG           |   163,502,460 |     256891.34 s |          1.43 s |   125190/61484 MB |                  19%/100% |
| libdeflate v.1.17         |   155,206,929 |        307.95 s |          1.45 s |      1216/1360 MB |                   99%/95% |
| ZSTD v.1.5.4 --no-asyncio |   103,971,207 |        938.69 s |          1.60 s |     14697/9622 MB |                  100%/99% |
| Crush                     |   145,475,548 |       1358.15 s |          2.11 s |        145/147 MB |                   99%/92% |
| Brotli v.1.0.9            |   100,196,025 |       1831.45 s |          2.23 s |     10349/9341 MB |                   99%/99% |
| BriefLZ v.1.3.0           |   127,907,864 |     166702.48 s |          2.44 s |    68103/39432 MB |                   99%/98% |
| RAR v.6.20                |   114,792,647 |         42.69 s |          2.45 s |      6224/6216 MB |                1039%/190% |
| 7zip's GZ v.22.01         |   155,788,537 |        758.55 s |          3.28 s |            5/9 MB |                  99%/100% |
| RAZOR v.1.03.7            |    87,975,377 |       1821.35 s |          5.76 s |     12048/9568 MB |                  120%/98% |
| BSC v.3.2.5               |   103,957,426 |         29.52 s |         10.86 s |      6023/6014 MB |                1376%/779% |
| 7zip's BZ2 v.22.01        |   127,200,962 |        112.44 s |         11.66 s |        188/166 MB |                1551%/124% |
| BWTSATAN                  |   106,445,480 |         50.57 s |         23.30 s |      5656/4241 MB |                   99%/99% |
| ZPAQ v.7.05               |    75,349,170 |       2735.90 s |             ? s |    19906/17994 MB |                     99%/? |
+---------------------------+---------------+-----------------+-----------------+-------------------+---------------------------+

Note1: The Decompression time includes LOAD-DECOMPRESS-DUMP times, that is, external-RAM -> internal-RAM -> external-RAM. Note2: The decompression (the best out of three attempts was taken) is done on RamDisk of size 12GB, both the compressed and the decompressed files are on it. Note3: Should LZSSE be threaded, it would scream insanely. Note4: Both BriefLZ and Crush were compiled with Intel Compiler v15.0, /O3 was used. Note5: Satanichi monstrously devours physical RAM, like 3TB, in order to flex its muscles. ! RAM needed to house B-trees: 74,272MB; RAM needed to build B-trees IN ONE PASS: 70,400MB ! Note6: Histogram (rotate it 90 degrees clockwise) of Unique-Building-Blocks appearing 2[+] times (e.g. 'alfalfa-alfa' stream has 1 BB with length 4 appearing 3 times thus adding 1 to the count for {004]): 84,128,904{256] 104,806,132{128] 117,863,080{064] 118,613,398{056] 118,503,512{048] 114,221,742{036] 106,492,579{028] 100,792,087{024] 88,298,701{018] 81,966,330{016] 73,630,414{014] 62,860,206{012] 49,695,329{010] 34,897,758{008] 19,319,634{006] 6,209,699{004]

Testdatafile #6, sorted by decompression time (4th column): Size: 1,377,922,560 bytes Name: FULG3+_Embarcadero_Studio_22.0_HelpDoc(136211_files).tar sha1: ad5853cd37d49c1f61f7213eca5b19bd70876a39 Testmachine: laptop 'Brutalitto' AMD 4800H max turbo 4.3GHz 8/16 cores/threads, DDR4 64GB 3200MHz RAM, Windows 10

+---------------------------+---------------+-----------------+-----------------+-------------------+---------------------------+
| [De]compressor,           | Compression,  | Compression,    | Decompression,  | Memory Footprint, | CPU (Process Time)        |
| MAX settings              | size in bytes | time in seconds,| time in seconds,| Virtual/Physical  | Utilization               |
|                           |               | Global Time     | Global Time     | Memory            | compression/decompression |
+---------------------------+---------------+-----------------+-----------------+-------------------+---------------------------+
| ZSTD v.1.5.4              |   144,422,202 |       1595.20 s |          0.91 s |     14697/9785 MB |                 100%/180% |
| LZSSE                     |   259,488,690 |       1331.59 s |          0.95 s |      3340/3298 MB |                   99%/92% |
| LZ4 v.1.9.3               |   224,182,293 |         15.18 s |          1.13 s |           9/11 MB |                  99%/100% |
| Satanichi_GCC             |   196,488,789 |            N.A. |          1.23 s |              N.A. |                 N.A./100% |
| Satanichi_CLANG           |   196,488,789 |      23501.19 s |          1.29 s |   125419/30901 MB |                  99%/100% |
| Satanichi_ICL             |   196,488,789 |            N.A. |          1.31 s |              N.A. |                 N.A./100% |
| ZSTD v.1.5.4 --no-asyncio |   144,422,202 |       1595.20 s |          1.43 s |     14697/9785 MB |                  100%/99% |
| libdeflate v.1.17         |   203,879,721 |        293.03 s |          1.56 s |      1330/1520 MB |                   99%/89% |
| Crush                     |   178,544,595 |        512.12 s |          1.75 s |        145/147 MB |                   99%/97% |
| BriefLZ v.1.3.0           |   166,865,069 |      26364.22 s |          2.02 s |    68103/31614 MB |                   99%/97% |
| RAR v.6.20                |   148,158,659 |         22.41 s |          2.14 s |      6225/6217 MB |                 941%/159% |
| Brotli v.1.0.9            |   142,621,172 |       1393.67 s |          2.31 s |     10343/9505 MB |                  99%/100% |
| 7zip's GZ v.22.01         |   204,832,067 |        609.32 s |          3.33 s |            5/9 MB |                   99%/99% |
| RAZOR v.1.03.7            |   136,135,276 |       1353.07 s |          6.03 s |     12084/6954 MB |                  119%/97% |
| 7zip's BZ2 v.22.01        |   169,849,958 |        136.80 s |         14.96 s |        188/175 MB |                1531%/122% |
| BSC v.3.2.5               |   139,825,058 |         35.22 s |         15.76 s |      6594/6584 MB |                1366%/757% |
| BWTSATAN                  |   145,246,444 |         37.07 s |         21.45 s |      4509/2810 MB |                   99%/99% |
| ZPAQ v.7.05               |   126,239,297 |       3014.02 s |             ? s |    19385/17543 MB |                     99%/? |
+---------------------------+---------------+-----------------+-----------------+-------------------+---------------------------+

Note1: The Decompression time includes LOAD-DECOMPRESS-DUMP times, that is, external-RAM -> internal-RAM -> external-RAM. Note2: The decompression (the best out of three attempts was taken) is done on RamDisk of size 12GB, both the compressed and the decompressed files are on it. Note3: Should LZSSE be threaded, it would scream insanely. Note4: Both BriefLZ and Crush were compiled with Intel Compiler v15.0, /O3 was used. Note5: Satanichi monstrously devours physical RAM, like 3TB, in order to flex its muscles. ! RAM needed to house B-trees: 26,727MB; RAM needed to build B-trees IN ONE PASS: 74,048MB ! Note6: Histogram (rotate it 90 degrees clockwise) of Unique-Building-Blocks appearing 2[+] times (e.g. 'alfalfa-alfa' stream has 1 BB with length 4 appearing 3 times thus adding 1 to the count for {004]): 42,514,510{256] 51,840,136{128] 60,064,855{064] 58,682,172{056] 54,290,197{048] 43,039,037{036] 33,719,146{028] 28,478,598{024] 20,620,339{018] 17,976,659{016] 15,311,319{014] 12,716,749{012] 10,327,492{010] 8,455,729{008] 6,956,542{006] 9,109,251{004]

Testdatafile #7, sorted by decompression time (4th column): Size: 1,358,735,360 bytes Name: FULG3+linux-6.1.11(78710_files).tar sha1: 6cd298edd45a5779bb2d0647ca554d129c030b0c Testmachine: laptop 'Brutalitto' AMD 4800H max turbo 4.3GHz 8/16 cores/threads, DDR4 64GB 3200MHz RAM, Windows 10

+---------------------------+---------------+-----------------+-----------------+-------------------+---------------------------+
| [De]compressor,           | Compression,  | Compression,    | Decompression,  | Memory Footprint, | CPU (Process Time)        |
| MAX settings              | size in bytes | time in seconds,| time in seconds,| Virtual/Physical  | Utilization               |
|                           |               | Global Time     | Global Time     | Memory            | compression/decompression |
+---------------------------+---------------+-----------------+-----------------+-------------------+---------------------------+
| LZSSE                     |   244,017,212 |        292.69 s |          0.94 s |      3340/3238 MB |                   99%/93% |
| LZ4 v.1.9.3               |   261,571,544 |         29.68 s |          1.22 s |            9/9 MB |                   99%/97% |
| ZSTD v.1.5.4              |   130,101,184 |        927.61 s |          1.39 s |     15727/9759 MB |                 100%/153% |
| Satanichi_GCC             |   197,914,434 |            N.A. |          1.65 s |              N.A. |                 N.A./100% |
| libdeflate v.1.17         |   206,272,933 |        315.59 s |          1.67 s |      1312/1504 MB |                   99%/92% |
| Satanichi_CLANG           |   197,914,434 |     601239.94 s |          1.74 s |   125382/61425 MB |                  14%/100% |
| Satanichi_ICL             |   197,914,434 |            N.A. |          1.76 s |              N.A. |                 N.A./100% |
| ZSTD v.1.5.4 --no-asyncio |   130,101,184 |        927.61 s |          1.99 s |     15727/9759 MB |                  100%/97% |
| Crush                     |   185,629,723 |       1852.54 s |          2.60 s |        145/147 MB |                  100%/96% |
| Brotli v.1.0.9            |   125,074,423 |       2060.84 s |          2.66 s |     10338/9219 MB |                  99%/100% |
| RAR v.6.20                |   144,235,895 |         64.38 s |          2.97 s |      6225/6204 MB |                1174%/151% |
| BriefLZ v.1.3.0           |   158,968,415 |      74158.14 s |          3.07 s |    68103/29964 MB |                   99%/98% |
| 7zip's GZ v.22.01         |   206,322,609 |       1030.71 s |          4.08 s |            5/9 MB |                   99%/99% |
| RAZOR v.1.03.7            |   107,623,194 |       2314.27 s |          7.74 s |    12021/10448 MB |                  128%/99% |
| 7zip's BZ2 v.22.01        |   163,087,598 |        112.07 s |         11.87 s |        188/151 MB |                1575%/136% |
| BSC v.3.2.5               |   131,663,428 |         32.19 s |         12.41 s |      6502/6492 MB |                1402%/849% |
| BWTSATAN                  |   131,834,764 |         65.36 s |         29.38 s |      6003/4675 MB |                   99%/99% |
| ZPAQ v.7.05               |    90,430,646 |       2988.13 s |             ? s |    19908/17886 MB |                     99%/? |
+---------------------------+---------------+-----------------+-----------------+-------------------+---------------------------+

Note1: The Decompression time includes LOAD-DECOMPRESS-DUMP times, that is, external-RAM -> internal-RAM -> external-RAM. Note2: The decompression (the best out of three attempts was taken) is done on RamDisk of size 12GB, both the compressed and the decompressed files are on it. Note3: Should LZSSE be threaded, it would scream insanely. Note4: Both BriefLZ and Crush were compiled with Intel Compiler v15.0, /O3 was used. Note5: Satanichi monstrously devours physical RAM, like 3TB, in order to flex its muscles. ! RAM needed to house B-trees: 84,244MB; RAM needed to build B-trees IN ONE PASS: 73,440MB ! Note6: Histogram (rotate it 90 degrees clockwise) of Unique-Building-Blocks appearing 2[+] times (e.g. 'alfalfa-alfa' stream has 1 BB with length 4 appearing 3 times thus adding 1 to the count for {004]): 109,104,192{256] 123,436,555{128] 113,571,799{064] 112,298,418{056] 111,598,139{048] 112,125,824{036] 113,880,467{028] 114,618,578{024] 112,312,382{018] 108,614,516{016] 101,168,965{014] 88,006,903{012] 68,070,992{010] 42,991,526{008] 18,265,615{006] 2,591,134{004]

Testdatafile #8, sorted by decompression time (4th column): Size: 1,372,453,888 bytes Name: DeepMind_Q_and_A_Dataset_dailymailstories(219506_files).tar sha1: 4966208f33d8e146396e2a4fbf93d29d84048b84 Testmachine: laptop 'Brutalitto' AMD 4800H max turbo 4.3GHz 8/16 cores/threads, DDR4 64GB 3200MHz RAM, Windows 10

+---------------------------+---------------+-----------------+-----------------+-------------------+---------------------------+
| [De]compressor,           | Compression,  | Compression,    | Decompression,  | Memory Footprint, | CPU (Process Time)        |
| MAX settings              | size in bytes | time in seconds,| time in seconds,| Virtual/Physical  | Utilization               |
|                           |               | Global Time     | Global Time     | Memory            | compression/decompression |
+---------------------------+---------------+-----------------+-----------------+-------------------+---------------------------+
| LZSSE                     |   415,187,401 |        559.18 s |          1.03 s |      3340/3246 MB |                   99%/86% |
| LZ4 v.1.9.3               |   451,557,802 |         28.56 s |          1.32 s |            9/9 MB |                  99%/100% |
| libdeflate v.1.17         |   374,386,581 |        256.27 s |          2.19 s |      1325/1675 MB |                   99%/92% |
| Satanichi_GCC             |   310,305,324 |            N.A. |          2.49 s |              N.A. |                 N.A./100% |
| ZSTD v.1.5.4              |   229,379,374 |      10960.78 s |          2.57 s |     14697/9813 MB |                 100%/129% |
| Satanichi_CLANG           |   310,305,324 |      46867.08 s |          2.65 s |   125408/61925 MB |                  96%/100% |
| Satanichi_ICL             |   310,305,324 |            N.A. |          2.71 s |              N.A. |                 N.A./100% |
| ZSTD v.1.5.4 --no-asyncio |   229,379,374 |      10960.78 s |          3.33 s |     14697/9813 MB |                  100%/97% |
| Crush                     |   334,064,129 |       3453.98 s |          3.58 s |        145/147 MB |                   99%/97% |
| RAR v.6.20                |   270,453,322 |        123.59 s |          4.68 s |      6225/6202 MB |                1429%/168% |
| Brotli v.1.0.9            |   223,198,932 |       2679.66 s |          5.24 s |     10329/9538 MB |                   99%/99% |
| 7zip's GZ v.22.01         |   377,171,070 |        680.61 s |          6.12 s |            5/8 MB |                   99%/99% |
| BriefLZ v.1.3.0           |   288,100,622 |      19139.92 s |          6.32 s |    68103/25834 MB |                   99%/98% |
| BSC v.3.2.5               |   182,558,508 |         40.88 s |         15.93 s |      6569/6557 MB |                1365%/925% |
| 7zip's BZ2 v.22.01        |   296,383,588 |        118.72 s |         16.46 s |        188/152 MB |                1581%/137% |
| RAZOR v.1.03.7            |   196,408,591 |       4205.90 s |         18.03 s |    12048/11596 MB |                  139%/99% |
| BWTSATAN                  |   187,849,236 |         93.78 s |         37.77 s |      6290/4712 MB |                   99%/99% |
| ZPAQ v.7.05               |   180,271,075 |       3556.16 s |             ? s |    20100/17971 MB |                     99%/? |
+---------------------------+---------------+-----------------+-----------------+-------------------+---------------------------+

Note1: The Decompression time includes LOAD-DECOMPRESS-DUMP times, that is, external-RAM -> internal-RAM -> external-RAM. Note2: The decompression (the best out of three attempts was taken) is done on RamDisk of size 12GB, both the compressed and the decompressed files are on it. Note3: Should LZSSE be threaded, it would scream insanely. Note4: Both BriefLZ and Crush were compiled with Intel Compiler v15.0, /O3 was used. Note5: Satanichi monstrously devours physical RAM, like 3TB, in order to flex its muscles. ! RAM needed to house B-trees: 61,539MB; RAM needed to build B-trees IN ONE PASS: 73,856MB ! Note6: Histogram (rotate it 90 degrees clockwise) of Unique-Building-Blocks appearing 2[+] times (e.g. 'alfalfa-alfa' stream has 1 BB with length 4 appearing 3 times thus adding 1 to the count for {004]): 27,446,883{256] 39,188,551{128] 57,556,658{064] 61,881,109{056] 67,055,736{048] 78,714,798{036] 93,536,624{028] 106,550,446{024] 126,652,190{018] 125,610,381{016] 114,388,185{014] 90,917,893{012] 59,402,716{010] 29,471,189{008] 10,940,521{006] 764,072{004]

Testdatafile #9, sorted by decompression time (4th column): Size: 1,081,004,544 bytes Name: FULG3+_github.com_jeffparkbarchive-master(177780_files).tar sha1: 223dcec6e5dddde54e22418815962f48ce4739b6 Testmachine: laptop 'Brutalitto' AMD 4800H max turbo 4.3GHz 8/16 cores/threads, DDR4 64GB 3200MHz RAM, Windows 10

+---------------------------+---------------+-----------------+-----------------+-------------------+---------------------------+
| [De]compressor,           | Compression,  | Compression,    | Decompression,  | Memory Footprint, | CPU (Process Time)        |
| MAX settings              | size in bytes | time in seconds,| time in seconds,| Virtual/Physical  | Utilization               |
|                           |               | Global Time     | Global Time     | Memory            | compression/decompression |
+---------------------------+---------------+-----------------+-----------------+-------------------+---------------------------+
| LZSSE                     |   177,947,568 |        613.41 s |          0.72 s |      3340/3233 MB |                   99%/94% |
| ZSTD v.1.5.4              |    56,384,139 |      33318.20 s |          0.77 s |     14697/9430 MB |                  99%/167% |
| LZ4 v.1.9.3               |   177,292,965 |         12.50 s |          0.93 s |            9/9 MB |                   99%/98% |
| Satanichi_GCC             |   102,964,818 |            N.A. |          1.14 s |              N.A. |                 N.A./100% |
| Satanichi_CLANG           |   102,964,818 |      18000.03 s |          1.20 s |   124851/38567 MB |                  99%/100% |
| Satanichi_ICL             |   102,964,818 |            N.A. |          1.20 s |              N.A. |                 N.A./100% |
| ZSTD v.1.5.4 --no-asyncio |    56,384,139 |      33318.20 s |          1.26 s |     14697/9430 MB |                   99%/98% |
| libdeflate v.1.17         |   152,298,598 |        207.16 s |          1.30 s |      1046/1187 MB |                   99%/91% |
| Brotli v.1.0.9            |    45,549,174 |       1242.62 s |          1.65 s |     10334/9251 MB |                   99%/99% |
| Crush                     |   119,122,603 |        857.14 s |          1.75 s |        145/147 MB |                   99%/95% |
| RAR v.6.20                |    74,857,944 |         39.65 s |          1.94 s |      6225/6203 MB |                1148%/150% |
| BriefLZ v.1.3.0           |    76,708,136 |     702977.76 s |          1.99 s |    68103/24636 MB |                   99%/96% |
| 7zip's GZ v.22.01         |   153,408,516 |        491.84 s |          3.11 s |            5/9 MB |                   99%/99% |
| RAZOR v.1.03.7            |    35,680,853 |       1789.96 s |          4.28 s |     12048/6332 MB |                  135%/98% |
| BSC v.3.2.5               |    41,801,676 |         25.60 s |         11.07 s |      5176/5168 MB |                1451%/763% |
| 7zip's BZ2 v.22.01        |   111,014,382 |         80.72 s |         11.41 s |        188/147 MB |                1579%/125% |
| BWTSATAN                  |    46,152,308 |         35.74 s |         13.60 s |      4881/3275 MB |                   99%/99% |
| ZPAQ v.7.05               |    45,600,199 |       2720.84 s |             ? s |    19810/16978 MB |                     99%/? |
+---------------------------+---------------+-----------------+-----------------+-------------------+---------------------------+

Note1: The Decompression time includes LOAD-DECOMPRESS-DUMP times, that is, external-RAM -> internal-RAM -> external-RAM. Note2: The decompression (the best out of three attempts was taken) is done on RamDisk of size 12GB, both the compressed and the decompressed files are on it. Note3: Should LZSSE be threaded, it would scream insanely. Note4: Both BriefLZ and Crush were compiled with Intel Compiler v15.0, /O3 was used. Note5: Satanichi monstrously devours physical RAM, like 3TB, in order to flex its muscles. ! RAM needed to house B-trees: 34,957MB; RAM needed to build B-trees IN ONE PASS: 64,960MB ! Note6: Histogram (rotate it 90 degrees clockwise) of Unique-Building-Blocks appearing 2[+] times (e.g. 'alfalfa-alfa' stream has 1 BB with length 4 appearing 3 times thus adding 1 to the count for {004]): 17,973,441{256] 21,679,496{128] 37,159,393{064] 45,523,710{056] 53,933,809{048] 65,726,702{036] 70,875,569{028] 70,934,076{024] 63,444,254{018] 57,437,350{016] 48,764,156{014] 38,481,382{012] 27,075,494{010] 16,310,907{008] 6,995,867{006] 1,251,825{004]

Finally, let's summarize.

Taking LZSSE as a baseline, its cumulative decompression speed score (for all the nine corpora) is: 0.08+0.21+0.49+0.81+0.85+0.95+0.94+1.03+0.72=6.08 seconds, which is 1.00x Taking LZSSE as a baseline, its cumulative compression strength score (for all the nine corpora) is: 23,423,825+70,011,827+215,528,254+428,308,549+193,412,757+259,488,690+244,017,212+415,187,401+177,947,568=2,027,326,083 bytes, which is 1.00x

Thus, talking digits not words, the roster of TOP-5 speedy ones, sorted by 3rd column:

+---------------------------+-----------------------+----------------------+
| [De]compressor,           | Compression Strength, | Decompression Speed, |
| MAX settings              | Bigger-Is-Better      | Bigger-Is-Worse      |
+---------------------------+-----------------------+----------------------+
| LZSSE                     |                 1.00x |                1.00x |
| LZ4 v.1.9.3               |                 0.98x |                1.23x |
| ZSTD v.1.5.4              |                 1.84x |                1.49x |
| Satanichi_GCC             |                 1.30x |                1.72x |
| libdeflate v.1.17         |                 1.17x |                1.83x |
+---------------------------+-----------------------+----------------------+
| RAZOR v.1.03.7            |                 2.11x |                9.88x |
+---------------------------+-----------------------+----------------------+

LZ4 v.1.9.3 speed sums: 0.07+0.24+0.53+0.97+1.08+1.13+1.22+1.32+0.93=7.49 seconds, which is 7.49/6.08=1.23x LZ4 v.1.9.3 strength sums: 25,218,044+77,690,671+187,022,882+453,298,991+192,926,894+224,182,293+261,571,544+451,557,802+177,292,965=2,050,762,086 bytes, which is 2,027,326,083/2,050,762,086=0.98x

ZSTD v.1.5.4 speed sums: 0.07+0.33+0.43+1.59+1.02+0.91+1.39+2.57+0.77=9.08 seconds, which is 9.08/6.08=1.49x ZSTD v.1.5.4 strength sums: 12,240,928+42,349,272+117,008,878+262,887,821+103,971,207+144,422,202+130,101,184+229,379,374+56,384,139=1,098,745,005 bytes, which is 2,027,326,083/1,098,745,005=1.84x

libdeflate v.1.17 speed sums: 0.09+0.37+0.89+1.61+1.45+1.56+1.67+2.19+1.30=11.13 seconds, which is 11.13/6.08=1.83x libdeflate v.1.17 strength sums: 20,349,168+66,803,854+176,912,170+363,952,523+155,206,929+203,879,721+206,272,933+374,386,581+152,298,598=1,720,062,477 bytes, which is 2,027,326,083/1,720,062,477=1.17x

Satanichi_GCC speed sums: 0.08+0.39+0.58+1.56+1.35+1.23+1.65+2.49+1.14=10.47 seconds, which is 10.47/6.08=1.72x Satanichi_GCC strength sums: 17,557,771+57,046,870+147,421,238+361,785,996+163,502,460+196,488,789+197,914,434+310,305,324+102,964,818=1,554,987,700 bytes, which is 2,027,326,083/1,554,987,700=1.30x

And adding for good measure the GOATSF (Greatest-Of-All-Time-So-Far):

RAZOR v.1.03.7 speed sums: 0.51+2.49+3.18+12.06+5.76+6.03+7.74+18.03+4.28=60.08 seconds, which is 60.08/6.08=9.88x RAZOR v.1.03.7 strength sums: 11,256,631+35,514,213+113,458,656+235,256,567+87,975,377+136,135,276+107,623,194+196,408,591+35,680,853=959,309,358 bytes, which is 2,027,326,083/959,309,358=2.11x

Roughly, the dump speed is analogous to PCIe Gen 5.0 SSD i.e. ~10GB/s, the ImDisk (build 20190130) RAMDISK in use is somewhat slowish.

facebook / zstd

Benchmarking Zstd versus 14 decompressors #3521