search

inikep / lzbench

lzbench is an in-memory benchmark of open-source LZ77/LZSS/LZMA compressors
885 stars 179 forks source link

Add ISA-L to the comparison #94

Open rhpvorderman opened 3 years ago

rhpvorderman commented 3 years ago

The Intel Storage Acceleration Library is an implementation of the DEFLATE algorithm written in assembly. As a result it is very fast. It can be used as a replacement for zlib.

Unlike libdeflate, ISA-L has a streaming implementation as well as an all-at-once implementation. Libdeflate only has an all-at-once implementation which makes it unsuitable for big compressed files of genomic data (FASTQ format) as these would need to be read into memory entirely.

It would be quite interesting to see how ISA-L compares to the rest of the compression programs. Especially since it is 100%-compatible with the zlib formats. This allows for smooth transition.

From my own experiments my ballpark estimates are that ISA-L is 6x faster when compressing data and 3x faster when decompressing data than zlib. So it would be a quite nice addition to this already impressive comparison of programs.

A disadvantage of adding it would be that it needs an assembler (NASM or YASM) besides a c-compiler since it is written in assembly with C-bindings provided. So it would complicate the build slightly.

jinfeihan57 commented 3 years ago

This “DEFLATE” can be used only by x86 CPUs. Am I right？

rhpvorderman commented 3 years ago

No. ARM64 is also supported. EDIT: And I have benchmarked it extensively on AMD processors at well. There are no Intel anti-competitive shennanigans going on.

rhpvorderman commented 3 years ago

@jinfeihan57 I have no problem with adding it myself when time allows. I am currently short for time in doing it. But I think it is an interesting comparison so I will try once time allows.

jinfeihan57 commented 3 years ago

Welcome to provide PR. Which API are you testing? Is that it? 94-114

rhpvorderman commented 3 years ago

Well those are the stateless functions. These are interesting, but the primary use case will be the streaming functions: https://github.com/intel/isa-l/blob/bee5180a1517f8b5e70b02fcd66790c623536c5d/include/igzip_lib.h#L795 . https://github.com/intel/isa-l/blob/bee5180a1517f8b5e70b02fcd66790c623536c5d/include/igzip_lib.h#L946

Streaming is essential when handling big data volumes. So testing these will be most representative for the performance I think.

ktfc76 commented 2 years ago

Hello, we're very interest in adding ISA-L into lzbench, is there any plan/progress for this?