search

richox / orz

a high performance, general purpose data compressor written in the crab-lang
MIT License
805 stars 53 forks source link

Performance? #28

Open chr4 opened 1 year ago

chr4 commented 1 year ago

I was super excited to see this! I'm currently looking for a fast compression alternative for zstd for compressing postgresql wal archives.

At least for this use-case, I wasn't able to reproduce the benchmarks you've provided.

(orz v1.6.2 installed using cargo install as described in the README, also tested with cargo build --release from current HEAD):

$ zstd 00000003000025EF0000007C
00000003000025EF0000007C : 50.55%   (  16.0 MiB =>   8.09 MiB, 00000003000025EF0000007C.zst)
'zstd 00000003000025EF0000007C' time: 0.064s, cpu: 104%

orz encode -l0 00000003000025EF0000007C 00000003000025EF0000007C.orz
[INFO] encode: 16777216 bytes => 8111757 bytes, 25.301MB/s
[INFO] statistics:
[INFO]   size:  16777216 bytes => 8111839 bytes
[INFO]   ratio: 48.35%
[INFO]   time:  0.669 sec

Which is factor ~10 slower than zstd :(

Platform: M1 Apple Silicon macOS (native), x86_64 Linux (musl cross-compiled)

richox commented 1 year ago

zstd's default compress level is much faster (and normally has worse compress ratio) than orz. however your input file seems not to have much redundant, so getting a similar output size. orz should perform better on data that have more redundant, like plain texts, logs, etc.