Cyan4973 / FiniteStateEntropy

New generation entropy codecs : Finite State Entropy and Huff0
BSD 2-Clause "Simplified" License
1.34k stars 144 forks source link

Could use updated benchmarks; identify OS #80

Closed JoeUX closed 7 years ago

JoeUX commented 7 years ago

A few notes on the benchmarks:

They're out of date

Given all the releases since those benchmarks, it would be helpful to have updated results for Zstd ≥ 1.1.4 and brotli ≥ 0.5.2.

Different compilers

Note also that the benchmarks here on GitHub use two different compilers: gcc 5.4 for the table, and gcc 5.2.1 for the graphs. Unless there's a compelling reason, there's no point in introducing a confound like two different compilers.

No operating system

There's no mention of an OS in any of the benchmarks, not even broad families like Linux vs. Windows.

Summary

Ideally, what we want is: updated builds, same and updated compiler, identify the OS, report the amount of memory (and type) the system has and whether it's SSD or spinner (unless the benchmarks are in-memory only).

I've got some useful machines for benchmarking, but lzbench only works with gcc (on Windows). I think @inikep and others would be better for gcc benchmarks. I could do Visual Studio 2017 or 2015 benchmarks.

Cyan4973 commented 7 years ago

If your comment apply to zstd project, please note that this issue is filled on the fse issue board.

JoeUX commented 7 years ago

Oh damn, yeah that's a mistake. Sorry. I'll move it over.