ZStd JNI vs Aircompressor pure java performance question

[nknize]

OpenSearch is evaluating the pure java implementation of Zstd using AirCompressor. I stumbled on a thoughtful comment here which certainly aligns to the reasons we avoid jni code as "top level" modules or plugins on OpenSearch core (we have similar reasons on Lucene). We realize that comment is now five years old, and so @reta ran benchmarks that seem to show significant performance differences between the pure java implementation (hotspot has gotten better of course) and jni. Do these number look valid to folks on this project? What are the potential pitfalls to running those benchmarks that we need to be aware of? Are there certain config conditions that should be followed to squeeze better performance?

Thanks in advance for any assistance that can be provided.

[reta]

For context, I run the io.airlift.compress.benchmark.CompressionBenchmark (i7-10750H × 12, 64Gb, Linux/Ubuntu, JDK 17.0.8), no modifications, selectively picked the airlift_zstd / zstd_jni since this is what we've been looking for, thank you.

[nknize]

Quick ping check in on this... any maintainers here able to give some guidance?

[tianjiqx]

Perhaps I have the same test results and found that zstd_jni actually performs better. The same implementation of Lz4 also results in better performance for JNI.

Jdk: 17 Os: ubuntu22.04 CPU: AMD ® Ryzen 7 7840hs