airlift / aircompressor

A port of Snappy, LZO, LZ4, and Zstandard to Java
Apache License 2.0
562 stars 111 forks source link

Thoughts on using ByteBuffer instead of UNSAFE in zstd? #146

Closed dlazerka closed 1 year ago

dlazerka commented 2 years ago

Hi, Thanks for the tool!

I'm wondering if there are any pitfalls or other considerations in trying to implement the same thing, but instead of UNSAFE.* family of functions -- using standard ByteBuffers? Seems like a straightforward change, but maybe I'm missing something.

Pros would be:

  1. No dependency on sun.misc.* which prevents using modern JDK --release for modern versions (sun.misc is unavailable).
  2. Potentially easier on AOT compilation.
  3. In some sense it would be more "pure java".
  4. Would support sliced heap byte buffers (with .arrayOffset > 0).

The code of course supports ByteBuffers, but only with zero arrayOffset, and internally it just uses them just as a byte[].

martint commented 2 years ago

It's mostly due to performance. It's currently hard (impossible?) to get the same performance as Unsafe when accessing ByteBuffers via their public APIs.

Once the Foreign Memory APIs (https://openjdk.org/jeps/393) stabilize and we update this library to a more recent version of the JDK, we may be able to rewrite it against those APIs.