Add native Lz4, Snappy, and Zstd

[dain]

Add support for native Lz4, Snappy, and Zstd using the java.lang.foreign APIs

[wendigo]

That's awesome!

[wendigo]

I think that you could add Enable-Native-Access: ALL-UNNAMED to the MANIFEST.MF entries (https://docs.oracle.com/en/java/javase/22/core/restricted-methods.html)

[wendigo]

@dain we should apply for ARM runners so we can add a coverage here as well

[wendigo]

Right now some of the tests are not passing on ARM (due to the lack of the libgplcompression for aarch64)

[wendigo]

Some benchmarks:

jmh-result.json benchmarks.log

[dain]

@wendigo

I think that you could add Enable-Native-Access: ALL-UNNAMED to the MANIFEST.MF entries (https://docs.oracle.com/en/java/javase/22/core/restricted-methods.html)

Ya, I think we should make this a proper module, not that most people use module capable systems.... follow up work

@dain we should apply for ARM runners so we can add a coverage here as well

Looks like the beta is open now https://github.blog/2024-06-03-arm64-on-github-actions-powering-faster-more-efficient-build-systems/

Some benchmarks:

For benchmarks, you'll want to look at these algorithms:

• airlift_lz4
• airlift_lz4_native
• airlift_lzo
• airlift_snappy
• airlift_snappy_native
• airlift_zstd
• airlift_zstd_native

Then in the DataSet you'll want to narrow down to one of the collections, or it will take forever.

[wendigo]

  compress    airlift_lz4             calgary/book2                  333,498   375.5MB/s ±    29.4MB/s ( 7.82%) (N = 3, α = 99.9%)
  compress    airlift_lz4_native      calgary/book2                  333,498   461.9MB/s ±    47.4MB/s (10.27%) (N = 3, α = 99.9%)
  compress    airlift_snappy          calgary/book2                  334,111   357.4MB/s ±    34.5MB/s ( 9.64%) (N = 3, α = 99.9%)
  compress    airlift_snappy_native   calgary/book2                  334,941   529.0MB/s ±   139.7MB/s (26.41%) (N = 3, α = 99.9%)
  compress    airlift_zstd            calgary/book2                  205,814   149.4MB/s ±    49.8MB/s (33.30%) (N = 3, α = 99.9%)
  compress    airlift_zstd_native     calgary/book2                  203,941   236.8MB/s ±    63.9MB/s (26.98%) (N = 3, α = 99.9%)
  decompress  airlift_lz4             calgary/book2                  333,498  2713.4MB/s ±   616.6MB/s (22.73%) (N = 3, α = 99.9%)
  decompress  airlift_lz4_native      calgary/book2                  333,498  3553.0MB/s ±   959.0MB/s (26.99%) (N = 3, α = 99.9%)
  decompress  airlift_snappy          calgary/book2                  334,111   735.0MB/s ±    26.7MB/s ( 3.64%) (N = 3, α = 99.9%)
  decompress  airlift_snappy_native   calgary/book2                  334,941  2225.0MB/s ±   105.1MB/s ( 4.72%) (N = 3, α = 99.9%)
  decompress  airlift_zstd            calgary/book2                  205,814   817.0MB/s ±    16.6MB/s ( 2.04%) (N = 3, α = 99.9%)
  decompress  airlift_zstd_native     calgary/book2                  203,941  1115.3MB/s ±   169.5MB/s (15.19%) (N = 3, α = 99.9%)

[wendigo]

Can you move the code to io.airlift.compressor package before merging? So it doesn't clash with the old aircompressor. That way we will be able to add a bridge that glues two APIs together.

[electrum]

you can specify the JAR-file manifest attribute Enable-Native-Access: ALL-UNNAMED in an executable JAR to enable warning-free use by all code on the class path

I think the key wording here is "an executable JAR". So this only works for the application's JAR being run with java -jar foo.jar. Otherwise, if any JAR could set this, it wouldn't protect the application (which is their goal with all of this).

[electrum]

Typo in commit message Add MessageSegment support ... should be MemorySegment