search

airlift / aircompressor

A port of Snappy, LZO, LZ4, and Zstandard to Java
Apache License 2.0
562 stars 111 forks source link

Version 0.19 can't work with heap buffer on JDK1.8 #133

Closed codelipenghui closed 3 years ago

codelipenghui commented 3 years ago

To reproduce:

ByteBuf source = ByteBufAllocator.DEFAULT.heapBuffer(1024);
byte[] bytes = new byte[1024];
source.writeBytes(bytes); 
ZstdCompressor compressor = new ZstdCompressor();
int maxLength = compressor.maxCompressedLength(source.readableBytes());
ByteBuf target = ByteBufAllocator.DEFAULT.heapBuffer(10240);
compressor.compress(source.nioBuffer(), target.nioBuffer(0, maxLength));

Output:

/Library/Java/JavaVirtualMachines/jdk1.8.0_281.jdk/Contents/Home/bin/java -agentlib:jdwp=transport=dt_socket,address=127.0.0.1:51570,suspend=y,server=n -javaagent:/Users/lipenghui/Library/Caches/JetBrains/IntelliJIdea2021.1/captureAgent/debugger-agent.jar -Dfile.encoding=UTF-8 -classpath /Library/Java/JavaVirtualMachines/jdk1.8.0_281.jdk/Contents/Home/jre/lib/charsets.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_281.jdk/Contents/Home/jre/lib/deploy.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_281.jdk/Contents/Home/jre/lib/ext/cldrdata.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_281.jdk/Contents/Home/jre/lib/ext/dnsns.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_281.jdk/Contents/Home/jre/lib/ext/jaccess.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_281.jdk/Contents/Home/jre/lib/ext/jfxrt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_281.jdk/Contents/Home/jre/lib/ext/localedata.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_281.jdk/Contents/Home/jre/lib/ext/nashorn.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_281.jdk/Contents/Home/jre/lib/ext/sunec.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_281.jdk/Contents/Home/jre/lib/ext/sunjce_provider.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_281.jdk/Contents/Home/jre/lib/ext/sunpkcs11.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_281.jdk/Contents/Home/jre/lib/ext/zipfs.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_281.jdk/Contents/Home/jre/lib/javaws.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_281.jdk/Contents/Home/jre/lib/jce.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_281.jdk/Contents/Home/jre/lib/jfr.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_281.jdk/Contents/Home/jre/lib/jfxswt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_281.jdk/Contents/Home/jre/lib/jsse.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_281.jdk/Contents/Home/jre/lib/management-agent.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_281.jdk/Contents/Home/jre/lib/plugin.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_281.jdk/Contents/Home/jre/lib/resources.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_281.jdk/Contents/Home/jre/lib/rt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_281.jdk/Contents/Home/lib/ant-javafx.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_281.jdk/Contents/Home/lib/dt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_281.jdk/Contents/Home/lib/javafx-mx.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_281.jdk/Contents/Home/lib/jconsole.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_281.jdk/Contents/Home/lib/packager.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_281.jdk/Contents/Home/lib/sa-jdi.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_281.jdk/Contents/Home/lib/tools.jar:/Users/lipenghui/GitHub/pulsar-test/target/classes:/Users/lipenghui/.m2/repository/org/apache/avro/avro/1.9.2/avro-1.9.2.jar:/Users/lipenghui/.m2/repository/com/fasterxml/jackson/core/jackson-core/2.10.2/jackson-core-2.10.2.jar:/Users/lipenghui/.m2/repository/com/fasterxml/jackson/core/jackson-databind/2.10.2/jackson-databind-2.10.2.jar:/Users/lipenghui/.m2/repository/com/fasterxml/jackson/core/jackson-annotations/2.10.2/jackson-annotations-2.10.2.jar:/Users/lipenghui/.m2/repository/org/apache/commons/commons-compress/1.19/commons-compress-1.19.jar:/Users/lipenghui/.m2/repository/org/slf4j/slf4j-api/1.7.25/slf4j-api-1.7.25.jar:/Users/lipenghui/.m2/repository/org/apache/pulsar/pulsar-client-original/2.9.0-SNAPSHOT/pulsar-client-original-2.9.0-SNAPSHOT.jar:/Users/lipenghui/.m2/repository/org/apache/pulsar/pulsar-client-api/2.9.0-SNAPSHOT/pulsar-client-api-2.9.0-SNAPSHOT.jar:/Users/lipenghui/.m2/repository/org/apache/pulsar/pulsar-common/2.9.0-SNAPSHOT/pulsar-common-2.9.0-SNAPSHOT.jar:/Users/lipenghui/.m2/repository/org/apache/pulsar/pulsar-client-admin-api/2.9.0-SNAPSHOT/pulsar-client-admin-api-2.9.0-SNAPSHOT.jar:/Users/lipenghui/.m2/repository/io/swagger/swagger-annotations/1.6.2/swagger-annotations-1.6.2.jar:/Users/lipenghui/.m2/repository/com/google/guava/guava/30.1-jre/guava-30.1-jre.jar:/Users/lipenghui/.m2/repository/com/google/guava/failureaccess/1.0.1/failureaccess-1.0.1.jar:/Users/lipenghui/.m2/repository/com/google/guava/listenablefuture/9999.0-empty-to-avoid-conflict-with-guava/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/Users/lipenghui/.m2/repository/com/google/code/findbugs/jsr305/3.0.2/jsr305-3.0.2.jar:/Users/lipenghui/.m2/repository/org/checkerframework/checker-qual/3.5.0/checker-qual-3.5.0.jar:/Users/lipenghui/.m2/repository/com/google/errorprone/error_prone_annotations/2.3.4/error_prone_annotations-2.3.4.jar:/Users/lipenghui/.m2/repository/com/google/j2objc/j2objc-annotations/1.3/j2objc-annotations-1.3.jar:/Users/lipenghui/.m2/repository/io/netty/netty-transport-native-unix-common/4.1.66.Final/netty-transport-native-unix-common-4.1.66.Final-linux-x86_64.jar:/Users/lipenghui/.m2/repository/org/apache/bookkeeper/cpu-affinity/4.14.1/cpu-affinity-4.14.1.jar:/Users/lipenghui/.m2/repository/io/airlift/aircompressor/0.19/aircompressor-0.19.jar:/Users/lipenghui/.m2/repository/io/netty/netty-codec-haproxy/4.1.66.Final/netty-codec-haproxy-4.1.66.Final.jar:/Users/lipenghui/.m2/repository/org/eclipse/jetty/jetty-util/9.4.42.v20210604/jetty-util-9.4.42.v20210604.jar:/Users/lipenghui/.m2/repository/com/fasterxml/jackson/dataformat/jackson-dataformat-yaml/2.12.3/jackson-dataformat-yaml-2.12.3.jar:/Users/lipenghui/.m2/repository/org/yaml/snakeyaml/1.27/snakeyaml-1.27.jar:/Users/lipenghui/.m2/repository/javax/ws/rs/javax.ws.rs-api/2.1/javax.ws.rs-api-2.1.jar:/Users/lipenghui/.m2/repository/org/apache/pulsar/pulsar-transaction-common/2.9.0-SNAPSHOT/pulsar-transaction-common-2.9.0-SNAPSHOT.jar:/Users/lipenghui/.m2/repository/org/apache/pulsar/bouncy-castle-bc/2.9.0-SNAPSHOT/bouncy-castle-bc-2.9.0-SNAPSHOT-pkg.jar:/Users/lipenghui/.m2/repository/io/netty/netty-codec-http/4.1.66.Final/netty-codec-http-4.1.66.Final.jar:/Users/lipenghui/.m2/repository/io/netty/netty-common/4.1.66.Final/netty-common-4.1.66.Final.jar:/Users/lipenghui/.m2/repository/io/netty/netty-buffer/4.1.66.Final/netty-buffer-4.1.66.Final.jar:/Users/lipenghui/.m2/repository/io/netty/netty-transport/4.1.66.Final/netty-transport-4.1.66.Final.jar:/Users/lipenghui/.m2/repository/io/netty/netty-codec/4.1.66.Final/netty-codec-4.1.66.Final.jar:/Users/lipenghui/.m2/repository/io/netty/netty-handler-proxy/4.1.66.Final/netty-handler-proxy-4.1.66.Final.jar:/Users/lipenghui/.m2/repository/io/netty/netty-codec-socks/4.1.66.Final/netty-codec-socks-4.1.66.Final.jar:/Users/lipenghui/.m2/repository/io/netty/netty-resolver-dns/4.1.66.Final/netty-resolver-dns-4.1.66.Final.jar:/Users/lipenghui/.m2/repository/io/netty/netty-resolver/4.1.66.Final/netty-resolver-4.1.66.Final.jar:/Users/lipenghui/.m2/repository/io/netty/netty-codec-dns/4.1.66.Final/netty-codec-dns-4.1.66.Final.jar:/Users/lipenghui/.m2/repository/org/apache/commons/commons-lang3/3.11/commons-lang3-3.11.jar:/Users/lipenghui/.m2/repository/org/asynchttpclient/async-http-client/2.12.1/async-http-client-2.12.1.jar:/Users/lipenghui/.m2/repository/org/asynchttpclient/async-http-client-netty-utils/2.12.1/async-http-client-netty-utils-2.12.1.jar:/Users/lipenghui/.m2/repository/org/reactivestreams/reactive-streams/1.0.3/reactive-streams-1.0.3.jar:/Users/lipenghui/.m2/repository/com/typesafe/netty/netty-reactive-streams/2.0.4/netty-reactive-streams-2.0.4.jar:/Users/lipenghui/.m2/repository/com/sun/activation/javax.activation/1.2.0/javax.activation-1.2.0.jar:/Users/lipenghui/.m2/repository/commons-codec/commons-codec/1.15/commons-codec-1.15.jar:/Users/lipenghui/.m2/repository/com/yahoo/datasketches/sketches-core/0.8.3/sketches-core-0.8.3.jar:/Users/lipenghui/.m2/repository/com/yahoo/datasketches/memory/0.8.3/memory-0.8.3.jar:/Users/lipenghui/.m2/repository/com/google/code/gson/gson/2.8.6/gson-2.8.6.jar:/Users/lipenghui/.m2/repository/org/apache/avro/avro-protobuf/1.10.2/avro-protobuf-1.10.2.jar:/Users/lipenghui/.m2/repository/com/fasterxml/jackson/module/jackson-module-jsonSchema/2.12.3/jackson-module-jsonSchema-2.12.3.jar:/Users/lipenghui/.m2/repository/javax/validation/validation-api/1.1.0.Final/validation-api-1.1.0.Final.jar:/Users/lipenghui/.m2/repository/net/jcip/jcip-annotations/1.0/jcip-annotations-1.0.jar:/Users/lipenghui/.m2/repository/org/apache/logging/log4j/log4j-slf4j-impl/2.10.0/log4j-slf4j-impl-2.10.0.jar:/Users/lipenghui/.m2/repository/org/apache/logging/log4j/log4j-api/2.10.0/log4j-api-2.10.0.jar:/Users/lipenghui/.m2/repository/org/apache/logging/log4j/log4j-core/2.10.0/log4j-core-2.10.0.jar:/Users/lipenghui/.m2/repository/org/apache/bookkeeper/bookkeeper-server/4.11.0/bookkeeper-server-4.11.0.jar:/Users/lipenghui/.m2/repository/org/apache/bookkeeper/bookkeeper-common/4.11.0/bookkeeper-common-4.11.0.jar:/Users/lipenghui/.m2/repository/org/apache/bookkeeper/stats/bookkeeper-stats-api/4.11.0/bookkeeper-stats-api-4.11.0.jar:/Users/lipenghui/.m2/repository/org/jctools/jctools-core/2.1.2/jctools-core-2.1.2.jar:/Users/lipenghui/.m2/repository/org/apache/bookkeeper/bookkeeper-common-allocator/4.11.0/bookkeeper-common-allocator-4.11.0.jar:/Users/lipenghui/.m2/repository/org/apache/bookkeeper/bookkeeper-proto/4.11.0/bookkeeper-proto-4.11.0.jar:/Users/lipenghui/.m2/repository/com/google/protobuf/protobuf-java/3.5.1/protobuf-java-3.5.1.jar:/Users/lipenghui/.m2/repository/org/apache/bookkeeper/bookkeeper-tools-framework/4.11.0/bookkeeper-tools-framework-4.11.0.jar:/Users/lipenghui/.m2/repository/org/rocksdb/rocksdbjni/5.13.1/rocksdbjni-5.13.1.jar:/Users/lipenghui/.m2/repository/org/apache/zookeeper/zookeeper/3.5.7/zookeeper-3.5.7.jar:/Users/lipenghui/.m2/repository/org/apache/zookeeper/zookeeper-jute/3.5.7/zookeeper-jute-3.5.7.jar:/Users/lipenghui/.m2/repository/org/apache/yetus/audience-annotations/0.5.0/audience-annotations-0.5.0.jar:/Users/lipenghui/.m2/repository/io/netty/netty-handler/4.1.32.Final/netty-handler-4.1.32.Final.jar:/Users/lipenghui/.m2/repository/io/netty/netty-transport-native-epoll/4.1.32.Final/netty-transport-native-epoll-4.1.32.Final-linux-x86_64.jar:/Users/lipenghui/.m2/repository/io/netty/netty-transport-native-unix-common/4.1.32.Final/netty-transport-native-unix-common-4.1.32.Final.jar:/Users/lipenghui/.m2/repository/io/netty/netty-tcnative-boringssl-static/2.0.20.Final/netty-tcnative-boringssl-static-2.0.20.Final.jar:/Users/lipenghui/.m2/repository/org/apache/bookkeeper/http/http-server/4.11.0/http-server-4.11.0.jar:/Users/lipenghui/.m2/repository/org/apache/bookkeeper/circe-checksum/4.11.0/circe-checksum-4.11.0.jar:/Users/lipenghui/.m2/repository/commons-cli/commons-cli/1.2/commons-cli-1.2.jar:/Users/lipenghui/.m2/repository/commons-io/commons-io/2.4/commons-io-2.4.jar:/Users/lipenghui/.m2/repository/org/apache/commons/commons-collections4/4.1/commons-collections4-4.1.jar:/Users/lipenghui/.m2/repository/org/bouncycastle/bcpkix-jdk15on/1.60/bcpkix-jdk15on-1.60.jar:/Users/lipenghui/.m2/repository/org/bouncycastle/bcprov-jdk15on/1.60/bcprov-jdk15on-1.60.jar:/Users/lipenghui/.m2/repository/org/bouncycastle/bcprov-ext-jdk15on/1.60/bcprov-ext-jdk15on-1.60.jar:/Users/lipenghui/.m2/repository/com/beust/jcommander/1.48/jcommander-1.48.jar:/Users/lipenghui/.m2/repository/net/java/dev/jna/jna/3.2.7/jna-3.2.7.jar:/Users/lipenghui/.m2/repository/org/apache/httpcomponents/httpclient/4.5.5/httpclient-4.5.5.jar:/Users/lipenghui/.m2/repository/org/apache/httpcomponents/httpcore/4.4.9/httpcore-4.4.9.jar:/Users/lipenghui/.m2/repository/commons-logging/commons-logging/1.2/commons-logging-1.2.jar:/Users/lipenghui/.m2/repository/commons-configuration/commons-configuration/1.10/commons-configuration-1.10.jar:/Users/lipenghui/.m2/repository/commons-lang/commons-lang/2.6/commons-lang-2.6.jar:/Users/lipenghui/Library/Application Support/JetBrains/Toolbox/apps/IDEA-U/ch-0/211.7628.21/IntelliJ IDEA.app/Contents/lib/idea_rt.jar io.streamnative.test.AirCompressorTest
Connected to the target VM, address: '127.0.0.1:51570', transport: 'socket'
Exception in thread "main" java.lang.NoSuchMethodError: java.nio.ByteBuffer.position(I)Ljava/nio/ByteBuffer;
    at io.airlift.compress.zstd.ZstdCompressor.compress(ZstdCompressor.java:101)
    at io.streamnative.test.AirCompressorTest.main(AirCompressorTest.java:19)

Context

JDK: 1.8.0_281 Aircompressor: 0.19

eolivelli commented 3 years ago

Would it be possible to cut a new release, building the artefacts with JDK8 or fixing the problem by explicitly forcing to use only JDK8 APIs ?

lhotari commented 3 years ago

The problem and solution is explained in this blog post: https://www.morling.dev/blog/bytebuffer-and-the-dreaded-nosuchmethoderror/ .

martint commented 3 years ago

Thanks for the report. This shouldn't be happening, but there's likely an issue with the build setup and how it's targeting Java 8. I'll take a look.

martint commented 3 years ago

We just released 0.20, which fixes this issue.