search

NVIDIA / spark-rapids

Spark RAPIDS plugin - accelerate Apache Spark with GPUs
https://nvidia.github.io/spark-rapids
Apache License 2.0
787 stars 228 forks source link

[BUG] some tests broken in the JDK11 build #2091

Open gerashegalov opened 3 years ago

gerashegalov commented 3 years ago

Describe the bug some tests fail when the repo is build using JDK11

Steps/Code to reproduce bug

JAVA_HOME=/usr/lib/jvm/java-1.11.0-openjdk-amd64 mvn clean verify -Dcuda.version=cuda11 -Dtests="test GpuArrowColumnarBatchBuilder retains reference of ArrowBuf"

GpuCoalesceBatchesSuite:
- test GpuArrowColumnarBatchBuilder retains reference of ArrowBuf *** FAILED ***
  java.lang.UnsupportedOperationException: sun.misc.Unsafe or java.nio.DirectByteBuffer.<init>(long, int) not available
  at io.netty.util.internal.PlatformDependent.directBuffer(PlatformDependent.java:490)
  at io.netty.buffer.NettyArrowBuf.getDirectBuffer(NettyArrowBuf.java:243)
  at io.netty.buffer.NettyArrowBuf.nioBuffer(NettyArrowBuf.java:233)
  at io.netty.buffer.NettyArrowBuf.nioBuffer(NettyArrowBuf.java:223)
  at io.netty.buffer.ArrowBuf.nioBuffer(ArrowBuf.java:241)
  at com.nvidia.spark.rapids.shims.spark300.Spark300Shims.getArrowDataBuf(Spark300Shims.scala:486)
  at com.nvidia.spark.rapids.HostColumnarToGpu$.$anonfun$arrowColumnarCopy$1(HostColumnarToGpu.scala:96)
  at com.nvidia.spark.rapids.HostColumnarToGpu$.getBufferAndAddReference$1(HostColumnarToGpu.scala:90)
  at com.nvidia.spark.rapids.HostColumnarToGpu$.arrowColumnarCopy(HostColumnarToGpu.scala:96)
  at com.nvidia.spark.rapids.HostColumnarToGpu.arrowColumnarCopy(HostColumnarToGpu.scala)

Expected behavior

JDK 11 is supported with Spark 3 with the caveat, h/t @pxLi

For Python 3.9, Arrow optimization and pandas UDFs might not work due to the supported Python versions in Apache Arrow. Please refer to the latest Python Compatibility page. For Java 11, -Dio.netty.tryReflectionSetAccessible=true is required additionally for Apache Arrow library. This prevents java.lang.UnsupportedOperationException: sun.misc.Unsafe or java.nio.DirectByteBuffer.(long, int) not available when Apache Arrow uses Netty internally.

Environment details (please complete the following information)

$ /usr/lib/jvm/java-1.11.0-openjdk-amd64/bin/java -version 
openjdk version "11.0.10" 2021-01-19
OpenJDK Runtime Environment (build 11.0.10+9-Ubuntu-0ubuntu1.18.04)
OpenJDK 64-Bit Server VM (build 11.0.10+9-Ubuntu-0ubuntu1.18.04, mixed mode, sharing)

Additional context Related to #2088

tgravescs commented 2 years ago

fyi - the community is looking at supporting jdk 17 for Spark 3.3 -> https://issues.apache.org/jira/browse/SPARK-33772

gerashegalov commented 2 years ago

looks related to https://github.com/netty/netty/issues/8708