[BUG] Cloudera Shims don't compile using `scala.maven.plugin` version 4.9.1

[razajafri]

To build Shims against JDK 17 we need to upgrade our maven plugin to circumvent the problem of relying on Java 8 runtime but when we do that the Cloudera shims fail compilation with the following errors

[INFO] compiling 459 Scala sources and 58 Java sources to /home/runner/work/spark-rapids/spark-rapids/sql-plugin/target/spark332cdh/classes ...
Error:  /home/runner/work/spark-rapids/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/MetaUtils.scala:112: value createUnintializedVector is not a member of com.google.flatbuffers.FlatBufferBuilder
Error:  /home/runner/work/spark-rapids/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/MetaUtils.scala:235: value createUnintializedVector is not a member of com.google.flatbuffers.FlatBufferBuilder
Error:  /home/runner/work/spark-rapids/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/MetaUtils.scala:252: type ByteBufferFactory is not a member of object com.google.flatbuffers.FlatBufferBuilder
Error:  /home/runner/work/spark-rapids/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/MetaUtils.scala:303: value createUnintializedVector is not a member of com.google.flatbuffers.FlatBufferBuilder
Error:  /home/runner/work/spark-rapids/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/MetaUtils.scala:319: overloaded method constructor FlatBufferBuilder with alternatives:
  (x$1: java.nio.ByteBuffer)com.google.flatbuffers.FlatBufferBuilder <and>
  ()com.google.flatbuffers.FlatBufferBuilder <and>
  (x$1: Int)com.google.flatbuffers.FlatBufferBuilder
 cannot be applied to (Int, com.nvidia.spark.rapids.DirectByteBufferFactory)
Error:  /home/runner/work/spark-rapids/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/MetaUtils.scala:328: overloaded method constructor FlatBufferBuilder with alternatives:
  (x$1: java.nio.ByteBuffer)com.google.flatbuffers.FlatBufferBuilder <and>
  ()com.google.flatbuffers.FlatBufferBuilder <and>
  (x$1: Int)com.google.flatbuffers.FlatBufferBuilder
 cannot be applied to (Int, com.nvidia.spark.rapids.DirectByteBufferFactory)
Error:  /home/runner/work/spark-rapids/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/MetaUtils.scala:342: overloaded method constructor FlatBufferBuilder with alternatives:
  (x$1: java.nio.ByteBuffer)com.google.flatbuffers.FlatBufferBuilder <and>
  ()com.google.flatbuffers.FlatBufferBuilder <and>
  (x$1: Int)com.google.flatbuffers.FlatBufferBuilder
 cannot be applied to (Int, com.nvidia.spark.rapids.DirectByteBufferFactory)
Error:  /home/runner/work/spark-rapids/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/MetaUtils.scala:368: overloaded method constructor FlatBufferBuilder with alternatives:
  (x$1: java.nio.ByteBuffer)com.google.flatbuffers.FlatBufferBuilder <and>
  ()com.google.flatbuffers.FlatBufferBuilder <and>
  (x$1: Int)com.google.flatbuffers.FlatBufferBuilder
 cannot be applied to (Int, com.nvidia.spark.rapids.DirectByteBufferFactory)
Error:  /home/runner/work/spark-rapids/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/MetaUtils.scala:322: local val finIndex in method buildMetaResponse is never used
Error:  /home/runner/work/spark-rapids/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/MetaUtils.scala:337: local val finIndex in method buildShuffleMetadataRequest is never used
Error:  /home/runner/work/spark-rapids/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/MetaUtils.scala:375: local val root in method buildBufferTransferResponse is never used
Error:  /home/runner/work/spark-rapids/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/MetaUtils.scala:390: local val transferRequestOffset in method buildTransferRequest is never used
Error: [ERROR] 12 errors found

Until we fix this the Cloudera shims cannot build against JDK 17. Look at https://github.com/NVIDIA/spark-rapids/actions/runs/9732795299/job/26860542937 for more details

[gerashegalov]

I cannot reproduce this issue. Please provide the exact command, Maven and JDK versions

[gerashegalov]

https://github.com/NVIDIA/spark-rapids/pull/10994#issuecomment-2204548267

[razajafri]

This can be repro'd in a docker using the instructions mentioned on https://github.com/NVIDIA/spark-rapids/pull/10994#issuecomment-2204966170

[gerashegalov]

To reproduce in branch-24.08

# Apply patch
$ git diff
diff --git a/shim-deps/cloudera/pom.xml b/shim-deps/cloudera/pom.xml
index f25af721a..37fdaed56 100644
--- a/shim-deps/cloudera/pom.xml
+++ b/shim-deps/cloudera/pom.xml
@@ -108,7 +108,7 @@
                                 <requireJavaVersion>
                                     <message>Only Java 8, 11 are supported for Cloudera! For details please
                                         see https://github.com/NVIDIA/spark-rapids/issues/11112</message>
-                                    <version>(,17)</version>
+                                    <version>(,18)</version>
                                 </requireJavaVersion>
                             </rules>
                         </configuration>
$ rm -rf /tmp/.m2
$ docker run -u $(id -u):$(id -g) --rm -it \
  -e JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64 \
  -v /tmp:/tmp:rw -v /etc/group:/etc/group:ro -v /etc/passwd:/etc/passwd:ro -v /etc/shadow:/etc/shadow:ro -v /etc/sudoers.d:/etc/sudoers.d:ro -v $PWD:$PWD:rw  -v $HOME/.m2:$HOME/.m2:rw \
  --workdir $PWD urm.nvidia.com/sw-spark-docker-local/plugin:dev-ubuntu20-cuda11.0.3-blossom-dev \
  mvn clean package -pl integration_tests,tests,tools -am  \
  -Dbuildver=321cdh -DskipTests -Dmaven.scaladoc.skip   -Dscala.plugin.version=4.7.1 -Dmaven.repo.local=/tmp/.m2