search

NVIDIA / spark-rapids

Spark RAPIDS plugin - accelerate Apache Spark with GPUs
https://nvidia.github.io/spark-rapids
Apache License 2.0
826 stars 236 forks source link

[BUG][SPARK-40436][BUILD] Upgrade Scala to 2.12.17 #10203

Open NVnavkumar opened 10 months ago

NVnavkumar commented 10 months ago

Describe the bug Spark updated the Scala version to 2.12.17 (See https://github.com/apache/spark/pull/37892)

EMR team found that these manifested in unit test failures in OpcodeSuite:

OpcodeSuite (rapids-4-spark-udf)
IFNONNULL opcode *** FAILED ***
[,z,] did not equal [,z,z] (OpcodeSuite.scala:62)
IFNE opcode *** FAILED ***
[1.0,2.0,2.0] did not equal [1.0,2.0,1.0] (OpcodeSuite.scala:62)
Conditional simplification - if (c) false else true => !c *** FAILED ***
[20,false] did not equal [20,true] (OpcodeSuite.scala:62)
TABLESWITCH TEST *** FAILED ***
[3,6] did not equal [3,9] (OpcodeSuite.scala:62)
LOOKUPSWITCH TEST *** FAILED ***
[100,200] did not equal [100,10000] (OpcodeSuite.scala:62)
java lang string builder test - append *** FAILED ***
[Hello,World,false,Hello World@@@ true] did not equal [Hello,World,false,World Hello!!! false] (OpcodeSuite.scala:62)
Conditional array buffer processing *** FAILED ***
[######hello,null,WrappedArray(######hello, null)] did not equal [######hello,null,WrappedArray(######hello, @@@@hello)] (OpcodeSuite.scala:62)
NVnavkumar commented 10 months ago

I was able to replicate this locally trying to build against Spark 3.5.0 using JDK 8 + Scala 2.12.17 by editing pom.xml and setting the scala.version to 2.12.17.

diff --git a/pom.xml b/pom.xml
index fd1c821cf..0d6ed32d9 100644
--- a/pom.xml
+++ b/pom.xml
@@ -639,7 +639,7 @@
             <id>scala-2.12</id>
             <properties>
                 <scala.binary.version>2.12</scala.binary.version>
-                <scala.version>2.12.15</scala.version>
+                <scala.version>2.12.17</scala.version>
             </properties>
         </profile>
         <profile>
@@ -727,7 +727,7 @@
         <scala.binary.version>2.12</scala.binary.version>
         <alluxio.client.version>2.8.0</alluxio.client.version>
         <scala.recompileMode>incremental</scala.recompileMode>
-        <scala.version>2.12.15</scala.version>
+        <scala.version>2.12.17</scala.version>
         <!--
         -processing
         to suppress unactionable "No processor claimed any of these annotations"
diff --git a/scala2.13/pom.xml b/scala2.13/pom.xml
index 5ccc17c4b..3a4806fe5 100644
--- a/scala2.13/pom.xml
+++ b/scala2.13/pom.xml
@@ -639,7 +639,7 @@
             <id>scala-2.12</id>
             <properties>
                 <scala.binary.version>2.12</scala.binary.version>
-                <scala.version>2.12.15</scala.version>
+                <scala.version>2.12.17</scala.version>
             </properties>
         </profile>
         <profile>

Interesting in that this issue doesn't occur in Scala 2.13 using JDK8 + Scala 2.13.8