AbsaOSS / ABRiS

Avro SerDe for Apache Spark structured APIs.
Apache License 2.0
230 stars 75 forks source link

Spark 3.20 <-> Scala 2.13 #235

Closed ex0ns closed 2 years ago

ex0ns commented 3 years ago

Spark 3.2.0 was released on October 20th, it's the first release of spark with support for Scala 2.13, is there any plan to support this version, if yes what would be the timeline ?

The first issue that comes to my mind is related to the upgrade to spark-avro 3.2.0, I've given this a try, and after a few changes:

diff --git a/pom.xml b/pom.xml
index a782ddd..800f283 100644
--- a/pom.xml
+++ b/pom.xml
@@ -51,6 +51,7 @@
         <spark-23.version>2.3.4</spark-23.version>
         <spark-24.version>2.4.6</spark-24.version>
         <spark-30.version>3.0.0</spark-30.version>
+        <spark-32.version>3.2.0</spark-32.version>

         <spark.version>${spark-24.version}</spark.version>

@@ -61,6 +62,7 @@

         <scala_2.11.version>2.11.12</scala_2.11.version>
         <scala_2.12.version>2.12.10</scala_2.12.version>
+        <scala_2.13.version>2.13.6</scala_2.13.version>

         <!-- Controlled by `scala-cross-build` plugin -->
         <scala.version>2.12.10</scala.version>
@@ -79,6 +81,7 @@

         <!--Libs-->
         <spark.avro.version>2.4.6</spark.avro.version>
+        <spark-32.avro.version>3.2.0</spark-32.avro.version>
         <avro.version>1.8.2</avro.version>

         <scm.url><!-- defined outside the pom --></scm.url>
@@ -323,6 +326,13 @@
                 <scala.version>${scala_2.12.version}</scala.version>
             </properties>
         </profile>
+        <profile>
+            <id>scala-2.13</id>
+            <properties>
+                <scala.compat.version>2.13</scala.compat.version>
+                <scala.version>${scala_2.13.version}</scala.version>
+            </properties>
+        </profile>

         <!-- Spark profiles -->
         <profile>
@@ -343,6 +353,13 @@
                 <spark.version>${spark-30.version}</spark.version>
             </properties>
         </profile>
+        <profile>
+            <id>spark-3.2</id>
+            <properties>
+                <spark.version>${spark-32.version}</spark.version>
+                <spark.avro.version>${spark-32.avro.version}</spark.avro.version>
+            </properties>
+        </profile>

         <profile>
             <id>uber</id>

Thanks