A framework for writing performant user-defined functions (UDFs) that are portable across a variety of engines including Apache Spark, Apache Hive, and Presto.
BSD 2-Clause "Simplified" License
297
stars
73
forks
source link
Produce two artifacts in Spark for scala_2.11 and scala_2.12 #85
renames the module "transportable-udfs-spark" to "transportable-udfs-spark_2.11"
adds a module "transportable-udfs-spark_2.12" which uses same sourceset as _2.11
renames the module "transportable-udfs-test-spark" to "transportable-udfs-test-spark_2.11"
adds a module "transportable-udfs-test-spark_2.12" which uses same sourceset as _2.11
enhances transportable-udfs-plugin to plan producing + testing artifacts for scala_2.12
Outcome of this PR is that:
artifact for spark 2.x is renamed from "udf-name-spark" to "udf-name-spark_2.11"
an additional artifact is produced for spark 3.x called "udf-name-spark_2.12"
Testing can be done:
by building transportable-udfs-examples module
testing out build/libs/transportable-udfs-example-udfs-spark_2.11.jar in spark2.x environment
testing out build/libs/transportable-udfs-example-udfs-spark_2.12.jar in spark3.x environment
How it works?
Scala plugin's compileScala task automatically configures the classpath required to build scala code. It does so by looking for "scala-library.jar" in all dependencies declared in the compile classpath. If a module declares dependency on all scala_2.12 artifacts, the "scala-library.jar" that gets inferred is for scala_2.12. Similarly for building scala_2.11 artifact, all dependencies should be built against scala_2.11. In this PR we create separate modules for scala_2.11 and scala_2.12 with appropriately versioned dependencies.
Changes that this PR brings:
Outcome of this PR is that:
Testing can be done:
How it works? Scala plugin's compileScala task automatically configures the classpath required to build scala code. It does so by looking for "scala-library.jar" in all dependencies declared in the compile classpath. If a module declares dependency on all scala_2.12 artifacts, the "scala-library.jar" that gets inferred is for scala_2.12. Similarly for building scala_2.11 artifact, all dependencies should be built against scala_2.11. In this PR we create separate modules for scala_2.11 and scala_2.12 with appropriately versioned dependencies.
cc: @shardulm94 @wmoustafa