rheem-ecosystem / rheem

Rheem - a cross-platform data processing system
https://rheem-ecosystem.github.io
5 stars 0 forks source link

Tag rheem-api artifacts with Scala version #32

Closed luckyasser closed 7 years ago

luckyasser commented 7 years ago

From @sekruse on February 22, 2017 13:36

The rheem-api module uses Scala. For Rheem to integrate well with any libraries that use Scala (such as Spark!) as well, it is necessary to tag the releases of the module with the Scala version. This measure is important to make sure that there is only one Scala library in the classpath and all libraries are compiled against the same Scala library.

Copied from original issue: daqcri/rheem#44

luckyasser commented 7 years ago

From @sekruse on February 23, 2017 9:56

Actually, this is not limited to rheem-api but any module that depends (transitively) on any Scala project (e.g., rheem-spark and rheem-graphchi). Given this, we should probably add the major Scala version to all modules.

luckyasser commented 7 years ago

From @atroudi on February 23, 2017 10:25

Quick note regarding this same issue, I faced an issue in rheem-api when compiling it with Scala 10, i.e. runnig spark job precompiled with scala 10. I used a workaround by compiling only rheem-api with Scala version 11 while the rest of Rheem modules are compiled with Scala version 10.

luckyasser commented 7 years ago

From @sekruse on February 23, 2017 10:40

In general, you should be able to choose with Scala version to use via

$ mvn install -Dscala.version=2.1x.y -Dscala.compat.version=2.1x

which should already help. And it is in any case advisable to only use a single Scala version for your complete build.

However, if you are going to use the Rheem versions deployed in Maven Central, you don't get to choose, which Scala version to use.

luckyasser commented 7 years ago

From @sekruse on February 23, 2017 11:7

In Flink, this is solved with bash scripts that modify the project descriptors. We could follow the same path. Otherwise, we would probably have to change to a different build tool, sbt in particular.