search

AbsaOSS / ABRiS

Avro SerDe for Apache Spark structured APIs.
Apache License 2.0
229 stars 75 forks source link

Support build target for Scala 2.11 #110

Closed shuwnyuantee closed 4 years ago

shuwnyuantee commented 4 years ago

Hi,

I'm using abris 3.1.1 for Scala 2.11 from https://mvnrepository.com/artifact/za.co.absa/abris_2.11/3.1.1

I need to make some changes to abris in order to add JMX metrics, by using https://github.com/groupon/spark-metrics

I cloned your github repo, but then realized the build only supports Scala 2.12. Currently my project is compiled against Scala 2.11, so I need to be able to compile abris for Scala 2.11.

From your git log, this looks like the commit you migrated to Scala 2.12, dated back in 19 Sep 2019

4178e5b264bc38e8af6bb570bf91226142c15539

I tried checkout the previous commit before the above, & do mvn package. It throws this error:

[INFO] /Users/shuwnyuan/ABRiS/src/main/scala: -1: info: compiling [INFO] Compiling 52 source files to /Users/shuwnyuan/ABRiS/target/classes at 1588921738756 [ERROR] error: scala.reflect.internal.MissingRequirementError: object java.lang.Object in compiler mirror not found. [ERROR] at scala.reflect.internal.MissingRequirementError$.signal(MissingRequirementError.scala:17) [ERROR] at scala.reflect.internal.MissingRequirementError$.notFound(MissingRequirementError.scala:18) [ERROR] at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:53) [ERROR] at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:45) [ERROR] at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:45) [ERROR] at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:66) [ERROR] at scala.reflect.internal.Mirrors$RootsBase.getClassByName(Mirrors.scala:102) [ERROR] at scala.reflect.internal.Mirrors$RootsBase.getRequiredClass(Mirrors.scala:105) [ERROR] at scala.reflect.internal.Definitions$DefinitionsClass.ObjectClass$lzycompute(Definitions.scala:257) [ERROR] at scala.reflect.internal.Definitions$DefinitionsClass.ObjectClass(Definitions.scala:257) [ERROR] at scala.reflect.internal.Definitions$DefinitionsClass.init(Definitions.scala:1394) [ERROR] at scala.tools.nsc.Global$Run.(Global.scala:1215) [ERROR] at scala.tools.nsc.Driver.doCompile(Driver.scala:31) [ERROR] at scala.tools.nsc.MainClass.doCompile(Main.scala:23) [ERROR] at scala.tools.nsc.Driver.process(Driver.scala:51) [ERROR] at scala.tools.nsc.Driver.main(Driver.scala:64) [ERROR] at scala.tools.nsc.Main.main(Main.scala) [ERROR] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [ERROR] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [ERROR] at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [ERROR] at java.base/java.lang.reflect.Method.invoke(Method.java:567) [ERROR] at scala_maven_executions.MainHelper.runMain(MainHelper.java:164) [ERROR] at scala_maven_executions.MainWithArgsInFile.main(MainWithArgsInFile.java:26) [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 6.112 s [INFO] Finished at: 2020-05-08T15:08:59+08:00 [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.3.2:compile (scala-compile) on project abris_2.11: wrap: org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1) -> [Help 1]

I'm compiling this on my Mac, here's my Java version:

java -version java version "1.8.0_102" Java(TM) SE Runtime Environment (build 1.8.0_102-b14) Java HotSpot(TM) 64-Bit Server VM (build 25.102-b14, mixed mode)

Any pointer how to work around this?

Can you provide official build target for Scala 2.11 in your github repo?

Thanks.

felipemmelo commented 4 years ago

Hi @shuwnyuantee , have you checked this?

shuwnyuantee commented 4 years ago

@felipemmelo Based on your suggestion, here's the steps I have tried:

<artifactId>abris_2.11</artifactId>
<scala.version>2.11.12</scala.version>
<scala.compat.version>2.11</scala.compat.version>

Then I run into the below error:

[ERROR] error: java.lang.NoClassDefFoundError: javax/tools/ToolProvider
[INFO]  at scala.reflect.io.JavaToolsPlatformArchive.iterator(ZipArchive.scala:301)
[INFO]  at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
[INFO]  at scala.reflect.io.AbstractFile.foreach(AbstractFile.scala:92)
[INFO]  at scala.tools.nsc.util.DirectoryClassPath.traverse(ClassPath.scala:277)
[INFO]  at scala.tools.nsc.util.DirectoryClassPath.x$15$lzycompute(ClassPath.scala:299)
[INFO]  at scala.tools.nsc.util.DirectoryClassPath.x$15(ClassPath.scala:299)
[INFO]  at scala.tools.nsc.util.DirectoryClassPath.packages$lzycompute(ClassPath.scala:299)
[INFO]  at scala.tools.nsc.util.DirectoryClassPath.packages(ClassPath.scala:299)
[INFO]  at scala.tools.nsc.util.DirectoryClassPath.packages(ClassPath.scala:264)
[INFO]  at scala.tools.nsc.util.MergedClassPath$$anonfun$packages$1.apply(ClassPath.scala:358)
[INFO]  at scala.tools.nsc.util.MergedClassPath$$anonfun$packages$1.apply(ClassPath.scala:358)
[INFO]  at scala.collection.Iterator$class.foreach(Iterator.scala:891)
[INFO]  at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
[INFO]  at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
[INFO]  at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
[INFO]  at scala.tools.nsc.util.MergedClassPath.packages$lzycompute(ClassPath.scala:358)
[INFO]  at scala.tools.nsc.util.MergedClassPath.packages(ClassPath.scala:353)
[INFO]  at scala.tools.nsc.symtab.SymbolLoaders$PackageLoader$$anonfun$doComplete$1.apply$mcV$sp(SymbolLoaders.scala:269)
[INFO]  at scala.tools.nsc.symtab.SymbolLoaders$PackageLoader$$anonfun$doComplete$1.apply(SymbolLoaders.scala:260)
[INFO]  at scala.tools.nsc.symtab.SymbolLoaders$PackageLoader$$anonfun$doComplete$1.apply(SymbolLoaders.scala:260)
[INFO]  at scala.reflect.internal.SymbolTable.enteringPhase(SymbolTable.scala:235)
[INFO]  at scala.tools.nsc.symtab.SymbolLoaders$PackageLoader.doComplete(SymbolLoaders.scala:260)
[INFO]  at scala.tools.nsc.symtab.SymbolLoaders$SymbolLoader.complete(SymbolLoaders.scala:211)
[INFO]  at scala.reflect.internal.Symbols$Symbol.info(Symbols.scala:1535)
[INFO]  at scala.reflect.internal.Mirrors$RootsBase.init(Mirrors.scala:256)
[INFO]  at scala.tools.nsc.Global.rootMirror$lzycompute(Global.scala:73)
[INFO]  at scala.tools.nsc.Global.rootMirror(Global.scala:71)
[INFO]  at scala.tools.nsc.Global.rootMirror(Global.scala:39)
[INFO]  at scala.reflect.internal.Definitions$DefinitionsClass.ObjectClass$lzycompute(Definitions.scala:257)
[INFO]  at scala.reflect.internal.Definitions$DefinitionsClass.ObjectClass(Definitions.scala:257)
[INFO]  at scala.reflect.internal.Definitions$DefinitionsClass.init(Definitions.scala:1390)
[INFO]  at scala.tools.nsc.Global$Run.<init>(Global.scala:1242)
[INFO]  at scala.tools.nsc.Driver.doCompile(Driver.scala:31)
[INFO]  at scala.tools.nsc.MainClass.doCompile(Main.scala:23)
[INFO]  at scala.tools.nsc.Driver.process(Driver.scala:51)
[INFO]  at scala.tools.nsc.Driver.main(Driver.scala:64)
[INFO]  at scala.tools.nsc.Main.main(Main.scala)
[INFO]  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[INFO]  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[INFO]  at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[INFO]  at java.base/java.lang.reflect.Method.invoke(Method.java:567)
[INFO]  at scala_maven_executions.MainHelper.runMain(MainHelper.java:164)
[INFO]  at scala_maven_executions.MainWithArgsInFile.main(MainWithArgsInFile.java:26)
[INFO] java.lang.reflect.InvocationTargetException
[INFO]  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[INFO]  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[INFO]  at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[INFO]  at java.base/java.lang.reflect.Method.invoke(Method.java:567)
[INFO]  at scala_maven_executions.MainHelper.runMain(MainHelper.java:164)
[INFO]  at scala_maven_executions.MainWithArgsInFile.main(MainWithArgsInFile.java:26)
[ERROR] Caused by: java.lang.NoClassDefFoundError: javax/tools/ToolProvider
[INFO]  at scala.reflect.io.JavaToolsPlatformArchive.iterator(ZipArchive.scala:301)
[INFO]  at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
[INFO]  at scala.reflect.io.AbstractFile.foreach(AbstractFile.scala:92)
[INFO]  at scala.tools.nsc.util.DirectoryClassPath.traverse(ClassPath.scala:277)
[INFO]  at scala.tools.nsc.util.DirectoryClassPath.x$15$lzycompute(ClassPath.scala:299)
[INFO]  at scala.tools.nsc.util.DirectoryClassPath.x$15(ClassPath.scala:299)
[INFO]  at scala.tools.nsc.util.DirectoryClassPath.packages$lzycompute(ClassPath.scala:299)
[INFO]  at scala.tools.nsc.util.DirectoryClassPath.packages(ClassPath.scala:299)
[INFO]  at scala.tools.nsc.util.DirectoryClassPath.packages(ClassPath.scala:264)
[INFO]  at scala.tools.nsc.util.MergedClassPath$$anonfun$packages$1.apply(ClassPath.scala:358)
[INFO]  at scala.tools.nsc.util.MergedClassPath$$anonfun$packages$1.apply(ClassPath.scala:358)
[INFO]  at scala.collection.Iterator$class.foreach(Iterator.scala:891)
[INFO]  at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
[INFO]  at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
[INFO]  at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
[INFO]  at scala.tools.nsc.util.MergedClassPath.packages$lzycompute(ClassPath.scala:358)
[INFO]  at scala.tools.nsc.util.MergedClassPath.packages(ClassPath.scala:353)
[INFO]  at scala.tools.nsc.symtab.SymbolLoaders$PackageLoader$$anonfun$doComplete$1.apply$mcV$sp(SymbolLoaders.scala:269)
[INFO]  at scala.tools.nsc.symtab.SymbolLoaders$PackageLoader$$anonfun$doComplete$1.apply(SymbolLoaders.scala:260)
[INFO]  at scala.tools.nsc.symtab.SymbolLoaders$PackageLoader$$anonfun$doComplete$1.apply(SymbolLoaders.scala:260)
[INFO]  at scala.reflect.internal.SymbolTable.enteringPhase(SymbolTable.scala:235)
[INFO]  at scala.tools.nsc.symtab.SymbolLoaders$PackageLoader.doComplete(SymbolLoaders.scala:260)
[INFO]  at scala.tools.nsc.symtab.SymbolLoaders$SymbolLoader.complete(SymbolLoaders.scala:211)
[INFO]  at scala.reflect.internal.Symbols$Symbol.info(Symbols.scala:1535)
[INFO]  at scala.reflect.internal.Mirrors$RootsBase.init(Mirrors.scala:256)
[INFO]  at scala.tools.nsc.Global.rootMirror$lzycompute(Global.scala:73)
[INFO]  at scala.tools.nsc.Global.rootMirror(Global.scala:71)
[INFO]  at scala.tools.nsc.Global.rootMirror(Global.scala:39)
[INFO]  at scala.reflect.internal.Definitions$DefinitionsClass.ObjectClass$lzycompute(Definitions.scala:257)
[INFO]  at scala.reflect.internal.Definitions$DefinitionsClass.ObjectClass(Definitions.scala:257)
[INFO]  at scala.reflect.internal.Definitions$DefinitionsClass.init(Definitions.scala:1390)
[INFO]  at scala.tools.nsc.Global$Run.<init>(Global.scala:1242)
[INFO]  at scala.tools.nsc.Driver.doCompile(Driver.scala:31)
[INFO]  at scala.tools.nsc.MainClass.doCompile(Main.scala:23)
[INFO]  at scala.tools.nsc.Driver.process(Driver.scala:51)
[INFO]  at scala.tools.nsc.Driver.main(Driver.scala:64)
[INFO]  at scala.tools.nsc.Main.main(Main.scala)
[INFO]  ... 6 more
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE

Any suggestion how to fix this? Thanks.

p/s: for pom.xml, after changes, git diff looks like this:

shuwnyuan@Tees-MacBook-Pro:~/ABRiS (scala_2_11)$ git diff
diff --git a/pom.xml b/pom.xml
index 120a38a..333d118 100644
--- a/pom.xml
+++ b/pom.xml
@@ -10,7 +10,7 @@
 <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
        <modelVersion>4.0.0</modelVersion>
        <groupId>za.co.absa</groupId>
-       <artifactId>abris_2.12</artifactId>
+       <artifactId>abris_2.11</artifactId>
        <version>3.1.3-SNAPSHOT</version>
        <name>abris</name>
        <description>Provides seamless integration between Avro and Spark Structured APIs.</description>
@@ -60,8 +60,8 @@
                <confluent.version>5.3.1</confluent.version>

                <!--Scala-->
-               <scala.version>2.12.10</scala.version>
-               <scala.compat.version>2.12</scala.compat.version>
+               <scala.version>2.11.12</scala.version>
+               <scala.compat.version>2.11</scala.compat.version>
                <specs.version>4.7.1</specs.version>

                <!--Libs-->
shuwnyuantee commented 4 years ago

@felipemmelo

Is there any particular version of Java or Maven I should use? Any other changes in pom.xml required?

Appreciate if you can share the environment you used to compile & release ABRiS for Scala 2.11.

Thanks.

felipemmelo commented 4 years ago

Hi @shuwnyuantee , what I have is right below.

scala --version
Scala code runner version 2.13.0 -- Copyright 2002-2019, LAMP/EPFL and Lightbend, Inc.
/usr/local/bin/mvn --version
Apache Maven 3.6.0 (97c98ec64a1fdfee7767ce5ffb20918da4f719f3; 2018-10-24T20:41:47+02:00)
Maven home: /usr/local/Cellar/maven/3.6.0/libexec
Java version: 1.8.0_202, vendor: AdoptOpenJdk, runtime: /Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/jre
Default locale: en_GB, platform encoding: UTF-8
OS name: "mac os x", version: "10.14.4", arch: "x86_64", family: "mac"
cerveada commented 4 years ago

As far as I know there is no special requirements on JDK from Abris side, but there is from Scala and it's different for Scala 11 and 12.

https://docs.scala-lang.org/overviews/jdk-compatibility/overview.html

felipemmelo commented 4 years ago

@shuwnyuantee , did it work?

shuwnyuantee commented 4 years ago

@felipemmelo I changed my build environment from Mac to Debian, building it inside Debian Docker container works now. No idea why it failed to compiled on my Mac.

Thanks for your help, appreciate it 👍 💯

shuwnyuantee commented 4 years ago

@cerveada Working for me now, my Java version is openjdk-8-jre. Thanks for help 👍

felipemmelo commented 4 years ago

Hi @shuwnyuantee , happy to read that. Closing this issue then. Cheers.