search

qubole / kinesis-sql

Kinesis Connector for Structured Streaming
http://www.qubole.com
Apache License 2.0
137 stars 80 forks source link

Support scala 2.12? #80

Closed arnehuang closed 4 years ago

arnehuang commented 4 years ago

Any thoughts on building with scala 2.12 since 2.11 is deprecated in spark now

itsvikramagr commented 4 years ago

@arnehuang

Are you looking for scala 2.12 for spark 3? or do you want to build older spark versions with scala 2.12 as well?

I will soon start a PR for spark 3 and publish the maven artifact.

arnehuang commented 4 years ago

For me personally I was looking for a spark 2.4 build with scala 2.12. Also I'm very new so I don't really know if this is something I can do easily myself?

More broadly, I'm not sure what the adoption rate for scala 2.12 builds is, however I see big providers like amazon EMR building spark with scala 2.12 now (e.g. here) with spark 2.4.4, so I was wondering if you plan to build older spark versions with scala 2.12

chadlagore commented 4 years ago

We also tend to compile this JAR for Spark 2.4/Scala 2.12 and publish it to our internal artifactory for use on EMR. This involves passing the --repositories flag to spark-submit, but I also think you can just pass the --jars flag if you have it locally. There are instructions in the README on how to compile the jar here.

itsvikramagr commented 4 years ago

You can use scala 2.12 with Spark 3 branch/jar #92