search

spotify / spark-bigquery

Google BigQuery support for Spark, SQL, and DataFrames
Apache License 2.0
155 stars 52 forks source link

Added support for Spark 2.4.0 and Scala 2.12. #69

Closed mrksmb closed 5 years ago

mrksmb commented 5 years ago

Adjusted code slightly since Databricks Avro library moved into Spark with small syntax changes.

nevillelyh commented 5 years ago

Just realized that spark-avro is not available for Scala 2.12. How did you get this to work in your build? https://mvnrepository.com/artifact/com.databricks/spark-avro

mrksmb commented 5 years ago

It compiled fine, but I have yet to test the code change. Plan to do that today and will make adjustments if necessary.

markus-braasch-ck commented 5 years ago

Neville, I think Databricks donated the Avro library to Spark. Adding it as a dependency like this seems to work for me: libraryDependencies ++= Seq( "org.apache.spark" %% "spark-avro" % sparkVersion )

https://mvnrepository.com/artifact/org.apache.spark/spark-avro

ogirardot commented 5 years ago

hi, thx for that, was the version published ?

regadas commented 5 years ago

It wasn't yet, I think we will do it next week.