search

spotify / spark-bigquery

Google BigQuery support for Spark, SQL, and DataFrames
Apache License 2.0
155 stars 52 forks source link

Does `spark-bigquery` work in Spark 2.0? #20

Closed yu-iskw closed 7 years ago

yu-iskw commented 7 years ago

Thank you for the great package.

When I just tried to execute saveAsBigQueryTable in Spark Shell and Apache Zeppelin, I got the below error. I guess BigQueryDataFrame doesn't work well. Does the package work in Spark 2.0?

Otherwise, what do I need to do anything like configuration about the spark-bigquery package?

> df.saveAsBigQueryTable(destinationName, WriteDisposition.WRITE_TRUNCATE)
com.spotify.spark.bigquery.package$.BigQueryDataFrame(Lorg/apache/spark/sql/Dataset;)Lcom/spotify/spark/bigquery/package$BigQueryDataFrame;
  ... 56 elided
nevillelyh commented 7 years ago

Have you tried 0.2.0-SNAPSHOT? We haven't released it but will do soon if it works for folks.

yu-iskw commented 7 years ago

Not yes. Can you tell me the link to 0.2.0-SNAPSHOT? I couldn't file any branch and tag for 0.2.0-SNAPSHOT in the repository.

I look forward to releasing the next version!

nevillelyh commented 7 years ago

It's a snapshot, i.e. not released/published. You'll have to build it yourself for now.

yu-iskw commented 7 years ago

I understand. Thank you for telling me know.