search

samelamin / spark-bigquery

Google BigQuery support for Spark, Structured Streaming, SQL, and DataFrames with easy Databricks integration.
Apache License 2.0
70 stars 28 forks source link

Write to bigquery using DataframeWriter #63

Open fbbergamo opened 6 years ago

fbbergamo commented 6 years ago

Is there any problem to implement the writing to Bigquery using DataFrameWriter,

What Do you guys think to implement like the redshift lib?

https://github.com/databricks/spark-redshift/blob/master/src/main/scala/com/databricks/spark/redshift/DefaultSource.scala#L68

I want to use the spark like other formats:

df.write.format("com.samelamin.spark.bigquery").option("writeDisposition", "WRITE_TRUNCATE").save("bigquery-dw:poc_data_pipeline.pricing")

This will help to use this library using the same interface that is already build in my project.

Is there any other way to achieve this?

samelamin commented 6 years ago

Sorry for the late reply @fbbergamo, It is a good idea, there isnt a specific reason we cant do this, as far as I know its only the redshift connector that uses this, so there isnt really a spark standard to writing a connector

I think we did start writing a class but it really didn't go anywhere

You can look here and send a pr

fbbergamo commented 6 years ago

@samelamin thanks, I will work on and let you know. :)