Big query export from GCP to AWS S3 using Spark

samelamin / spark-bigquery

Google BigQuery support for Spark, Structured Streaming, SQL, and DataFrames with easy Databricks integration.

Apache License 2.0

70 stars 28 forks source link

Big query export from GCP to AWS S3 using Spark #77

Closed ChitturiPadma closed 5 years ago

ChitturiPadma commented 5 years ago

Hi Folks,

I would like to pull data from one of the tables in big query (GCP) to AWS S3. Both are different clusters. Is it possible to pull data from GCP to AWS, using Spark ?

ameyamahajan commented 5 years ago

Yes, this works well. I have done this. Just follow the documentation and use latest jars (com.google.cloud.bigdataoss) 2.0.0-RC2. It would work like charm.

samelamin commented 5 years ago

Yup I also have done this before, assuming spark has access via IAM roles etc then you are good to go