search

spotify / spark-bigquery

Google BigQuery support for Spark, SQL, and DataFrames
Apache License 2.0
155 stars 52 forks source link

Expose the SQLContext and DataFrame Extensions for PySpark #34

Closed richwhitjr closed 7 years ago

richwhitjr commented 7 years ago

It is doable but difficult to call the implicit classes from PySpark. Instead this pulls out the main logic into two new classes and extends those with the implicit classes.

Tested in both a local Spark standalone for pyspark and spark-shell.

richwhitjr commented 7 years ago

Makes sense, little bit cleaner compared to implicit class that just extends the base class.