search

samelamin / spark-bigquery

Google BigQuery support for Spark, Structured Streaming, SQL, and DataFrames with easy Databricks integration.
Apache License 2.0
70 stars 28 forks source link

Method saveAsBigQueryTable([class java.lang.String]) does not exist: Issue persists even after shading guava dependecies #83

Open dduran28 opened 4 years ago

dduran28 commented 4 years ago

I am still experiencing the above issue when trying to run the PySpark example in Databricks.

`bq = bigquery.BigQuerySQLContext(session._wrapped._jsqlContext) bq.setGcpJsonKeyFile(KEY_FILE) bq.setBigQueryProjectId(BQ_PROJECT_ID) bq.setGSProjectId(BQ_PROJECT_ID) bq.setBigQueryGcsBucket(STAGING_BUCKET) bq.setBigQueryDatasetLocation(DATASET_LOCATION)

bqDF = bigquery.BigQueryDataFrame(df._jdf) bqDF.saveAsBigQueryTable( "{0}:{1}.{2}".format(BQ_PROJECT_ID, DATASET_ID, TABLE_NAME)`

gives me the following error

Py4JError: An error occurred while calling o333.saveAsBigQueryTable. Trace: py4j.Py4JException: Method saveAsBigQueryTable([class java.lang.String]) does not exist at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:341) at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:349) at py4j.Gateway.invoke(Gateway.java:286) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:251) at java.lang.Thread.run(Thread.java:748)>

This is my cluster configuration: image

These are my installed JARs and libraries image

dduran28 commented 4 years ago

Bump

samelamin commented 4 years ago

Is this still an issue?

dduran28 commented 4 years ago

Yes it is

samelamin commented 4 years ago

this is dependant on your cluster, can you let me know where you are running it?

spark version, manually create cluster vs EMR/Databricks/Dataproc