search

samelamin / spark-bigquery

Google BigQuery support for Spark, Structured Streaming, SQL, and DataFrames with easy Databricks integration.
Apache License 2.0
70 stars 28 forks source link

Conflict occurred creating export directory already exists #84

Open ashay10001 opened 4 years ago

ashay10001 commented 4 years ago

when i write two bigquery dataframes in for loop using function saveAsBigQueryTable(projectid+schemaname+tablename) it gives the error when writing the table in bigquery where it gives error Conflict occurred creating export directory. Path gs://mybucket_name/hadoop/tmp/spark-bigquery/spark-bigquery-1595825912980=521003067 already exists

plz reply with solution

samelamin commented 3 years ago

hmmm that sounds like a race condition issue? can you validate if that is the case? try sleeping between writing frames for a certain amount of time

Id like to know more if its due to a race condition or perhaps the way you are saving the table