Unable to overwrite partition

GoogleCloudDataproc / spark-bigquery-connector

BigQuery data source for Apache Spark: Read data from BigQuery into DataFrames, write DataFrames into BigQuery tables.

Apache License 2.0

378 stars 198 forks source link

Closed DevalNaik closed 9 months ago

DevalNaik commented 9 months ago

I want to overwrite existing partition; replace all data for that partition and retain other partition data.

I am using indirect method to load data. partitionField and datePartition is not working.

df.write.format("bigquery")
         .option("datePartition", "20220331")
         .mode("overwrite")
         .save("table")

davidrabinowitz commented 9 months ago

There is no partitionField in the provided sample, can you please check?

isha97 commented 9 months ago

Hi @DevalNaik Can you please share the full stack trace of the error? And the schema of the table that you are writing to?

isha97 commented 9 months ago

Closing this issue now, @DevalNaik please create a new issue with all the details if you are still facing this.