snowflakedb / spark-snowflake

Snowflake Data Source for Apache Spark.
http://www.snowflake.net
Apache License 2.0
220 stars 99 forks source link

Overwrite with truncation does not work with column_map #551

Closed imrimt closed 8 months ago

imrimt commented 8 months ago

We have a use case where we want the following:

However, it looks like column_map is not supported in overwrite mode.

How should we go about this use case? It feels like an oversight as truncate_table: on is essentially the same as SaveMode.Append with the additional step of clearing out the data.

We are currently using 2.11.3-spark_3.1 connector.

imrimt commented 8 months ago

In fact, as I typed out the description, I figured out my own workaround:

Will close this issue but hopefully it'll be helpful for others who might have similar use cases.