snowflakedb / spark-snowflake

Snowflake Data Source for Apache Spark.
http://www.snowflake.net
Apache License 2.0
219 stars 99 forks source link

Snowflake Sink should automatically create the table if not exists even when using columnMapping == "name" #550

Open zshao9 opened 8 months ago

zshao9 commented 8 months ago

However, it seems to fail when the jdbcWrapper.resolveTable is called.

In this code: https://github.com/snowflakedb/spark-snowflake/commit/ccfbe44e387c1ccef6f0ee3366b326237dc66547#diff-743d6c0c9fa21fd5da1d11fa72cbf0636eabb4204b3507f7881c0fdcded20cfcR69

      try {
        schema = jdbcWrapper.resolveTable(conn, params.table.get.name, params)
        params.setColumnMap(
          generateColumnMap(
            data.schema,
            schema,
            params.columnMismatchBehavior == "error")
        )

      } finally conn.close()
    }

cc @binglihub @adam-richardson-openai