[CT-2819] default__alter_relation_add_remove_columns macro does not use quoting with case sensitive Snowflake relation

McKnight-42 commented 2 months ago

https://github.com/dbt-labs/dbt-adapters/issues/250 docs dbt-labs/docs.getdbt.com/#

Problem

user raised issue around "While working with DBT incremental config: on_schema_change='append_new_columns' The append new columns flag is not able to capture the correct case-sensitive column name and add it to the incremental table causing the run to fail."

they stated they were using snowflake.

Solution

add a new test to dbt-adapters-tests to check that column quoting case sensitivity is expressed correctly, update all macros in adapters as needed if they do not use the default implementation and test default implementing macros to see if we need to update the dbt-adapters macro as well.

Todo:

[ ] test with rest of adapters (see if dbt-adapter default version of macro needs to be changed or any other adapter override version of macro).
[ ] make sure new tests works in all adapters as needed
[ ] update requirements files either dev-requirements or project.toml files before merger

Checklist

[ ] I have read the contributing guide and understand what's expected of me
[ ] I have run this code in development and it appears to resolve the stated issue
[ ] This PR includes tests, or tests are not required/relevant for this PR
[ ] This PR has no interface changes (e.g. macros, cli, logs, json artifacts, config files, adapter interface, etc) or this PR has already received feedback and approval from Product or DX

github-actions[bot] commented 2 months ago

Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the dbt-spark contributing guide.

McKnight-42 commented 2 months ago

dbt-databricks doesn't appear to have a version of this macro. (checked due to inheritance) how to best proceed with this bit?

McKnight-42 commented 1 month ago

oddly this is showing failure in logs for

[0m12:16:58.377748 [debug] [Thread-65 ]: Spark adapter: Error while running:
/* {"app": "dbt", "dbt_version": "1.9.0a1", "profile_name": "test", "target_name": "default", "node_id": "model.test.dim_jobs"} */

    insert into table test17231373447238588752_test_incremental_on_schema_change.dim_jobs
    select `inserted_at`, `name`, `Job` from dim_jobs__dbt_tmp

error: [UNRESOLVED_COLUMN.WITH_SUGGESTION] A column or function parameter with name `name` cannot be resolved. Did you mean one of the following? [`dim_jobs__dbt_tmp`.`Job`, `dim_jobs__dbt_tmp`.`inserted_at`].

this seems to happen because we quote columns by default in

{% macro get_insert_into_sql(source_relation, target_relation) %}

    {%- set dest_columns = adapter.get_columns_in_relation(target_relation) -%}
    {%- set dest_cols_csv = dest_columns | map(attribute='quoted') | join(', ') -%}
    insert into table {{ target_relation }}
    select {{dest_cols_csv}} from {{ source_relation }}

{% endmacro %}

dbt-labs / dbt-spark