Fixed the behavior of the incremental schema change ignore option to properly handle the scenario when columns are dropped

[case-k-git]

resolves #

Problem

Fix the same issue that solved by databricks-dbt https://github.com/databricks/dbt-databricks/pull/580

When processing incrementally, adding new columns is ignored by the ignore setting. However, when a SQL model is modified to remove columns, it fails despite the ignore setting. This is because it attempts to query a column that does not exist in the created temp table. According to the dbt documentation, the job should be designed not to fail when ignored, so it has been corrected.

For example, in this use case, even if we remove column_2 from the SQL model, the query still attempts to include column_2 because it exists in the current table schema. However, since column_2 does not exist in the temporary table, the query fails.

The intended SQL insert statement looks like this:

insert into table `catalog_demo`.`test17060620400077328677_Incremental_strategies`.`append_delta`
      select  column_1, column_2 from `append_delta__dbt_tmp`

Dbt documentation

Similarly, if you remove a column from your incremental model, and execute a dbt run, this column will not be removed from your target table.

So this should not be happen https://docs.getdbt.com/docs/build/incremental-models#default-behavior

Solution

Checklist

• [x] I have read the contributing guide and understand what's expected of me
• [x] I have run this code in development and it appears to resolve the stated issue
• [x] This PR includes tests, or tests are not required/relevant for this PR
• [ ] This PR has no interface changes (e.g. macros, cli, logs, json artifacts, config files, adapter interface, etc) or this PR has already received feedback and approval from Product or DX

[cla-bot[bot]]

Thanks for your pull request, and welcome to our community! We require contributors to sign our Contributor License Agreement and we don't seem to have your signature on file. Check out this article for more information on why we have a CLA.

In order for us to review and merge your code, please submit the Individual Contributor License Agreement form attached above above. If you have questions about the CLA, or if you believe you've received this message in error, please reach out through a comment on this PR.

CLA has not been signed by users: @case-k-git

[cla-bot[bot]]

Thanks for your pull request, and welcome to our community! We require contributors to sign our Contributor License Agreement and we don't seem to have your signature on file. Check out this article for more information on why we have a CLA.

In order for us to review and merge your code, please submit the Individual Contributor License Agreement form attached above above. If you have questions about the CLA, or if you believe you've received this message in error, please reach out through a comment on this PR.

CLA has not been signed by users: @case-k-git