Unable to change column types in an incremental model

[ajsquared]

Describe the bug

When trying to backfill an incremental model with --full-refresh, the backfill fails if column types change.

Steps To Reproduce

We created an incremental model and had it running in production for some time. We wanted to change the type of two columns in this model from int to double. In doing so, we wanted to backfill the entire table using the --full-refresh flag. We ran DBT with

dbt run --vars '<redacted>' --models incremental_model_name --full-refresh --project-dir /dbt-dir-redacted --profiles-dir /dbt-dir-redacted --profile databricks --target prod

This failed with

Compilation Error in model incremental_model_name (<redacted>)
This model has an enforced contract that failed.
Please ensure the name, data_type, and number of columns in your contract match the columns in your model's definition.

| column_name  | definition_type | contract_type | mismatch_reason    |
| ------------ | --------------- | ------------- | ------------------ |
| n_ready_gpus | double          | int           | data type mismatch |
| n_total_gpus | double          | int           | data type mismatch |

> in macro assert_columns_equivalent (macros/relations/column/columns_spec_ddl.sql)
> called by macro default__get_assert_columns_equivalent (macros/relations/column/columns_spec_ddl.sql)
> called by macro get_assert_columns_equivalent (macros/relations/column/columns_spec_ddl.sql)
> called by macro databricks__create_table_as (macros/relations/table/create.sql)
> called by macro create_table_as (macros/relations/table/create.sql)
> called by macro statement (macros/etc/statement.sql)
> called by macro materialization_incremental_databricks (macros/materializations/incremental/incremental.sql)
> called by model incremental_model_name (<redacted>)

We do have contract.enforced set to true in the config for this model.

Expected behavior

Since the --full-refresh flag was specified, I would expect it to recreate the table in full, so the existing column types should not be relevant. Based on https://docs.getdbt.com/docs/build/incremental-models#how-do-i-rebuild-an-incremental-model, I'd expect DBT to drop the existing table first, though even if that doesn't happen the current state should not be relevant.

System information

The output of dbt --version:

Core:
  - installed: 1.8.0
  - latest:    1.8.1 - Update available!

  Your version of dbt-core is out of date!
  You can find instructions for upgrading here:
  https://docs.getdbt.com/docs/installation

Plugins:
  - databricks: 1.8.0 - Update available!
  - spark:      1.8.0 - Up to date!

  At least one plugin is out of date or incompatible with dbt-core.
  You can find instructions for upgrading here:
  https://docs.getdbt.com/docs/installation

The operating system you're using: Linux

The output of python --version:

Python 3.11.7

Additional context

This is on a Databricks shared cluster running DBR 14.3

[ajsquared]

If it's relevant this model is using the insert_overwrite strategy

[benc-db]

Per my comment on your other ticket, it's possible this is a function of us incorrectly detecting the previously existing table. Please let me know if 1.8.2rc1 fixes your issue, and if I not, email me a relevant dbt.log file.

[benc-db]

Since the --full-refresh flag was specified, I would expect it to recreate the table in full, so the existing column types should not be relevant. Based on https://docs.getdbt.com/docs/build/incremental-models#how-do-i-rebuild-an-incremental-model, I'd expect DBT to drop the existing table first, though even if that doesn't happen the current state should not be relevant.

Some context on this - since it's a delta table, our default is to use create or replace. I'll need to think on how to handle this.

[ajsquared]

I've done some more testing on this one, and I think there's not actually a bug here. I found we had some incorrect deployments and the model definition was out of sync with the contract defined in the YAML file. I was not able to reproduce this failure mode, and column types did update correctly.

I'll close this one unless I can find another case where this behaves incorrectly.