dbt-labs / dbt-core

dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.
https://getdbt.com
Apache License 2.0
9.93k stars 1.63k forks source link

[CT-3124] [Bug] Compilation Error when altering an index on a materialized view #8655

Open dbeatty10 opened 1 year ago

dbeatty10 commented 1 year ago

Is this a new bug in dbt-core?

Current Behavior

When changing an index definition with dbt-postgres, it raises an error.

Expected Behavior

I was expecting no errors when running dbt, and that the index on the materialized view would be altered.

Steps To Reproduce

Using dbt-core~1.6 with postgres:

Start with this single model in models/my_mv.sql:

{{ config(
    materialized="materialized_view",
    indexes=[
      {'columns': ['column_a'], 'type': 'hash'},
    ],
    on_configuration_change="apply",
)}}

select 1 as column_a, 2 as column_b

Then this will work fine:

dbt run -s my_mv

But if the index is altered with a different column name within models/my_mv.sql like this:

{{ config(
    materialized="materialized_view",
    indexes=[
      {'columns': ['column_b'], 'type': 'hash'},
    ],
    on_configuration_change="apply",
)}}

select 1 as column_a, 2 as column_b

Then this will yield a Compilation Error:

dbt run -s my_mv

Relevant log output

Works fine on the initial run:

(postgres_1.6) $ dbt run -s my_mv
22:39:19  Running with dbt=1.6.0
22:39:19  Registered adapter: postgres=1.6.0
22:39:19  Unable to do partial parsing because of a version mismatch
22:39:20  Found 2 models, 0 sources, 0 exposures, 0 metrics, 349 macros, 0 groups, 0 semantic models
22:39:20  
22:39:20  Concurrency: 5 threads (target='postgres')
22:39:20  
22:39:20  1 of 1 START sql materialized_view model dbt_dbeatty.my_mv ..................... [RUN]
22:39:20  1 of 1 OK created sql materialized_view model dbt_dbeatty.my_mv ................ [REFRESH MATERIALIZED VIEW in 0.12s]
22:39:20  
22:39:20  Finished running 1 materialized_view model in 0 hours 0 minutes and 0.31 seconds (0.31s).
22:39:20  
22:39:20  Completed successfully
22:39:20  
22:39:20  Done. PASS=1 WARN=0 ERROR=0 SKIP=0 TOTAL=1

Breaks when re-running after the index definition is changed:

(postgres_1.6) $ dbt run -s my_mv
22:39:27  Running with dbt=1.6.0
22:39:27  Registered adapter: postgres=1.6.0
22:39:27  Found 2 models, 0 sources, 0 exposures, 0 metrics, 349 macros, 0 groups, 0 semantic models
22:39:27  
22:39:27  Concurrency: 5 threads (target='postgres')
22:39:27  
22:39:27  1 of 1 START sql materialized_view model dbt_dbeatty.my_mv ..................... [RUN]
22:39:27  1 of 1 ERROR creating sql materialized_view model dbt_dbeatty.my_mv ............ [ERROR in 0.09s]
22:39:28  
22:39:28  Finished running 1 materialized_view model in 0 hours 0 minutes and 0.27 seconds (0.27s).
22:39:28  
22:39:28  Completed with 1 error and 0 warnings:
22:39:28  
22:39:28  Compilation Error in model my_mv (models/my_mv.sql)
22:39:28    Could not parse index config: at path []: Undefined is not of type 'object'
22:39:28    
22:39:28    > in macro postgres__get_create_index_sql (macros/adapters.sql)
22:39:28    > called by macro postgres__update_indexes_on_materialized_view (macros/materializations/materialized_view.sql)
22:39:28    > called by macro postgres__get_alter_materialized_view_as_sql (macros/materializations/materialized_view.sql)
22:39:28    > called by macro get_alter_materialized_view_as_sql (macros/materializations/models/materialized_view/alter_materialized_view.sql)
22:39:28    > called by macro materialized_view_get_build_sql (macros/materializations/models/materialized_view/materialized_view.sql)
22:39:28    > called by macro materialization_materialized_view_default (macros/materializations/models/materialized_view/materialized_view.sql)
22:39:28    > called by model my_mv (models/my_mv.sql)
22:39:28  
22:39:28  Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1

Environment

- OS: macOS
- Python: 3.8.13
- dbt: 1.6.0

Which database adapter are you using with dbt?

postgres

Additional Context

The presumed problem/solution is described in this comment.

dbeatty10 commented 1 year ago

I believe the problem is here:

https://github.com/dbt-labs/dbt-core/blob/3f5ebe81b9039b33a57b9aa8b1f5cb1edcd8790f/plugins/postgres/dbt/adapters/postgres/relation_configs/index.py#L103

I think this will fix it:

            "type": self.method,