databricks / dbt-databricks

A dbt adapter for Databricks.
https://databricks.com
Apache License 2.0
226 stars 119 forks source link

Relation comments are not being updated for incremental models #732

Closed henlue closed 3 months ago

henlue commented 4 months ago

Describe the bug

Relation comments for incremental models are only persisted when the relation is first being created. On subsequent incremental runs an updated model description will not be persisted to the relation.

Steps To Reproduce

1) Create an incremental model with:

model.sql

{{
    config(
        materialized="incremental",
        persist_docs={"relation": true},
    )
}}
select 1 as id

schema.yml

version: 2
models:
  - name: model
    description: "Model Description"

2) dbt run 3) Change the model description 4) dbt run 5) Check the table comment: describe detail model. The table still has the old model description as table comment.

Expected behavior

After the second dbt run the new model description should be persisted as the table comment.

System information

The output of dbt --version:

Core:
  - installed: 1.8.3
  - latest:    1.8.3 - Up to date!

Plugins:
  - databricks: 1.8.3 - Up to date!
  - spark:      1.8.0 - Up to date!

The operating system you're using: Ubuntu 22.04

The output of python --version: Python 3.10.12

Additional context

I'm interested in contributing to this.

henlue commented 4 months ago

I've had a look at the code and it looks like the relation comments are typically persisted as part of the create sql statements, but the insert or merge statements of the incremental runs don't persist relation comments. The persist_docs macro is being called in the incremental materialization, but it only persists column comments currently.

I would propose to add the functionality to persist relation comments in the persist_docs macro if for_relation is set. And change the non-incremental calls to persist_docs to pass for_relation=false (it is true by default).