search

dbt-labs / dbt-spark

dbt-spark contains all of the code enabling dbt to work with Apache Spark and Databricks
https://getdbt.com
Apache License 2.0
395 stars 221 forks source link

[ADAP-970] [Feature] Incremental updates should update table description #928

Closed andreas-gjensidige closed 4 months ago

andreas-gjensidige commented 11 months ago

Is this your first time submitting a feature request?

Describe the feature

When running an incremental model the table comment should be updated if the model description has changed.

I think this should be pretty straight forward to implement. dbt-core has a alter_relation_comment-macro which can update the model description. This is then called in the persist_docs-macro. So to implement this in dbt-spark we should only need to create the spark__alter_relation_comment-macro, which can use something like ALTER TABLE table SET TBLPROPERTIES ('comment' = MODEL_DESCRIPTION), and then add a call to the sparkalter_relation_comment-macro in the sparkpersist_docs-macro.

I think ALTER TABLE table SET TBLPROPERTIES ('comment' = MODEL_DESCRIPTION) should work generally in Spark, but I have only been able to test this on Databricks with Unity Catalog.

Describe alternatives you've considered

Running a full-refresh after updating the model-description. But I see no reason why doing that should be needed. That just feels like wasted computational resources.

Manually updating the table description, but that's something that will manually have to be kept in sync, which will be forgotten eventually.

Who will this benefit?

People using incremental models who use the persist_docs feature and has model-descriptions.

Are you interested in contributing this feature?

Yes, but I need to clarify with my employers legal department if I'm allowed to or not...

Anything else?

No response

andreas-gjensidige commented 11 months ago

I think I might have messed up this workflow (https://github.com/dbt-labs/dbt-spark/actions/runs/6690653166) when editing the title... I accidentally removed the Iíssue title before submitting, and when updating it straight after submitting I accidentally removed the "[ADAP 970]" part which was automatically added.

github-actions[bot] commented 5 months ago

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.

github-actions[bot] commented 4 months ago

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest. Just add a comment to notify the maintainers.