z3z1ma / dbt-osmosis

Provides automated YAML management, a dbt server, streamlit workbench, and git-integrated dbt model output diff tools
https://z3z1ma.github.io/dbt-osmosis/
Apache License 2.0
422 stars 45 forks source link

[feature] Add progenitor information to meta #83

Closed syou6162 closed 10 months ago

syou6162 commented 10 months ago

Background

If you use dbt-osmosis to maintain column descriptions for a long period of time, you often experience the following.

I want to modify the description of this column to a better description. However, the description of this column seems to have been propagated by dbt-osmosis. Which model column description should I modify?

It would be helpful if this information could be kept statically in yaml instead of at runtime. For this reason, the --add-progenitor-to-meta option has been added in this pull request (Default value is False so as not to change the existing behavior).

Example

For example, suppose you have the following data source

version: 2
sources:
  - name: my_dataset
    database: my-project
    tables:
      - name: order
        columns:
          - name: my_description
            description: "This is description"

dbt-osmosis yaml refactor --add-progenitor-to-meta ... will output the following yaml. You can include information about which model the column description is derived from in meta.osmosis_progenitor. This information is very useful because it tells you which model to modify when modifying column descriptions.

version: 2
models:
  - name: mart_order
    columns:
      - name: my_description
        description: "This is description"
        meta:
          osmosis_progenitor: source.my_project.my_dataset.order

This example is very simple, but when models are linked in multiple layering using dbt, it is difficult to quickly determine which model description is the progenitor. The meta.osmosis_progenitor is very useful because it tells you which model to modify when modifying a column description.

syou6162 commented 10 months ago

@z3z1ma Could you review this pull request?

z3z1ma commented 10 months ago

LGTM