search

z3z1ma / dbt-osmosis

Provides automated YAML management, a dbt server, streamlit workbench, and git-integrated dbt model output diff tools
https://z3z1ma.github.io/dbt-osmosis/
Apache License 2.0
456 stars 46 forks source link

Read column metadata from catalog even if source is specified #125

Closed syou6162 closed 8 months ago

syou6162 commented 8 months ago

Motivation

dbt-osmosis is also useful for filling in the list of source columns.

% dbt-osmosis yaml refactor models/sources/my_source.yml

When there are many sources, dbt-osmosis yaml refactor is often time consuming. In some cases, you may want to use a catalog file to speed up the process. However, dbt-osmosis yaml refactor fails with the following error, even if a catalog file is specified.

% dbt-osmosis yaml refactor --catalog-file target/catalog.json models/sources/my_source.yml
INFO     🌊 Executing dbt-osmosis                                                                                                                        main.py:197

INFO     📈 Searching project stucture for required updates and building action plan                                                                  osmosis.py:615
INFO     ...building project structure mapping in memory                                                                                              osmosis.py:496
INFO     🥇 Project structure approved                                                                                                                osmosis.py:649
INFO     ...building project structure mapping in memory                                                                                              osmosis.py:496
INFO     👉 Processing model: source.my_project.sources.my_source
INFO     🔍 Resolving columns in database                                                                                                             osmosis.py:799
INFO     🦺 Unable to resolve columns in database, falling back to using yaml columns as base column set for model                                    osmosis.py:808
         source.my_project.sources.my_source
INFO     🔬 Looking for actions for source.my_project.sources.my_source                                                              osmosis.py:1019
INFO     ✨ Schema file is up to date for model source.my_project.sources.my_source                                                   osmosis.py:902

What I Changed

Fixed dbt-osmosis yaml refactor to work even for source when retrieving column information from a catalog file.

syou6162 commented 8 months ago

@z3z1ma Could you review this pull request?

z3z1ma commented 8 months ago

LGTM, very nice 💯

syou6162 commented 8 months ago

@z3z1ma Thank you for merging! Could you cut a new release?