search

z3z1ma / dbt-osmosis

Provides automated YAML management, a dbt server, streamlit workbench, and git-integrated dbt model output diff tools
https://z3z1ma.github.io/dbt-osmosis/
Apache License 2.0
422 stars 45 forks source link

Fix to load dbt vars using `yaml` instead of `json` #107

Closed yu-iskw closed 9 months ago

yu-iskw commented 9 months ago

Overview

When I make the pull request #99 fresh with git merge --ff, the implementation to load the values passed by the --vers option was degraded.

Test

Test without --vars

$ dbt-osmosis yaml refactor  --profiles-dir . --project-dir .
INFO     � Executing dbt-osmosis                                                                                                    main.py:197

INFO     � Searching project stucture for required updates and building action plan                                              osmosis.py:619
INFO     ...building project structure mapping in memory                                                                          osmosis.py:500
INFO     � Project structure approved                                                                                            osmosis.py:653
INFO     ...building project structure mapping in memory                                                                          osmosis.py:500
INFO     � Processing model: model.jaffle_shop.orders                                                                            osmosis.py:792
INFO     � Resolving columns in database                                                                                         osmosis.py:803
INFO     � Processing model: model.jaffle_shop.customers_alias                                                                   osmosis.py:792
INFO     � Resolving columns in database                                                                                         osmosis.py:803
INFO     � Processing model: model.jaffle_shop.stg_customers                                                                     osmosis.py:792
INFO     � Resolving columns in database                                                                                         osmosis.py:803
INFO     � Processing model: model.jaffle_shop.stg_payments                                                                      osmosis.py:792
INFO     � Resolving columns in database                                                                                         osmosis.py:803
INFO     � Processing model: model.jaffle_shop.stg_orders                                                                        osmosis.py:792
INFO     � Resolving columns in database                                                                                         osmosis.py:803
INFO     ✨ Schema file is up to date for model model.jaffle_shop.orders                                                          osmosis.py:906
INFO     � Looking for actions for model.jaffle_shop.stg_customers                                                              osmosis.py:1105
INFO     ✨ Schema file is up to date for model model.jaffle_shop.stg_customers                                                   osmosis.py:906
INFO     � Looking for actions for model.jaffle_shop.stg_payments                                                               osmosis.py:1105
INFO     ✨ Schema file is up to date for model model.jaffle_shop.stg_payments                                                    osmosis.py:906
INFO     � Looking for actions for model.jaffle_shop.stg_orders                                                                 osmosis.py:1105
INFO     ✨ Schema file is up to date for model model.jaffle_shop.stg_orders                                                      osmosis.py:906
INFO     � Looking for actions for model.jaffle_shop.customers_alias                                                            osmosis.py:1105
INFO     ✨ Schema file is up to date for model model.jaffle_shop.customers_alias                                                 osmosis.py:906

Test with --vars of a JSON file

$ cat test-vars.json
{
  "a": "1",
  "b": {
    "b1": 1,
    "b2": [1, 2, 3],
    "b3": ["1", "2", "3"]
  }
}

$ dbt-osmosis yaml refactor --vars "$(cat test-vars.json)" --profiles-dir . --project-dir .
INFO     � Executing dbt-osmosis                                                                                                    main.py:197

INFO     � Searching project stucture for required updates and building action plan                                              osmosis.py:619
INFO     ...building project structure mapping in memory                                                                          osmosis.py:500
INFO     � Project structure approved                                                                                            osmosis.py:653
INFO     ...building project structure mapping in memory                                                                          osmosis.py:500
INFO     � Processing model: model.jaffle_shop.orders                                                                            osmosis.py:792
INFO     � Resolving columns in database                                                                                         osmosis.py:803
INFO     � Processing model: model.jaffle_shop.customers_alias                                                                   osmosis.py:792
INFO     � Resolving columns in database                                                                                         osmosis.py:803
INFO     � Processing model: model.jaffle_shop.stg_customers                                                                     osmosis.py:792
INFO     � Resolving columns in database                                                                                         osmosis.py:803
INFO     � Processing model: model.jaffle_shop.stg_payments                                                                      osmosis.py:792
INFO     � Resolving columns in database                                                                                         osmosis.py:803
INFO     � Processing model: model.jaffle_shop.stg_orders                                                                        osmosis.py:792
INFO     � Resolving columns in database                                                                                         osmosis.py:803
INFO     � Looking for actions for model.jaffle_shop.stg_customers                                                              osmosis.py:1105
INFO     ✨ Schema file is up to date for model model.jaffle_shop.stg_customers                                                   osmosis.py:906
INFO     � Looking for actions for model.jaffle_shop.stg_payments                                                               osmosis.py:1105
INFO     ✨ Schema file is up to date for model model.jaffle_shop.stg_payments                                                    osmosis.py:906
INFO     � Looking for actions for model.jaffle_shop.customers_alias                                                            osmosis.py:1105
INFO     ✨ Schema file is up to date for model model.jaffle_shop.customers_alias                                                 osmosis.py:906
INFO     ✨ Schema file is up to date for model model.jaffle_shop.orders                                                          osmosis.py:906
INFO     � Looking for actions for model.jaffle_shop.stg_orders                                                                 osmosis.py:1105
INFO     ✨ Schema file is up to date for model model.jaffle_shop.stg_orders                                                      osmosis.py:906

Test with --vars of a YAML file

$ cat test-vars.yml
---
a: '1'
b:
  b1: 1
  b2:
  - 1
  - 2
  - 3
  b3:
  - '1'
  - '2'
  - '3'

$ dbt-osmosis yaml refactor --vars "$(cat test-vars.yml)" --profiles-dir . --project-dir .
INFO     � Executing dbt-osmosis                                                                                                    main.py:197

INFO     � Searching project stucture for required updates and building action plan                                              osmosis.py:619
INFO     ...building project structure mapping in memory                                                                          osmosis.py:500
INFO     � Project structure approved                                                                                            osmosis.py:653
INFO     ...building project structure mapping in memory                                                                          osmosis.py:500
INFO     � Processing model: model.jaffle_shop.orders                                                                            osmosis.py:792
INFO     � Resolving columns in database                                                                                         osmosis.py:803
INFO     � Processing model: model.jaffle_shop.customers_alias                                                                   osmosis.py:792
INFO     � Resolving columns in database                                                                                         osmosis.py:803
INFO     � Processing model: model.jaffle_shop.stg_customers                                                                     osmosis.py:792
INFO     � Resolving columns in database                                                                                         osmosis.py:803
INFO     � Processing model: model.jaffle_shop.stg_payments                                                                      osmosis.py:792
INFO     � Resolving columns in database                                                                                         osmosis.py:803
INFO     � Processing model: model.jaffle_shop.stg_orders                                                                        osmosis.py:792
INFO     � Resolving columns in database                                                                                         osmosis.py:803
INFO     � Looking for actions for model.jaffle_shop.customers_alias                                                            osmosis.py:1105
INFO     ✨ Schema file is up to date for model model.jaffle_shop.customers_alias                                                 osmosis.py:906
INFO     � Looking for actions for model.jaffle_shop.stg_customers                                                              osmosis.py:1105
INFO     ✨ Schema file is up to date for model model.jaffle_shop.stg_customers                                                   osmosis.py:906
INFO     � Looking for actions for model.jaffle_shop.stg_orders                                                                 osmosis.py:1105
INFO     ✨ Schema file is up to date for model model.jaffle_shop.stg_orders                                                      osmosis.py:906
INFO     ✨ Schema file is up to date for model model.jaffle_shop.orders                                                          osmosis.py:906
INFO     � Looking for actions for model.jaffle_shop.stg_payments                                                               osmosis.py:1105
INFO     ✨ Schema file is up to date for model model.jaffle_shop.stg_payments                                                    osmosis.py:906
yu-iskw commented 9 months ago

@z3z1ma When I resolve the conflicts in #99, the implementation to load the value of --vars was degraded. So, I fixed the issue. Can you review this?

z3z1ma commented 9 months ago

lgtm

yu-iskw commented 9 months ago

@z3z1ma I look forward to the next release. Thank you for your understanding and cooperation.