search

z3z1ma / dbt-osmosis

Provides automated YAML management, a dbt server, streamlit workbench, and git-integrated dbt model output diff tools
https://z3z1ma.github.io/dbt-osmosis/
Apache License 2.0
422 stars 45 forks source link

Add `--vars` options to subcommands of `dbt-osumoais yaml` #99

Closed yu-iskw closed 9 months ago

yu-iskw commented 9 months ago

Partially resolves: https://github.com/z3z1ma/dbt-osmosis/issues/59

Overview

We add the --vars options to the sub commands of dbt-osmosis yaml so that we give custom variables to dbt.

As dbt originally expect the format of the --vars option is YAML. So, it would be great for dbt-osmosis to support not only JSON but also YAML for the compatibility. Fortunately, the yaml package in python enables us to load both of YAML and JSON with the yaml.safe_load function.

$ dbt build --help
...
  --vars YAML                     Supply variables to the project. This
                                  argument overrides variables defined in your
                                  dbt_project.yml file. This argument should
                                  be a YAML string, eg. '{my_variable:
                                  my_value}'

Test

I executed the subsequent commands with the jaffle_shop project locally.

Test without --vars

$ dbt-osmosis yaml refactor --profiles-dir . --project-dir .
INFO     � Executing dbt-osmosis                                                                                       main.py:189

INFO     � Searching project stucture for required updates and building action plan                                 osmosis.py:619
INFO     ...building project structure mapping in memory                                                             osmosis.py:507
INFO     � Project structure approved                                                                               osmosis.py:653
INFO     ...building project structure mapping in memory                                                             osmosis.py:507
INFO     � Processing model: model.jaffle_shop.orders                                                               osmosis.py:867
INFO     � Resolving columns in database                                                                            osmosis.py:878
INFO     � Processing model: model.jaffle_shop.customers_alias                                                      osmosis.py:867
INFO     � Resolving columns in database                                                                            osmosis.py:878
INFO     � Processing model: model.jaffle_shop.stg_customers                                                        osmosis.py:867
INFO     � Resolving columns in database                                                                            osmosis.py:878
INFO     � Processing model: model.jaffle_shop.stg_payments                                                         osmosis.py:867
INFO     � Resolving columns in database                                                                            osmosis.py:878
INFO     � Processing model: model.jaffle_shop.stg_orders                                                           osmosis.py:867
INFO     � Resolving columns in database                                                                            osmosis.py:878
INFO     � Looking for actions for model.jaffle_shop.customers_alias                                               osmosis.py:1150
INFO     ✨ Schema file is up to date for model model.jaffle_shop.customers_alias                                    osmosis.py:971
INFO     � Looking for actions for model.jaffle_shop.stg_customers                                                 osmosis.py:1150
INFO     ✨ Schema file is up to date for model model.jaffle_shop.stg_customers                                      osmosis.py:971
INFO     ✨ Schema file is up to date for model model.jaffle_shop.orders                                             osmosis.py:971
INFO     � Looking for actions for model.jaffle_shop.stg_orders                                                    osmosis.py:1150
INFO     ✨ Schema file is up to date for model model.jaffle_shop.stg_orders                                         osmosis.py:971
INFO     � Looking for actions for model.jaffle_shop.stg_payments                                                  osmosis.py:1150
INFO     ✨ Schema file is up to date for model model.jaffle_shop.stg_payments                                       osmosis.py:971

Test with --vars of a JSON file

$ cat test-vars.json
{
  "a": "1",
  "b": {
    "b1": 1,
    "b2": [1, 2, 3],
    "b3": ["1", "2", "3"]
  }
}

$ dbt-osmosis yaml refactor --vars "$(cat test-vars.json)" --profiles-dir . --project-dir .
INFO     � Executing dbt-osmosis                                                                                       main.py:189

INFO     � Searching project stucture for required updates and building action plan                                 osmosis.py:619
INFO     ...building project structure mapping in memory                                                             osmosis.py:507
INFO     � Project structure approved                                                                               osmosis.py:653
INFO     ...building project structure mapping in memory                                                             osmosis.py:507
INFO     � Processing model: model.jaffle_shop.orders                                                               osmosis.py:867
INFO     � Resolving columns in database                                                                            osmosis.py:878
INFO     � Processing model: model.jaffle_shop.customers_alias                                                      osmosis.py:867
INFO     � Resolving columns in database                                                                            osmosis.py:878
INFO     � Processing model: model.jaffle_shop.stg_customers                                                        osmosis.py:867
INFO     � Resolving columns in database                                                                            osmosis.py:878
INFO     � Processing model: model.jaffle_shop.stg_payments                                                         osmosis.py:867
INFO     � Resolving columns in database                                                                            osmosis.py:878
INFO     � Processing model: model.jaffle_shop.stg_orders                                                           osmosis.py:867
INFO     � Resolving columns in database                                                                            osmosis.py:878
INFO     � Looking for actions for model.jaffle_shop.stg_customers                                                 osmosis.py:1150
INFO     ✨ Schema file is up to date for model model.jaffle_shop.stg_customers                                      osmosis.py:971
INFO     � Looking for actions for model.jaffle_shop.stg_orders                                                    osmosis.py:1150
INFO     ✨ Schema file is up to date for model model.jaffle_shop.stg_orders                                         osmosis.py:971
INFO     � Looking for actions for model.jaffle_shop.stg_payments                                                  osmosis.py:1150
INFO     ✨ Schema file is up to date for model model.jaffle_shop.stg_payments                                       osmosis.py:971
INFO     � Looking for actions for model.jaffle_shop.customers_alias                                               osmosis.py:1150
INFO     ✨ Schema file is up to date for model model.jaffle_shop.customers_alias                                    osmosis.py:971
INFO     ✨ Schema file is up to date for model model.jaffle_shop.orders                                             osmosis.py:971

Test with --vars of a YAML file

$ cat test-vars.yml
---
a: '1'
b:
  b1: 1
  b2:
  - 1
  - 2
  - 3
  b3:
  - '1'
  - '2'
  - '3'

$ dbt-osmosis yaml refactor --vars "$(cat test-vars.yml)" --profiles-dir . --project-dir .
INFO     � Executing dbt-osmosis                                                                                       main.py:189

INFO     � Searching project stucture for required updates and building action plan                                 osmosis.py:619
INFO     ...building project structure mapping in memory                                                             osmosis.py:507
INFO     � Project structure approved                                                                               osmosis.py:653
INFO     ...building project structure mapping in memory                                                             osmosis.py:507
INFO     � Processing model: model.jaffle_shop.orders                                                               osmosis.py:867
INFO     � Resolving columns in database                                                                            osmosis.py:878
INFO     � Processing model: model.jaffle_shop.customers_alias                                                      osmosis.py:867
INFO     � Resolving columns in database                                                                            osmosis.py:878
INFO     � Processing model: model.jaffle_shop.stg_customers                                                        osmosis.py:867
INFO     � Resolving columns in database                                                                            osmosis.py:878
INFO     � Processing model: model.jaffle_shop.stg_payments                                                         osmosis.py:867
INFO     � Resolving columns in database                                                                            osmosis.py:878
INFO     � Processing model: model.jaffle_shop.stg_orders                                                           osmosis.py:867
INFO     � Resolving columns in database                                                                            osmosis.py:878
INFO     � Looking for actions for model.jaffle_shop.stg_customers                                                 osmosis.py:1150
INFO     ✨ Schema file is up to date for model model.jaffle_shop.stg_customers                                      osmosis.py:971
INFO     � Looking for actions for model.jaffle_shop.customers_alias                                               osmosis.py:1150
INFO     ✨ Schema file is up to date for model model.jaffle_shop.customers_alias                                    osmosis.py:971
INFO     ✨ Schema file is up to date for model model.jaffle_shop.orders                                             osmosis.py:971
INFO     � Looking for actions for model.jaffle_shop.stg_payments                                                  osmosis.py:1150
INFO     ✨ Schema file is up to date for model model.jaffle_shop.stg_payments                                       osmosis.py:971
INFO     � Looking for actions for model.jaffle_shop.stg_orders                                                    osmosis.py:1150
INFO     ✨ Schema file is up to date for model model.jaffle_shop.stg_orders                                         osmosis.py:971
z3z1ma commented 9 months ago

Thanks for the contribution. Looks perfect!