Support the `--profile` and `--vars` options

yu-iskw commented 1 year ago

Motivation

I would like to specify a dbt profile by --profile and dbt variables by --vars.

First, we have multiple profiles and select one based on the running environment. So, our default profile is set with a local development one. We would like to pass a profile to the CLI.

Second, our dbt models requires variables set by the --vars option so that we dynamically switch the Google Cloud project. We have different YAML files corresponding the destination environments respectively. We run a dbt command like dbt build --vars "$(cat vars-production.yml)". So, we would like to pass dbt variables to dbt-osmosis too.

# vars-production.yml
projects:
  project-a: "project-a-production"

# vars-staging.yml
projects:
  project-a: "project-a-staging"

This is a pseudocode to dynamically get the destination Google Cloud project ID

{% set gcp_project = var('projects')['project-a'] %}

{{
  config(
    materialized="table",
    database=gcp_project,
    schema="dwh_ai_inquiry",
    alias="answer",
  )
}}

SELECT ...

z3z1ma commented 1 year ago

I will take care of this, thanks for the issue!

yu-iskw commented 1 year ago

@z3z1ma Sounds good! If there is anything I can help, please let me know.

yu-iskw commented 11 months ago

@z3z1ma can you briefly tell me how we can implement it? I took a look at the code before. I couldn't exactly understand how we modify. If I can understand that, I am interested in contributing. Thanks!

z3z1ma commented 11 months ago

Hello 👋 @yu-iskw

The props need to be added here https://github.com/z3z1ma/dbt-osmosis/blob/26a93bb07b7ba74b1d394346814528bb0cacd7e8/src/dbt_osmosis/vendored/dbt_core_interface/project.py#L205

Then the appropriate kwargs need to go in the DbtProject constructor here https://github.com/z3z1ma/dbt-osmosis/blob/26a93bb07b7ba74b1d394346814528bb0cacd7e8/src/dbt_osmosis/vendored/dbt_core_interface/project.py#L353

The DbtYamlManager inherits from the above. So from there it is straightforward. And you would just need to pass them through click.

yu-iskw commented 11 months ago

@z3z1ma Thank you for the instruction. I will find some time to implement it.

yu-iskw commented 9 months ago

@z3z1ma I created a pull request to support the --vars option to the sub commands of dbt-osmosis yaml. Can you review it? I will create another pull request to support the --profile option later.

https://github.com/z3z1ma/dbt-osmosis/pull/99

yu-iskw commented 9 months ago

@z3z1ma I created the other pull request to support the --profile option in the sub commands of dbt-osmosis yaml. Can you review the pull request?

https://github.com/z3z1ma/dbt-osmosis/pull/101

z3z1ma / dbt-osmosis

Support the `--profile` and `--vars` options #59

Motivation