InfuseAI / piperider

Code review for data in dbt
https://www.piperider.io/
Apache License 2.0
478 stars 23 forks source link

Enable to parse the `threads` property in `profiles.yml` #907

Closed yu-iskw closed 10 months ago

yu-iskw commented 10 months ago

Is your feature request related to a problem? Please describe.

We can configure dbt configurations with environment variables as DBT_DEBUG. Meanwhile, dbt doesn't support such features to vary properties in profiles.yml.

We implement macros to vary properties profiles.yml based on environments. When I tried to use the profiles.yml including the threads property with a macro reference, I got the subsequent error message.

Error message

$ piperider run  --select "tag:piperider"
────────────────────────────────────────────────────────────────── Validating ──────────────────────────────────────────────────────────────────
everything is OK.
Error: '>' not supported between instances of 'LiteralScalarString' and 'int'

Example of the threads property with a macro

The custom environment variable called DBT_THREADS enables us to vary the number of threads in a dbt execution. Though I also tried to use the int Jinja2 function, I got the same error.

https://docs.getdbt.com/reference/dbt-jinja-functions/env_var

jaffle_shop:
  target: dev
  outputs:
    dev:
      type: bigquery
      method: oauth
      project: "your-project"
      dataset: "jaffle_shop"
      priority: "{{ env_var('DBT_BIGQUERY_PRIORITY', 'interactive') }}"
      location: "asia-northeast1"
      threads: |
        {{ env_var("DBT_THREADS", 3) | as_number }}

Describe the solution you'd like I appriciate if piperider could parse the use case with profiles.yml.

Describe alternatives you've considered N/A

Additional context N/A

popcornylu commented 10 months ago

Thanks for your request.

sc-32473 for internal tracking

popcornylu commented 10 months ago

Fixed in v0.38.0