kestra-io / plugin-dbt

Apache License 2.0
8 stars 4 forks source link

dbt CLI doesn't render results when specifying a custom target from CLI #127

Closed anna-geller closed 1 month ago

anna-geller commented 2 months ago

The target provided via CLI arguments needs to take precedence over the target specified in the profiles property or profiles.yml file. Currently, that works for execution, but when overwriting the target from CLI, the dbt DAG is not parsed at runtime.

Reproducer:

id: dwh_and_analytics
namespace: company.team
description: |
  ## Data Platform
  Clone a [Git repository](https://github.com/kestra-io/dbt-example) and build dbt models

tasks:
  - id: dbt
    type: io.kestra.plugin.core.flow.WorkingDirectory
    tasks:
    - id: clone_repository
      type: io.kestra.plugin.git.Clone
      url: https://github.com/kestra-io/dbt-example
      branch: master

    - id: dbt_build
      type: io.kestra.plugin.dbt.cli.DbtCLI
      taskRunner:
        type: io.kestra.plugin.scripts.runner.docker.Docker
      containerImage: ghcr.io/kestra-io/dbt-duckdb:latest
      commands:
        - dbt deps --project-dir dbt --target prod
        - dbt build --project-dir dbt --target prod
      profiles: |
        my_dbt_project:
          outputs:
            dev:
              type: duckdb
              path: dbt.duckdb
              extensions: 
                - parquet
              fixed_retries: 1
              threads: 16
              timeout_seconds: 300
            prod:
              type: duckdb
              path: dbt2.duckdb
              extensions: 
                - parquet
              fixed_retries: 1
              threads: 16
              timeout_seconds: 300
          target: dev

no dbt models/tests are parsed when overwriting the target via CLI image

loicmathieu commented 1 month ago

It has nothing to do with the target, dynamic task results are not computed because you forget to set the projectDir property. As this is error prone I propose to use this example explaining why you need to set both and emit a warning if --project-dir is in one of the command but the property is not set.