The projectDir property has no effect — we should fix it to work the same way as --project-dir CLI flag

[anna-geller]

Issue description

The projectDir property doesn't work as expected — it has no effect atm.

Workaround that works:

id: dwh_and_analytics_2
namespace: tutorial
description: Data Warehouse and Analytics

tasks:
  - id: dbt
    type: io.kestra.plugin.core.flow.WorkingDirectory
    tasks:
    - id: clone_repository
      type: io.kestra.plugin.git.Clone
      url: https://github.com/kestra-io/dbt-example
      branch: master

    - id: dbt_build
      type: io.kestra.plugin.dbt.cli.DbtCLI
      taskRunner:
        type: io.kestra.plugin.scripts.runner.docker.Docker
      containerImage: ghcr.io/kestra-io/dbt-duckdb:latest
      # projectDir: dbt
      commands:
        - dbt deps --project-dir dbt
        - dbt build --project-dir dbt
      profiles: |
        my_dbt_project:
          outputs:
            dev:
              type: duckdb
              path: dbt.duckdb
              extensions: 
                - parquet
              fixed_retries: 1
              threads: 16
              timeout_seconds: 300
          target: dev

TODO

• [ ] Make the projectDir property work the same way as when setting the --project-dir via CLI
• [ ] Ensure that this projectDir is set/appended for all commands done via CLI

[anna-geller]

@loicmathieu it seems this didn't fix the issue, reopening.

you can reproduce on develop using:

id: dwh_and_analytics_2
namespace: tutorial
description: Data Warehouse and Analytics

tasks:
  - id: dbt
    type: io.kestra.plugin.core.flow.WorkingDirectory
    tasks:
    - id: clone_repository
      type: io.kestra.plugin.git.Clone
      url: https://github.com/kestra-io/dbt-example
      branch: master

    - id: dbt_build
      type: io.kestra.plugin.dbt.cli.DbtCLI
      taskRunner:
        type: io.kestra.plugin.scripts.runner.docker.Docker
      containerImage: ghcr.io/kestra-io/dbt-duckdb:latest
      projectDir: dbt
      commands:
        - dbt deps # --project-dir dbt
        - dbt build # --project-dir dbt
      profiles: |
        my_dbt_project:
          outputs:
            dev:
              type: duckdb
              path: dbt.duckdb
              extensions: 
                - parquet
              fixed_retries: 1
              threads: 16
              timeout_seconds: 300
          target: dev

this will still fail.

[anna-geller]

too soon, seems it will be fixed after this is done https://github.com/kestra-io/kestra/actions/runs/9711976378

[anna-geller]

it's still not fixed - we should investigate more

[anna-geller]

for now added the workaround as a blueprint https://kestra.io/blueprints/git/233-git-workflow-for-dbt-pull-latest-dbt-changes-from-github-and-build-dbt-models-from-your-dbt-project-directory