kestra-io / plugin-dbt

Apache License 2.0
8 stars 4 forks source link

dbt outputs are not captured when using other task runners than Docker or Process #113

Closed anna-geller closed 2 months ago

anna-geller commented 4 months ago

You can see that when using KubernetesTaskRunner the outputs are not captured and therefore the run_results.json is also not parsed:

image

(no models rendered)

image

When you instead use DockerTaskRunner, you can see properly captured outputs and models:

image

image

Reproducer:

id: dbt_kubernetes
namespace: dwh
description: Download https://github.com/kestra-io/dbt-example as a zipfile and import to the Namespace Files editor. 
tasks:
  - id: dbt_build
    type: io.kestra.plugin.dbt.cli.DbtCLI
    namespaceFiles:
      enabled: true
    containerImage: ghcr.io/kestra-io/dbt-duckdb:latest
    commands:
      - dbt build --profiles-dir {{workingDir}}/dbt/ --project-dir {{workingDir}}/dbt/

taskDefaults:
  - type: io.kestra.plugin.dbt.cli
    values:
      taskRunner:
        type: io.kestra.plugin.kubernetes.runner.KubernetesTaskRunner
        config:
          # masterUrl: https://docker-for-desktop:6443 # when running kestra in docker
          masterUrl: https://127.0.0.1:6443
          clientKey: xxx==
          clientCert: xxx==
          caCert: xxx
          username: docker-desktop
          namespace: default