kestra-io / plugin-dbt

Apache License 2.0
9 stars 4 forks source link

Investigate why dbt test command fails on dbt duckdb #91

Open anna-geller opened 9 months ago

anna-geller commented 9 months ago

Feature description

id: dbt_duckdb
namespace: blueprint

tasks:
  - id: dbt
    type: io.kestra.core.tasks.flows.WorkingDirectory
    tasks:
    - id: cloneRepository
      type: io.kestra.plugin.git.Clone
      url: https://github.com/kestra-io/dbt-example
      branch: main

    - id: dbt-build
      type: io.kestra.plugin.dbt.cli.DbtCLI
      runner: DOCKER
      docker:
        image: ghcr.io/kestra-io/dbt-duckdb:latest
      commands:
        - dbt deps
        - dbt build
        - dbt test # remove this line and everything will work just fine
      profiles: |
        my_dbt_project:
          outputs:
            dev:
              type: duckdb
              path: ":memory:"
              fixed_retries: 1
              threads: 1
              timeout_seconds: 300
          target: dev

this will fail with:

Runtime Error in test unique_my_second_dbt_model_id (models/example/schema.yml)
2024-02-19 12:43:11.523  Catalog Error: Table with name my_second_dbt_model does not exist!
2024-02-19 12:43:11.527  Did you mean "temp.pg_catalog.pg_enum"?
2024-02-19 12:43:11.528  LINE 15: from "memory"."main"."my_second_dbt_model"

Removing dbt test works.

Funny enough, dbt build already runs dbt test under the hood and it works just fine

anna-geller commented 9 months ago

the same happens with earlier versions of dbt-duckdb:

id: dbt_custom_dependencies
namespace: dev
inputs:
  - id: dbt_version
    type: STRING
    defaults: "dbt-duckdb==1.6.0"
tasks:
  - id: git
    type: io.kestra.core.tasks.flows.WorkingDirectory
    tasks:
      - id: clone_repository
        type: io.kestra.plugin.git.Clone
        url: https://github.com/kestra-io/dbt-example
        branch: main
      - id: dbt
        type: io.kestra.plugin.dbt.cli.DbtCLI
        runner: DOCKER
        docker:
          image: python:3.11-slim
        beforeCommands:
          - pip install uv
          - uv venv --quiet
          - . .venv/bin/activate --quiet
          - uv pip install --quiet {{ inputs.dbt_version }}
        commands:
          - dbt deps
          - dbt build
          - dbt test
        profiles: |
          my_dbt_project:
            outputs:
              dev:
                type: duckdb
                path: ":memory:"
                fixed_retries: 1
                threads: 16
                timeout_seconds: 300
            target: dev