search

kestra-io / plugin-dbt

Apache License 2.0
8 stars 4 forks source link

Fix the dbt CLI task to work with volume (without requiring bind-mount) when running with Docker taskRunner #135

Open craigbloodworth opened 2 months ago

craigbloodworth commented 2 months ago

Expected Behavior

In order to run dbt against MS SQL Server a custom container image is required and should function just like supported containers in the dbt.cli.Build plugin.

Actual Behaviour

No matter how the plugin is configured it always results in the error:

java.nio.file.NoSuchFileException: /tmp/kestra-wd/tmp/5oYlYsnswKF7R7wM6pIIYn/dbt-env/bin/python3.10

Before the tmp folder is removed I've managed to connect to the container shell & verify that the file and directory does in fact exist

The solution proposed by @loicmathieu in Slack is to add fileHandlingStrategy: MOUNT on the Docker task runner. This fixed the issue.

Steps To Reproduce

Running docker compose in ubuntu with the task:

Environment Information

Example flow

No response

anna-geller commented 2 months ago

@loicmathieu I guess the next step is to make fileHandlingStrategy: MOUNT the default setting on the Docker task runner?

loicmathieu commented 2 months ago

I guess the next step is to make fileHandlingStrategy: MOUNT the default setting on the Docker task runner?

No, we should avoid using MOUNT as much as possible, as it has issues with a lot of setups. We need to fix DBT to work with a volume.

loicmathieu commented 2 months ago

@craigbloodworth our typical example works:

id: dbt-build
namespace: company.team

tasks:
  - id: working-directory
    type: io.kestra.plugin.core.flow.WorkingDirectory
    tasks:
    - id: cloneRepository
      type: io.kestra.plugin.git.Clone
      url: https://github.com/kestra-io/dbt-demo
      branch: main

    - id: dbt-build
      type: io.kestra.plugin.dbt.cli.Build
      taskRunner:
        type: io.kestra.plugin.scripts.runner.docker.Docker
      dbtPath: /usr/local/bin/dbt
      containerImage: ghcr.io/kestra-io/dbt-duckdb
      profiles: |
        jaffle_shop:
          outputs:
            dev:
              type: duckdb
              path: ':memory:'
              extensions:
                - parquet
          target: dev

Can you test it on your side? If it also works, please give us a reproducer that fail. If it didn't work, please provide logs and machine specification including Docker version.

anna-geller commented 2 weeks ago

cc @craigbloodworth do you still see the issue? can you reproduce with the example Loic shared?