Open craigbloodworth opened 2 months ago
@loicmathieu I guess the next step is to make fileHandlingStrategy: MOUNT
the default setting on the Docker task runner?
I guess the next step is to make fileHandlingStrategy: MOUNT the default setting on the Docker task runner?
No, we should avoid using MOUNT as much as possible, as it has issues with a lot of setups. We need to fix DBT to work with a volume.
@craigbloodworth our typical example works:
id: dbt-build
namespace: company.team
tasks:
- id: working-directory
type: io.kestra.plugin.core.flow.WorkingDirectory
tasks:
- id: cloneRepository
type: io.kestra.plugin.git.Clone
url: https://github.com/kestra-io/dbt-demo
branch: main
- id: dbt-build
type: io.kestra.plugin.dbt.cli.Build
taskRunner:
type: io.kestra.plugin.scripts.runner.docker.Docker
dbtPath: /usr/local/bin/dbt
containerImage: ghcr.io/kestra-io/dbt-duckdb
profiles: |
jaffle_shop:
outputs:
dev:
type: duckdb
path: ':memory:'
extensions:
- parquet
target: dev
Can you test it on your side? If it also works, please give us a reproducer that fail. If it didn't work, please provide logs and machine specification including Docker version.
cc @craigbloodworth do you still see the issue? can you reproduce with the example Loic shared?
Expected Behavior
In order to run dbt against MS SQL Server a custom container image is required and should function just like supported containers in the dbt.cli.Build plugin.
Actual Behaviour
No matter how the plugin is configured it always results in the error:
java.nio.file.NoSuchFileException: /tmp/kestra-wd/tmp/5oYlYsnswKF7R7wM6pIIYn/dbt-env/bin/python3.10
Before the tmp folder is removed I've managed to connect to the container shell & verify that the file and directory does in fact exist
The solution proposed by @loicmathieu in Slack is to add
fileHandlingStrategy: MOUNT
on the Docker task runner. This fixed the issue.Steps To Reproduce
Running docker compose in ubuntu with the task:
Environment Information
Example flow
No response