PrefectHQ / prefect

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
https://prefect.io
Apache License 2.0
17.63k stars 1.65k forks source link

prefect-dbt: synchronous invocation of `run_dbt_build` does not execute the task or raise an exception #14532

Open seanpwlms opened 4 months ago

seanpwlms commented 4 months ago

First check

Bug summary

The code example below results in a completed flow run with no task runs, instead of raising an exception about not awaiting coroutines.

run_dbt_build is an async function that calls trigger_dbt_cli_command. Calling trigger_dbt_cli_command synchronously works because it has a sync_compatible decorator. I would think adding the sync_compatible decorator to the command-specific tasks would resolve this issue.

Reproduction

from prefect_dbt.cli.commands import run_dbt_build, trigger_dbt_cli_command
import asyncio
from prefect import flow

@flow
async def async_run_flow():
    await run_dbt_build()

@flow
def sync_run_flow():
    run_dbt_build()

@flow
def dbt_deps():
    trigger_dbt_cli_command(command="dbt deps")

if __name__ == '__main__':
    # dbt_deps() # works
    # asyncio.run(async_run_flow()) # works
    sync_run_flow() # does not run the build task, does not result in an error

Error

Exception ignored in: <coroutine object run_task_async at 0x116438f40>
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/warnings.py", line 537, in _warn_unawaited_coroutine
    warn(msg, category=RuntimeWarning, stacklevel=2, source=coro)
RuntimeWarning: coroutine 'run_task_async' was never awaited

Versions (prefect version output)

Version:             3.0.0rc10
API version:         0.8.4
Python version:      3.11.4
Git commit:          200cf212
Built:               Wed, Jul 3, 2024 1:39 PM
OS/Arch:             darwin/arm64
Profile:             default
Server type:         cloud
Pydantic version:    2.7.4

Additional context

prefect-dbt==0.6.0rc2

Reproduction requires a dbt project in the working directory and a profiles file in ~/.dbt

EmilRex commented 1 month ago

@seanpwlms I can't reproduce this with the latest version, but it does look like nothing actually happens which is what's documented in #15647.

Version:             3.0.10
API version:         0.8.4
Python version:      3.12.7
Git commit:          3aa2d893
Built:               Tue, Oct 15, 2024 1:31 PM
OS/Arch:             darwin/arm64
Profile:             sandbox
Server type:         cloud
Pydantic version:    2.9.2
Integrations:
  prefect-shell:     0.3.0
  prefect-dbt:       0.6.2