PrefectHQ / prefect

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
https://prefect.io
Apache License 2.0
17.54k stars 1.65k forks source link

[prefect-dbt] `--log-level warn` not working anymore #16013

Open etiennecallies opened 1 week ago

etiennecallies commented 1 week ago

Bug summary

Hi! We upgraded from prefect-dbt[postgres]==0.3.1 to prefect-dbt[postgres]==0.6.3

And this command used to print only warning and error logs in prefect logs :

            return await trigger_dbt_cli_command(
                command="dbt --log-level warn build",
                overwrite_profiles=True,
                dbt_cli_profile=dbt_cli_profile,
                project_dir=project_dir_str,
            )

But now with 0.6.3, it just outputs every logs. It seems that the --log-level warn is ignored (cf dbt documentation), I tried also --quiet and adding global_configs=GlobalConfigs(extras={"log_level": "warn"}), in my DbtCliProfile in vain.

What would be the right way to reduce DBT logs in prefect logs ?

Version info

Version:             3.1.0
API version:         0.8.4
Python version:      3.12.4
Git commit:          a83ba39b
Built:               Thu, Oct 31, 2024 12:43 PM
OS/Arch:             darwin/arm64
Profile:             default
Server type:         cloud
Pydantic version:    2.10.0b1
Integrations:
  prefect-dbt:       0.6.3
  prefect-shell:     0.3.0

Additional context

No response

etiennecallies commented 1 week ago

Hi again, I found a workaround using DbtCoreOperation, but this is recoding the wheel. Is it possible to add a log_level argument to trigger_dbt_cli_command ?

            logger.info(f"Running DBT command: '{command}'")
            with DbtCoreOperation(
                commands=[command],
                stream_output=True,
                env={"DBT_LOG_LEVEL": log_level},
                overwrite_profiles=True,
                dbt_cli_profile=dbt_cli_profile,
                project_dir=project_dir,
                working_dir=project_dir,
            ) as dbt_run:
                dbt_process = await dbt_run.trigger()
                await dbt_process.wait_for_completion()
                return_code = dbt_process.return_code
                if return_code != 0:
                    raise Exception(f"DBT command '{command}' failed")