search

astronomer / astronomer-cosmos

Run your dbt Core projects as Apache Airflow DAGs and Task Groups with a few lines of code
https://astronomer.github.io/astronomer-cosmos/
Apache License 2.0
782 stars 172 forks source link

Print full log on dbt error #1356

Open fjmacagno opened 21 hours ago

fjmacagno commented 21 hours ago

I am trying to debug cosmos, which is not successfully creating the DAG. Unfortunately, the error message i am getting is useless, just using legacy validation callback. Could this line be changed so that the error prints out stdout as well, so that i have more context to go on?

As an example, this is the full error:

[2024-12-02T15:49:40.651+0000] {graph.py:136} INFO - Running command: `/home/astro/.local/bin/dbt ls --output json --project-dir /tmp/tmpme078_tw --profiles-dir /tmp/cosmos/profile/3cc5d8bec2a8c876092d2b790d2cde171ff387a024cbaa5d615bf88275988ecd --profile derived_eng_dbt --target dev`
Traceback (most recent call last):
  File "/usr/local/airflow/dags/dbt/cosmos_dag.py", line 21, in <module>
    my_cosmos_dag = DbtDag(
                    ^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/cosmos/airflow/dag.py", line 26, in __init__
    DbtToAirflowConverter.__init__(self, *args, **specific_kwargs(**kwargs))
  File "/usr/local/lib/python3.12/site-packages/cosmos/converter.py", line 260, in __init__
    self.dbt_graph.load(method=render_config.load_method, execution_mode=execution_config.execution_mode)
  File "/usr/local/lib/python3.12/site-packages/cosmos/dbt/graph.py", line 402, in load
    self.load_via_dbt_ls()
  File "/usr/local/lib/python3.12/site-packages/cosmos/dbt/graph.py", line 461, in load_via_dbt_ls
    self.load_via_dbt_ls_without_cache()
  File "/usr/local/lib/python3.12/site-packages/cosmos/dbt/graph.py", line 580, in load_via_dbt_ls_without_cache
    nodes = self.run_dbt_ls(dbt_cmd, self.project_path, tmpdir_path, env)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/cosmos/dbt/graph.py", line 442, in run_dbt_ls
    stdout = run_command(ls_command, tmp_dir, env_vars)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/cosmos/dbt/graph.py", line 156, in run_command
    raise CosmosLoadDbtException(f"Unable to run {command} due to the error:\n{details}")
cosmos.dbt.graph.CosmosLoadDbtException: Unable to run ['/home/astro/.local/bin/dbt', 'ls', '--output', 'json', '--project-dir', '/tmp/tmpme078_tw', '--profiles-dir', '/tmp/cosmos/profile/3cc5d8bec2a8c876092d2b790d2cde171ff387a024cbaa5d615bf88275988ecd', '--profile', 'derived_eng_dbt', '--target', 'dev'] due to the error:
using legacy validation callback

If i edit the line referenced above to concat stdout + stderr i get

...

16:17:46  Running with dbt=1.8.7
16:17:47  Registered adapter: databricks=1.8.7
16:17:47  Unable to do partial parsing because config vars, config profile, or config target have changed
16:17:47  Unable to do partial parsing because profile has changed
16:17:48  Encountered an error:
Compilation Error
  Model 'model.derived_eng_dbt.testing_platform_test_results' (models/testing_platform/testing_platform_test_results.sql) depends on a source named 'silver_mysql_eng_testing.test_run' which was not found

which is exactly what i need.

fjmacagno commented 21 hours ago

(also it seems that "create an issue from this line" both bypasses ticket templates and doesnt actually refer to the line. Let me know if you want me to recreate)