TobikoData / sqlmesh

Efficient data transformation and modeling framework that is backwards compatible with dbt.
https://sqlmesh.com
Apache License 2.0
1.47k stars 122 forks source link

Specify models #2811

Open zoltanmihaly opened 1 week ago

zoltanmihaly commented 1 week ago

I've tried to use SQLMesh on a dbt project but got the following error when launching the UI:

image

It'd be great if the error logs would specify the exact models that are causing the circular references the same way dbt does, for example: RuntimeError: Found a cycle: model.dbt_cyclical_project.model_a --> model.dbt_cyclical_project.model_b

I didn't get the circular references error when running the same models with dbt instead of SQLMesh, so it'd be great for me to understand what models are causing an error in SQLMesh but not in dbt

georgesittas commented 6 days ago

Hey @zoltanmihaly,

This error message should generally contain the models that produced the cycle, or at least an estimate - see https://github.com/TobikoData/sqlmesh/blob/main/sqlmesh/utils/dag.py#L134-L138. Can you check the stack trace / logs to see if there's additional info in there?

It would also be helpful if you could share a reproducible example.

izeigerman commented 3 days ago

@zoltanmihaly most likely one of the models has a dbt test (audit in SQLMesh terms) which references a model downstream.

Unlike dbt, SQLMesh audits are part of the model evaluation and not a separate step. That's why a cyclic reference in an audit leads to a failure in SQLMesh but not dbt.