search

TobikoData / sqlmesh

Efficient data transformation and modeling framework that is backwards compatible with dbt.
https://sqlmesh.com
Apache License 2.0
1.49k stars 123 forks source link

Application plan in airflow is broken #2780

Open serkef opened 3 weeks ago

serkef commented 3 weeks ago

When I apply some changes in multiple models, the application plan is broken. Some expected tasks are missing and the tasks are not in the correct order. This affects the correctness of the data. I haven't managed to create a small reproduction of this, but I've met it multiple times across 0.101 and 0.104 versions.

Here I changed all my models' types to be incremental by partition instead of date (new model kind supporting bq). The DAG is wrong. Most of the models are missing snapshots and task dependency is wrong.

image image
serkef commented 3 weeks ago

Some relevant chats:

serkef commented 1 week ago

In the chat @izeigerman mentioned they suspected the sensor. I met the same problem again, tried multiple times planning prod getting the same weird behavior repeatedly. I changed the DAG leaving out the external sensor and it worked. I still don't know if this is incidental, but it may help track the issue down. Also, nobody else has this issue apart from me :P so that would be another reason to suspect the sensor.

serkef commented 1 week ago

Actually I can confirm it. While the DAG was running, I changed the dag python file to re-add the sensor and the DAG got messed up (pic 1). Removing again the sensor fixed the dag (pic 2). That's the exact same DAG in the middle of its run.

image image