search

astronomer / astronomer-cosmos

Run your dbt Core projects as Apache Airflow DAGs and Task Groups with a few lines of code
https://astronomer.github.io/astronomer-cosmos/
Apache License 2.0
612 stars 154 forks source link

Support associating tests to multiple parents, if they have multiple parents #978

Open tatiana opened 4 months ago

tatiana commented 4 months ago

Original discussion in: https://github.com/astronomer/astronomer-cosmos/pull/751#issuecomment-1878578854

Cosmos can improve its implementation towards tests with multiple parents - we should make it consistent with dbt.

kzajaczkowski commented 3 months ago

I was about to raise an issue about this and found this ticket. We're having an issue with dbt relationship testing while using DbtTaskGroup operator.

Let's say we have two models model1 and model2, where model2 references model1. We define relationship test on model2 column column1 referencing model1.column1.

When we run a dag with DbtTaskGroup selecting these 2 models. Tests for model1 will fail, because model2 doesn't exist. Cosmos will run the relationship test twice -- in testing nodes for each of the models.

Changing test_indirect_selection option in ExecutionConfig() doesn't help, because each Airflow task node only selects one dbt model, so test_indirect_selection doesn't really apply here.

When using dbt build command, this will work correctly, because dbt will execute relationship test only once after both model1 and model2 were run.

tatiana commented 2 months ago

@kzajaczkowski do you think you may be able to contribute with this feature? We could aim to release it in the next Cosmos release, if so!

kzajaczkowski commented 2 months ago

@tatiana, thank you, but unfortunately, I wouldn't be able to contribute at this time.