search

astronomer / astronomer-cosmos

Run your dbt Core projects as Apache Airflow DAGs and Task Groups with a few lines of code
https://astronomer.github.io/astronomer-cosmos/
Apache License 2.0
767 stars 170 forks source link

[Feature request] Build operator as a test behavior: `TestBehavior.BUILD` #892

Open dwreeves opened 8 months ago

dwreeves commented 8 months ago

In Cosmos 1.4, the "build" operator was added.

This is not currently being used in the DbtTaskGroup / DbtDag API. But there is one use case that can be supported: tests! You can run tests and a model in a single dbt invocation using dbt build.

Why would someone want this? A big reason is to reduce the time of DAG runs. Splitting out tests into two Airflow tasks-- one for the node and one for the test-- can slow things down for users who are facing huge bottlenecks in spinning up nodes, e.g. they are not using partial parsing.

This is probably not a simple change because it most likely requires changing some internal assumptions regarding how Cosmos parses DAGs.

I'm very open to someone else taking on this issue, too.

dosubot[bot] commented 8 months ago

It seems like you've got this under control, if you want help or have specific questions, let me know what I can do for you!

#### About Dosu This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant. If you want to continue the conversation, start your reply with @dosu-bot.
tatiana commented 6 months ago

Hi @dwreeves , I like your proposal. What do you think we introduced this as TestBehaviour.BUILD? I believe this is a relatively small change.