astronomer / astronomer-cosmos

Run your dbt Core projects as Apache Airflow DAGs and Task Groups with a few lines of code
https://astronomer.github.io/astronomer-cosmos/
Apache License 2.0
597 stars 152 forks source link

Implement `DbtBuild<EXECUTION_MODE>Operator` in Astronomer Cosmos #720

Closed sanromeo closed 7 months ago

sanromeo commented 9 months ago

Hello, I want to suggest the implementation of a new operator: DbtBuild<EXECUTION_MODE>Operator. This operator would be a valuable addition for users looking to streamline their data build processes using dbt within the Astronomer Cosmos.

Proposed Feature: The DbtBuild<EXECUTION_MODE>Operator would unify the functionality of existing operators such as DbtRun<EXECUTION_MODE>Operator, DbtTest<EXECUTION_MODE>Operator, and others. This unified operator would handle running models, testing, seeding seeds, and taking snapshots, all in a single task. (https://docs.getdbt.com/reference/commands/build)

Benefits:

I believe this feature would greatly enhance the utility and user experience of Astronomer Cosmos, particularly for those integrating dbt build command into their workflows.

Looking forward to your thoughts on this suggestion.

dosubot[bot] commented 9 months ago

🤖

It seems like you've got this under control, if you want help or have specific questions, let me know what I can do for you!

tatiana commented 9 months ago

We'd love to see this feature implemented, @sanromeo! Please, let us know if you or your team would like to make this contribution.

dylanharper-qz commented 8 months ago

@tatiana I started work on this here: https://github.com/astronomer/astronomer-cosmos/pull/780

But then I noticed this work and wonder if I should wait for it to be finished: https://github.com/astronomer/astronomer-cosmos/pull/774

At Quizlet, we're excited about having a Build operator because it will let us completely migrate to Cosmos. One of our daily dbt runs uses third party packages with thousands of models and tests. Without this sort of operator we weren't sure how to handle it with Cosmos.

tatiana commented 8 months ago

Hi @dylanharper-qz thanks a lot for working on this!

Good news: we merged #774 yesterday.

At Quizlet, we're excited about having a Build operator because it will let us completely migrate to Cosmos. One of our daily dbt runs uses third party packages with thousands of models and tests. Without this sort of operator we weren't sure how to handle it with Cosmos.

This is certainly a challenge more people face. There's a lot of value in the work you started!