search

astronomer / astronomer-cosmos

Run your dbt Core projects as Apache Airflow DAGs and Task Groups with a few lines of code
https://astronomer.github.io/astronomer-cosmos/
Apache License 2.0
772 stars 170 forks source link

Clone DBT Project #318

Closed KarthikRajashekaran closed 6 months ago

KarthikRajashekaran commented 1 year ago

We don’t have DBT Project inside the Airflow Dag , Both Airflow Repo and DBT Repo are separate

There should a feature where it can clone the DBT Project into the Airflow Directory and Run the DBT Commands

Currently in my work with Airflow running EKS , I have written Custom DBTBashOperator which extends BashOperator , where I clone the DBT Project first and then run the DBT Commands

dweaver33 commented 1 year ago

We have this same issue. Would be nice to be able to provide the location of the dbt repo on disk as well. The use case being that we already clone the repo via submodule in the Airflow repo.

dosubot[bot] commented 1 year ago

Hi, @KarthikRajashekaran! I'm Dosu, and I'm helping the Cosmos team manage their backlog. I wanted to let you know that we are marking this issue as stale.

From what I understand, the issue is about the lack of a feature to clone a DBT project into the Airflow directory and run DBT commands. You have implemented a custom DBTBashOperator to handle this, but you suggest that a built-in feature would be more convenient. Another user, dweaver33, has also expressed the same issue and suggested being able to provide the location of the dbt repo on disk.

Before we close this issue, we wanted to check if it is still relevant to the latest version of the Cosmos repository. If it is, please let us know by commenting on the issue. Otherwise, feel free to close the issue yourself or it will be automatically closed in 7 days.

Thank you for your understanding and contributions to the Cosmos repository!

dweaver33 commented 1 year ago

I would say this is still relevant. As I understand it, you still have to have the dbt project within your Astro repo to use Cosmos. This is one of the things preventing us from moving to Cosmos.

@KarthikRajashekaran, your thoughts?

dosubot[bot] commented 1 year ago

@tatiana Could you please help @KarthikRajashekaran with this issue? They have indicated that the lack of a feature to clone a DBT project into the Airflow directory and run DBT commands is still relevant and preventing them from moving to Cosmos. Thank you!

KarthikRajashekaran commented 1 year ago

I am currently not using cosmos coz of lack of a feature to clone DBT project in Airflow directory

dosubot[bot] commented 9 months ago

Hi, @KarthikRajashekaran

I'm helping the Cosmos team manage their backlog and am marking this issue as stale. The issue you opened addresses the integration of a DBT project into an Airflow DAG, with a suggestion to clone the DBT project into the Airflow directory and run the DBT commands. Currently, a custom DBTBashOperator is being used to handle this. Other users, such as dweaver33, have expressed similar concerns and suggested being able to provide the location of the dbt repo on disk.

Could you please confirm if this issue is still relevant to the latest version of the Cosmos repository? If it is, please let the Cosmos team know by commenting on the issue. Otherwise, feel free to close the issue yourself, or the issue will be automatically closed in 7 days.

Thank you for your understanding and cooperation.

dweaver33 commented 9 months ago

As far as I know, this is still an issue. In order to use Cosmos, it requires that our dbt project lives in the same project as Astro, which is not optimal.

dosubot[bot] commented 6 months ago

Hi, @KarthikRajashekaran

I'm helping the Cosmos team manage their backlog and am marking this issue as stale. The issue you opened addresses the integration of a DBT project into an Airflow DAG, with a suggestion to have a feature that allows cloning the DBT project into the Airflow directory and running the DBT commands. Currently, a custom DBTBashOperator is being used to handle this. Other users, such as dweaver33, have expressed similar concerns and suggested being able to provide the location of the dbt repo on disk.

Could you please confirm if this issue is still relevant to the latest version of the Cosmos repository? If it is, please let the Cosmos team know by commenting on the issue. Otherwise, feel free to close the issue yourself, or the issue will be automatically closed in 7 days.

Thank you for your understanding and cooperation.