search

astronomer / astronomer-cosmos

Run your dbt Core projects as Apache Airflow DAGs and Task Groups with a few lines of code
https://astronomer.github.io/astronomer-cosmos/
Apache License 2.0
513 stars 130 forks source link

Add support for `dbt Cloud` as Execution Mode #888

Open jaklan opened 3 months ago

jaklan commented 3 months ago

Currently astronomer-cosmos supports following execution modes:

Execution Modes

Cosmos can run dbt commands using five different approaches, called execution modes:

  1. local: Run dbt commands using a local dbt installation (default)

  2. virtualenv: Run dbt commands from Python virtual environments managed by Cosmos

  3. docker: Run dbt commands from Docker containers managed by Cosmos (requires a pre-existing Docker image)

  4. kubernetes: Run dbt commands from Kubernetes Pods managed by Cosmos (requires a pre-existing Docker image)

  5. azure_container_instance: Run dbt commands from Azure Container Instances managed by Cosmos (requires a pre-existing Docker image)

The choice of the execution mode can vary based on each user’s needs and concerns. For more details, check each execution mode described below.

It would be great to see dbt Cloud in that list as well, so we can automatically create and trigger remote jobs using dbt Cloud API under the hood.

dosubot[bot] commented 3 months ago

That sounds like a great idea! Adding support for dbt Cloud as an Execution Mode would definitely enhance the functionality of the astronomer-cosmos project. If you're interested in contributing this feature, feel free to create a pull request with your proposed changes. Your contribution would be much appreciated!

#### About Dosu This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant. If you want to continue the conversation, start your reply with @dosu-bot.
tatiana commented 3 months ago

@jaklan, this would be a nice feature, especially when leveraging deferrable operators. Would you like to contribute it?

jaklan commented 3 months ago

We don't have resources for that, I would propose to reach to guys responsible for dbt Cloud Airflow provider: https://github.com/apache/airflow/tree/main/airflow/providers/dbt/cloud

tatiana commented 1 month ago

@jaklan I agree this is a relevant feature, but we also have limited bandwidth. If someone from the community is keen on implementing this, we can support them.