[Feature] Support adding concurrency argument to DbtTaskGroup

astronomer / astronomer-cosmos

Run your dbt Core projects as Apache Airflow DAGs and Task Groups with a few lines of code

https://astronomer.github.io/astronomer-cosmos/

Apache License 2.0

774 stars 170 forks source link

[Feature] Support adding concurrency argument to DbtTaskGroup #1323

Open evanvolgas opened 1 week ago

evanvolgas commented 1 week ago

Description

Currently, there doesn't appear to be a way to support the number of Dbt threads for a DbtTaskGroup. It feels like there should be.

Use case/motivation

Complex DAGs often have the ability to run concurrent tasks. The inability of DbtTaskGroup to support threads feels like a severe limitation

Related issues

None, to my knowledge

Are you willing to submit a PR?

[ ] Yes, I am willing to submit a PR!

pankajastro commented 1 week ago

Hey @evanvolgas, Thank you for raising this issue! It looks like a few profiles in the cosmos define it like https://github.com/astronomer/astronomer-cosmos/blob/main/cosmos/profiles/exasol/user_pass.py#L23. Would you like to submit a PR?

evanvolgas commented 1 week ago

I was able to work around the problem with

profile_config = ProfileConfig(
    profile_name="Reporting",
    target_name=SCHEMA_NAME,
    profile_mapping=GoogleCloudOauthProfileMapping(
        conn_id=CONNECTION_ID,
        profile_args={"schema": SCHEMA_NAME, "threads": CONCURRENCY},
    ),
)