search

astronomer / astronomer-cosmos

Run your dbt Core projects as Apache Airflow DAGs and Task Groups with a few lines of code
https://astronomer.github.io/astronomer-cosmos/
Apache License 2.0
767 stars 170 forks source link

[Feature] Task timeout and SLA support #1316

Open t0momi219 opened 1 week ago

t0momi219 commented 1 week ago

Description

In Airflow, both DAGs and tasks can have timeout and SLA times specified. Since dbt models likely have varying expected execution times for each layer, there could be cases where users want to apply SLAs individually to each node.

How about having Cosmos retrieve timeout and SLA times from the node metadata and apply them individually when rendering nodes?

Use case/motivation

No response

Related issues

No response

Are you willing to submit a PR?

pankajkoti commented 1 week ago

hi @t0momi219 . Sounds like a great idea. I am definitely up for the idea of timeouts. When it comes to SLA, I'm hearing that they might take away the concept of SLAs in Airflow 3 and come up with something better in later releases of AF 3. So, if you're willing, definitely would encourage the contribution for supporting timeout from node metadata and applying it to AF tasks.