Nike-Inc / brickflow

Pythonic Programming Framework to orchestrate jobs in Databricks Workflow
https://engineering.nike.com/brickflow/
Apache License 2.0
183 stars 36 forks source link

[FEATURE] Enable task dependency for Brickflow tasks #109

Closed IMC07 closed 3 months ago

IMC07 commented 4 months ago

Is your feature request related to a problem? Please describe. We would like to be able to let workflows be dependent on specific tasks of other Brickflows workflows (just like there is for Airflow a TaskDependencySensor). This makes it more flexible to orchestrate our pipelines in a way that makes sense, whilst not hurting having unnecessary wait time for other pipelines that are only partially dependent on another pipeline.

Cloud Information

Describe the solution you'd like I think it would be nice to have the option of waiting for the completion of a specific task as optional parameter of WorkFlowDependencySensor. This because a task is always tied to a workflow, so it is easy to see to which task is referred when setting the dependency, as well as the fact that the WorkFlowDependencySensor already makes a call to /api/2.1/jobs/runs/get, which also contains the task statuses, and therefore can be used without making extra calls.

Describe alternatives you've considered Ro-organising our workflows, but this is suboptimal.

Additional context