Nike-Inc / brickflow

Pythonic Programming Framework to orchestrate jobs in Databricks Workflow
https://engineering.nike.com/brickflow/
Apache License 2.0
187 stars 44 forks source link

[FEATURE] Add support for inter-workflow dependancy [parent-child workflows] #64

Open pariksheet opened 11 months ago

pariksheet commented 11 months ago

Databricks workflow now natively supports parent workflows to have child workflows.

Similar to Airflow - Dags and sub-dags.

refer - https://www.databricks.com/blog/modular-orchestration-databricks-workflows

Cloud Information

Describe the solution you'd like User should be able to create workflow task like Notebook, Python, Shell task.

Describe alternatives you've considered Currently user needs to use WorkflowDependancySensor which is not the effective way.

Additional context Add any other context or screenshots about the feature request here.