Nike-Inc / brickflow

Pythonic Programming Framework to orchestrate jobs in Databricks Workflow
https://engineering.nike.com/brickflow/
Apache License 2.0
183 stars 36 forks source link

[FEATURE] Add capability to specify task/workflow status to satisfy `WorkflowTaskDependencySensor` and `WorkflowDependencySensor` #118

Open chaelant opened 3 months ago

chaelant commented 3 months ago

Is your feature request related to a problem? Please describe. Currently, sensors in workflow_dependency_sensor.py are marked successful if the target workflow or task has a result_state of SUCCESS. This does not allow for workflows that depend on streaming workflows to check if their upstream streaming tasks are currently running or use cases where a task is allowed to have been skipped.

Cloud Information

Describe the solution you'd like Update the parameters for the sensors to allow users to specify the success conditions for the dependency sensors, leaving the default condition as SUCCESS as is currently implemented

Describe alternatives you've considered

Additional context