Leverage Databricks API to retrieve the state of the specific task within the workflow run. The code mostly inherits from the WorkflowDependencySensor.
Related Issue
109
Motivation and Context
In many cases it's more benificial to put a dependency on a specific task within the workflow instead the workflow itself. It reduces the wait time of the downstream workflows.
How Has This Been Tested?
Unit tests + workflow in Databricks with a success sensor and a failed sensor
Screenshots (if appropriate):
Types of changes
[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)
Checklist:
[x] My code follows the code style of this project.
[x] My change requires a change to the documentation.
Description
Leverage Databricks API to retrieve the state of the specific task within the workflow run. The code mostly inherits from the
WorkflowDependencySensor.Related Issue
109
Motivation and Context
In many cases it's more benificial to put a dependency on a specific task within the workflow instead the workflow itself. It reduces the wait time of the downstream workflows.
How Has This Been Tested?
Unit tests + workflow in Databricks with a success sensor and a failed sensor
Screenshots (if appropriate):
Types of changes
Checklist: