Nike-Inc / brickflow

Pythonic Programming Framework to orchestrate jobs in Databricks Workflow
https://engineering.nike.com/brickflow/
Apache License 2.0
183 stars 36 forks source link

[BUG] WorkflowTaskDependencySensor fails if the task is in PENDING / NOT STARTED state #122

Closed maxim-mityutko closed 2 months ago

maxim-mityutko commented 3 months ago

Describe the bug The value in result_state which indicates the outcome of the task run is only available when the task actually finished. Otherwise the API does not return the key at all. That means that if the task state is poked before that, the sensor crashes.

To Reproduce Steps to reproduce the behavior:

Expected behavior The sensor keeps poking till the expected state or timeout is reached.

Cloud Information

Desktop (please complete the following information):

Additional context Add any other context about the problem here.