Nike-Inc / brickflow

Pythonic Programming Framework to orchestrate jobs in Databricks Workflow
https://engineering.nike.com/brickflow/
Apache License 2.0
183 stars 36 forks source link

[BUG] Airflow Task dep sensor fails when the dependent task is still running #125

Open randomrama opened 2 months ago

randomrama commented 2 months ago

Describe the bug Airflow Task dep sensor fails when the dependent task is still running

Additional context Logs `/api/v1/dags/s4_fact_dl_sales_order_na_cleansed_harmonized/dagRuns?execution_date_gte=2024-05-25T07:14:07Z [2024-05-26 07:24:27,305] [INFO] [brickflow-plugins-0.11.5rc3+9acb216] {external_tasks.py:get_execution_stats:280} - Latest run for the dag is with execution date of scheduled2024-05-25T07:15:00+00:00 [2024-05-26 07:24:27,306] [INFO] [brickflow-plugins-0.11.5rc3+9acb216] {external_tasks.py:get_execution_stats:281} - Poking s4_fact_dl_sales_order_na_cleansed_harmonized dag for scheduled2024-05-25T07:15:00+00:00 run_id status as latest flag is set to False [2024-05-26 07:24:27,307] [INFO] [brickflow-plugins-0.11.5rc3+9acb216] {external_tasks.py:get_execution_stats:296} - Pinging airflow API /api/v1/dags/s4_fact_dl_sales_order_na_cleansed_harmonized/dagRuns/scheduled__2024-05-25T07:15:00+00:00/taskInstances/success for task status [2024-05-26 07:24:27,933] [INFO] [brickflow-plugins-0.11.5rc3+9acb216] {external_tasks.py:poke:307} - task_status= None [2024-05-26 07:25:27,993] [INFO] [brickflow-plugins-0.11.5rc3+9acb216] {external_tasks.py:poke:303} - executing poke.. 5 [2024-05-26T07:25:27.994+0000] {external_tasks.py:305} INFO - Poking.. 6 round [2024-05-26 07:25:28,368] [INFO] [brickflow-plugins-0.11.5rc3+9acb216] {external_tasks.py:get_execution_stats:259} - URL to poke for dag runs /api/v1/dags/s4_fact_dl_sales_order_na_cleansed_harmonized/dagRuns?execution_date_gte=2024-05-25T07:15:09Z

IndexError: list index out of range `

maxim-mityutko commented 2 months ago

Can you share the full stacktrace for the error?