Nike-Inc / brickflow

Pythonic Programming Framework to orchestrate jobs in Databricks Workflow
https://engineering.nike.com/brickflow/
Apache License 2.0
187 stars 44 forks source link

[BUG] WorkflowDependencySensor doesn't fails the workflow run if invalid "dependency_job_id" is given #76

Open anshima03 opened 9 months ago

anshima03 commented 9 months ago

To do a negative testing we tried using a random dependency_job_id in WorkflowDependencySensor function expecting that the workflow will fail but it continued searching for the job_id.

sensor = WorkflowDependencySensor( databricks_host="https://......cloud.databricks.com", databricks_token='' , dependency_job_id='some random job id', poke_interval_seconds=60, timeout_seconds=3600, delta=timedelta(days=1) )

The expectation is that the job fails saying "this job id does not exist" or similar but we see that it keeps searching that random job id and is in loop.

Screenshots image (6)

Brickflow 10.3 version