search

Nike-Inc / brickflow

Pythonic Programming Framework to orchestrate jobs in Databricks Workflow
https://engineering.nike.com/brickflow/
Apache License 2.0
190 stars 45 forks source link

[FEATURE] TaskDependencySensor flexibility Airflow sensor #154

Open IMC07 opened 2 months ago

IMC07 commented 2 months ago

Is your feature request related to a problem? Please describe. Given the fact that in the taskdependencysensor we can only filter the API response on greater then or equal to timestamp-x, the workflow we make is non-deterministic, because when we would rerun a pipeline, the window of time changes because the end-time is 'now'. We would like to be able to make this deterministic, to really be able to see if there was a succes within the timeframe we allow for our pipeline.

Cloud Information

Describe the solution you'd like I have created a simple, slightly ugly solution in order to stay backwards compatible, see PR. However, if you agree, I would like to refactor the sensor to allow for more flexibility on filtering results from the API.

Describe alternatives you've considered Create a more flexible API filter as said above.

Additional context Add any other context or screenshots about the feature request here.

asingamaneni commented 2 months ago

Let's go with the patch right now and we can go for breaking change in the future releases.