These changes are in line with the reasoning that's described in #101.
Make sensor results repeatable if workflow is restarted by using the execution_date which is derived from Brickflow context and not the current time. This ensures that relative delta between the workflow run (based on it's schedule) and the upstream DAG run always stays the same.
Sensor will always poke upstream, unless timeout is reached. This ensure that if upstream is failed or running, or in some other state, the sensor will continue waiting for the success (by default)
Related Issue
101
Motivation and Context
Improve sensor stability and simplify workflow reruns.
How Has This Been Tested?
Unit tests + this code had been overriding the original sensor and running in production pipelines
Screenshots (if appropriate):
Types of changes
[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)
Checklist:
[x] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
Description
These changes are in line with the reasoning that's described in #101.
execution_datewhich is derived from Brickflow context and not the current time. This ensures that relative delta between the workflow run (based on it's schedule) and the upstream DAG run always stays the same.success(by default)Related Issue
101
Motivation and Context
Improve sensor stability and simplify workflow reruns.
How Has This Been Tested?
Unit tests + this code had been overriding the original sensor and running in production pipelines
Screenshots (if appropriate):
Types of changes
Checklist: