Add a preprocessor required to convert string to time object, which is required to add the time_sensor to boundary-layer.
Context / Why are we making this change?
WINS squad is working on an automated slot management service for BigQuery called Slot Machine. The service lives in CloudRun and needs to be deployed at specific times of the day to resolve slot contention issues in BigQuery while controlling costs and optimizing performance. We would like to simplify our DAGs in Airflow by keeping relevant tasks together in a single DAG, but scheduling them for specific times of the day. The time_sensor in Airflow solves for this, however, it needs to be added to Boundary Layer, alongside this processor.
hyoung87
commented
2 years ago
Description
Add a preprocessor required to convert string to time object, which is required to add the
time_sensorto boundary-layer.Context / Why are we making this change?
WINS squad is working on an automated slot management service for BigQuery called Slot Machine. The service lives in CloudRun and needs to be deployed at specific times of the day to resolve slot contention issues in BigQuery while controlling costs and optimizing performance. We would like to simplify our DAGs in Airflow by keeping relevant tasks together in a single DAG, but scheduling them for specific times of the day. The
time_sensorin Airflow solves for this, however, it needs to be added to Boundary Layer, alongside this processor.Testing and QA Plan
Function has been tested successfully.
Impact
NA.