search

CityofToronto / bdit_data-sources

Data sources used by the Big Data Innovation Team
https://github.com/orgs/CityofToronto/teams/bigdatainnovationteam
GNU General Public License v3.0
39 stars 8 forks source link

ExternalTaskSensor with execution_delta on timezone change #1100

Open gabrielwol opened 6 days ago

gabrielwol commented 6 days ago

Affected ecocounter_pull -> ecocounter_check and miovision_pull -> miovision_check DAG pairings on 2024-11-03 (EDT->EST time change).

Possible solution: use execution_date_fn (callable) to make delta reactive to time zone. 😵 https://airflow.apache.org/docs/apache-airflow/1.10.12/_api/airflow/sensors/external_task_sensor/index.html

gabrielwol commented 5 days ago

Confirmed failures were due to upstream failures with time change, not extra hour. Only logical_replication_health affected by timeout issue: https://github.com/Toronto-Big-Data-Innovation-Team/vfh_data_mgmt/issues/277