search

cal-itp / data-infra

Cal-ITP data infrastructure
https://docs.calitp.org/data-infra
GNU Affero General Public License v3.0
48 stars 13 forks source link

Refactor(GTFS Schedule): Merge updates should leverage task groups #1153

Closed lauriemerrell closed 2 years ago

lauriemerrell commented 2 years ago

There's a TODO in merge_updates to use task_groups for that task. Now that we have upgraded to Airflow v2, we can do that.

Alternatively we could at least use a macro and make each task distinct. But right now, making changes that affect merge_updates is very difficult (ran into this on #1148).

lauriemerrell commented 2 years ago

Type 2 logic will be refactored with dbt transition, we will not be doing this.