search

edanalytics / edu_edfi_airflow

Manages extract-load of Ed-Fi data in Airflow
Other
4 stars 0 forks source link

Feature/refactor change version post copy #9

Closed jayckaiser closed 1 year ago

jayckaiser commented 1 year ago

This PR builds off of feature/refactor_change_version (PR #8). It uses xcoms to collect a list of successful copies into Snowflake and updates the meta-change-version table with a single insert statement. This should improve run-time performance even more, as it strips out up to another ~200 tasks from each run.

jayckaiser commented 1 year ago

This PR merges all new change-version refactoring into a single branch for easier merging to main later.