Incremental Loading - Stage 1 - Metrics

[Ben-Hodgkiss]

Overview In order to monitor the progress of incremental loading, and monitor how much of an effect the changes have, we need to gather metrics about the collection run times so they can be compared before and after the change.

Pull Request(PR):

Tech Approach

• The Airflow dashboard shows the state and run time of recent runs
• A simple manual (but time consuming) process could record there
• The Airflow API would allow them to be obtained via code
• There could be Airflow plugins to do this
• Investigate a solution that offers a fair manual effort / development cost trade off. Speak to Ben

Acceptance Criteria/Tests

• We have a set of times for each collection run
• Organise time with Ben to show him the metrics in airflow check he's happy with the solution

Resourcing & Dependencies

• Developer, airflow access. AWS / devops assistance.
• No dependencies.

[cjohns-scottlogic]

Will use the airflow UI as an initial, if manual process. To be reviewed if this gets too unwieldly to add some automation.