Implement a method to 'fill in' gaps in the in-situ data.

[rstrange-scottlogic]

Due to an issue our ETL scripts stopped capturing data for awhile.

This has left us with gaps in the data for certain dates. We would like a way to remedy this.

• This story only covers the in-situ ETL script
• We will add a new configuration setting to drive how far back in the past we want to go (default to 7 days) - we can reuse the one from the forecast ETL?
• The script checks the database for missing in situ time values values and retrieves any missing datasets
• Includes integration tests that the data is returned.
• We don't need to completely replicate the cron job as we are already fetching a full days worth of data each time.

Test Checklist:

• [x] Test Analysis & discussion with developer
• [ ] Write Test Cases / Charters
• [x] Test Cases & Charters Reviewed
• [ ] Test Cases Executed
• [x] Test Charters Executed
• [x] Regression tests executed
• [ ] Automation Tests added to project

[mwalker-scottlogic]

Test Analysis

Manual approach:

• using the local QA database
• run main
• establish how many documents are in the last 7 days
• delete a days
• run main
  • check for specific locations being re-uploaded
• Captured in Charter 29: Explore QA local database with deleted items to discover if fill in gaps method is working as anticipated ✅

Automation:

• similar approach, will need to be relative to whatever day the test is executed on
• run main
• store as variable the number of documents for a specific city on today -1 days
• delete these
• run main
• verify if they have been re-uploaded
• Ran out of time to implement automation, but have been able to manually verify in the above charter the behaviour operating as expected

[mwalker-scottlogic]

Frontend regression PASSED ✅

Playwright Regression 29.08.24.pdf

Backend regression PASSED ✅