github-actions[bot]
commented
3 weeks ago PR checklist
Default guide for a PR (if multiple PRs for the work, only keep one version of it and link to it on the other PRs)
- [ ] From the UCLH data science desktop, a validation run has been set off
- [ ] load times in UCL teams has been populated with the run information
- [ ] During the run, glowroot has been checked for any queries which are taking a substantial proportion of the total processing time. This can be useful to identify indexes that are required.
- [ ] After the run, look for any unexpected errors in the
etl_per_message_logging table, the error_search.sql file on the shared drive can be used for this\\sharefs6\UCLH6\EMAP\Shared\EmapSqlScripts\devops\error_search.sql. Create an issue if you find an unexpected exception and is not related to the changes you've made, otherwise fix them! - [ ] After the run, populate the end time in load times
- [ ] Let Aasiyah know about the completed validation and give her information on the changes and where to start with the validation
- [ ] Check validation report and give any feedback to Aasiyah if there are any changes needed on her side, iterate on getting the validation to match at least 99% (validation and emap code).
Fixes the rest of #50 by generating more realistic timestamps. Ie. more up to the moment, which is important because very fresh data is treated differently in collation.
Detect overlapping data, which would suggest a mismatch in actual vs expected sampling rate.
Simplify gap detection since the comparison vs rounded timestamps wasn't adding anything useful.