The exception handling in process_ses_results only catches NoResultFound exceptions. We need to broaden the exceptions that this method catches as we cannot and should not continue processing an ses results without all required data elements.
An incident occurred on 2023-12-15 where celery logs were experiencing a large number of errors. While this was due to an AWS service blip, it caused DB read errors, which we are not catching before we attempt to continue processing the ses result. With proper exception handling in place we can gracefully retry processing instead of blowing up our logs and sounding alarms when AWS has a blip affecting RDS.
Acceptance Criteria
[ ] Exceptions caught by process_ses_results have been expanded
[ ] Tests are written that throw random exceptions at the method to ensure we gracefully handle and retry the task
Description
The exception handling in
process_ses_resultsonly catchesNoResultFoundexceptions. We need to broaden the exceptions that this method catches as we cannot and should not continue processing an ses results without all required data elements.An incident occurred on 2023-12-15 where celery logs were experiencing a large number of errors. While this was due to an AWS service blip, it caused DB read errors, which we are not catching before we attempt to continue processing the ses result. With proper exception handling in place we can gracefully retry processing instead of blowing up our logs and sounding alarms when AWS has a blip affecting RDS.
Acceptance Criteria
process_ses_resultshave been expanded