Search error logging

[mxk0]

Summary

To meet our quad 1 commitments for search, we need charting that allows us to prove:

The monthly average error rate remains below 5%, meaning fewer than five 5xx errors per 100 API and five 5xx errors per 100 page requests in production.

Acceptance criteria

• [x] Ticket to prove we have the needed API logging and charting (see screenshot below)
• [x] Ticket to prove we have the needed frontend logging and charting (#2633)
• [ ] Ticket to set alerts for error rates >5% (both API and frontend)
• [x] Ticket to screenshot API logging (at the end of the quad) (#2660)
• [x] Ticket to screenshot frontend logging (at the end of the quad) (#2660)

[coilysiren]

[mxk0]

We have the logging we need for the API. We can view these logs in CloudWatch today, and can use a future monitoring tool (e.g. NewRelic) in the future.

Still need to check what we have for the frontend.

[coilysiren]

https://docs.aws.amazon.com/opensearch-service/latest/developerguide/monitoring-pipeline-logs.html

[mxk0]

@mdragon do we have alerts like this in place already? (Copied from the acceptance criteria above.)

Ticket to set alerts for error rates >5% (both API and frontend)

[mdragon]

Confirming we have the same level of detail for Frontend as they're both ELB and those natively report out their Target server and ELB level 5xx error metrics.