HHS / simpler-grants-gov

https://simpler.grants.gov
Other
43 stars 13 forks source link

[Task]: Create Alert for Sendy Failures #1066

Open acouch opened 9 months ago

acouch commented 9 months ago

Summary

When there is an error subscribing users to Sendy, we should get an alert.

Errors for subscribing are logged here: https://github.com/HHS/simpler-grants-gov/blob/00ca83fb01d41d317c8fe6d40d9a1a4d2f0d332e/frontend/src/pages/api/subscribe.ts#L63

These errors are available in the service/frontend-prod log group (see: Error subscribing user: fetch failed).

Steps

  1. Evaluate if current error logging messages are ideal in the subscribe.ts file. Ie it might be better to have them more uniform or start with Sendy error: to make them easier to find in the logs.
  2. Determine number of errors to initiate an alert. I would imagine this is 1 since any errors subscribing someone would require investigation
  3. Determine best strategy for log / filtering and alerts
    • Anomoly detector or metric and alarm
  4. Create alarm

Questions

  1. Is there a way to test this once it is implemented?

Acceptance criteria

SammySteiner commented 8 months ago

Here is where the code needs to be updated: https://github.com/HHS/simpler-grants-gov/blob/6b3be7e3a52f3328534191632310d2322b2eb0ea/frontend/src/pages/api/subscribe.ts#L69

mxk0 commented 3 weeks ago

@acouch is this still an open issue?

acouch commented 3 weeks ago

@mxk0 this was never implemented.