cds-snc / notification-planning-core

Project planning for GC Notify Core Team
0 stars 0 forks source link

New Relic Tuning #257

Open jzbahrai opened 6 months ago

jzbahrai commented 6 months ago

Description

We get this error a lot:

Issue ab41fe52-8794-4dfb-87dc-53c25b2b98bd active (opened) Duration 5 min impacted entity (Target) AwsLambdaInvocationError, AwsLambdaInvocation query condition [Lambda API] Error percentage (Fuzzy attack) policy Notify production Issue name (Threshold) api-lambda query result is > 10.0 for 5 minutes on '[Lambda API] Error percentage (Fuzzy attack)'

We want to make this more useful. We get a lot of auth token/ invalid pass errors.

One suggestion made by Jimmy is to surface these common errors in a NR dashboard, and only raise up these warning unless we have a lot more and we would want to action on it. .

jimleroyer commented 6 months ago

I created a user-centric dashboard that might address some of the New Relic alarms we get in #notification-ops. I want to create another one on fuzzy attacks as well.

image.png