department-of-veterans-affairs / va.gov-team

Public resources for building on and in support of VA.gov. Visit complete Knowledge Hub:
https://depo-platform-documentation.scrollhelp.site/index.html
283 stars 205 forks source link

Pension | datadog monitors clean-up #77236

Closed buddhima-coforma closed 1 month ago

buddhima-coforma commented 8 months ago

Overview

The pension monitors are inconsistently named and in some cases don't seem to be correct.

https://vagov.ddog-gov.com/monitors/manage?q=team%3Abenefits%20service%3A%22pension-application%22&order=desc

Tasks

Naming convention: Benefits <form_id> <form name> <type> <controller/job> <event> <method> e.g.

Benefits 21P-527 Pension Metric SideKiq exhausted count
Benefits 21P-527 Pension Metric Pension_claims 422/500 errors count
Benefits 21P-527 Pension Logs Pension_claims Status:!200 >20%
aplatt-coforma commented 8 months ago

Hey team! Please add your planning poker estimate with Zenhub @danlim715 @mjknight50 @TaiWilkin @ToddWebDev @wayne-weibel @scottsdevelopment

TaiWilkin commented 8 months ago

The PensionClaimsController doesn't get much traffic, typically only one request per fifteen minutes. If the average over 15 minutes is that 20% of responses are errors, the alert will trigger, meaning that in general we will get an alert if there are any errors on this controller. Therefore, increasing the sensitivity of this controller wouldn't make much of a change at this time. However, if the frequency of requests to the endpoint increases, we should revisit this.

wayne-weibel commented 1 month ago

https://dsva.slack.com/archives/C053U7BUT27/p1727294692767399

93599