search

department-of-veterans-affairs / va.gov-team

Public resources for building on and in support of VA.gov. Visit complete Knowledge Hub:
https://depo-platform-documentation.scrollhelp.site/index.html
281 stars 198 forks source link

Catch 503/504 errors for LGY API #92416

Open mjknight50 opened 1 week ago

mjknight50 commented 1 week ago

Overview

We need to catch the 503/504 errors for COE and log them separately than other errors. We are dealing with excessive monitor noise on errors that are not related to our code.

app/controllers/v0/coe_controller.rb app/models/saved_claim/coe_claim.rb

The goal here is to catch 503/504 errors and log them with Rails.logger.info message and then tie them to a monitor with a super high threshold to be set off.

Then the regular errors should be tied to the existing monitor: https://vagov.ddog-gov.com/monitors/186673

Tasks

mjknight50 commented 1 week ago

More context about the errors can be found in these tickets: https://github.com/department-of-veterans-affairs/va.gov-team/issues/90411 https://github.com/department-of-veterans-affairs/va.gov-team/issues/91111

tfink419 commented 4 days ago

I've created the monitors https://vagov.ddog-gov.com/monitors/272234 and https://vagov.ddog-gov.com/monitors/272235 Each monitors one of the new messages and compares to amount of controller calls that have been logged similarly to the old error. Note that these monitors are of the Log type and the old one was of type Metric, there was no way to monitor based on the message content with a metric-type monitor