Add monitoring to NoK/EC requests from VA Profile v3 service

radavis commented 9 months ago

Description

Follow-up from #67088

User story

As a Cartographer, I want to know if a request for NOK/EC data from VA Profile fails, so that I can understand why it failed.

As a Cartographer, I want to know how many requests were made for NOK/EC data, so that I can know if users are taking advantage of this feature.

Notes

Follow examples from other vets-api integrations, especially Profile ones
There is monitoring on the controller function, but not on the "client" that makes the actual call to Profile back-end
~Datadog should not log a 404 for the application if the connection from front-end to vets-api is successful~ https://github.com/department-of-veterans-affairs/va.gov-team/issues/76239#issuecomment-1999573696

Possible tasks:

Check with Patrick about monitoring
Confer with Profile team about Datadog dash setup, particularly Travis and Tom

Tasks

[x] Determine what to monitor (e.g. - requests made, requests failed, status codes, etc)
[x] Determine which monitoring solution makes sense, here (e.g. - Datadog, Sentry, Statsd)
[x] Gain access to monitoring solution
[x] Update vets-api to catch failing requests, and report to monitoring solution
[x] Verify failures are reported within monitoring tool

Acceptance Criteria

Requests from the front-end to vets-api for NoK/EC are logged in Datadog and tracked in the Profile dashboard alongside other sections of the profile

wesrowe commented 8 months ago

Richard, Lynne and I met with Travis C today, and Travis showed us the existing Profile dashboard in datadog. I modified the ticket above to capture our learnings.

radavis commented 8 months ago

@wesrowe: "Datadog should not log a 404 for the application if the connection from front-end to vets-api is successful" is outside the scope of monitoring. This will require a separate ticket for changes to vets-api and vets-website.

radavis commented 8 months ago

Added graphs of response status codes for /v0/profile/contacts in staging to the Authenticated Experience: Profile dashboard

Screenshot 2024-03-15 085100

radavis commented 8 months ago

To add StatsD metrics to a backend client, add the Monitoring concern to the client, then wrap HTTP calls in a with_monitoring block. Client should have STATSD_KEY_PREFIX declared on the client or in the inheritance chain.

radavis commented 7 months ago

Datadog monitor: https://vagov.ddog-gov.com/monitors/210521?view=spans

wesrowe commented 7 months ago

we had a review meeting with Carlos yesterday – looked good, can close.

department-of-veterans-affairs / va.gov-team