Monitors should be of /facilities_api/v#/va and /v1/facilities/va

eselkin commented 6 months ago

User Story or Problem Statement

We have monitoring and alerting for /v1/facilities/va and ~but that is a VAOS endpoint not our~ /facilities_api/v#/va and /facilities_api/v#/ccp for #=1 and #=2

We also should add rack_attack rate limiting to vets-api facilities_api/v2 similar to our v1 endpoints

Monitor tagging docs: https://depo-platform-documentation.scrollhelp.site/developer-docs/monitor-tagging-standards Our itportfolio is digital-experience

Description or Additional Context

We used to be under the assumption that /v1/facilities/va and /facilities_api/v1/va were the same endpoint, but they just happened to resolve the same data from lighthouse.

We do own both, as the facilities team.

Related tickets

18351 - this ticket will tell us more when we are able to resolve it. Marking it as a blocker to closing this, til we understand we've covered everything.

Original Acceptance Criteria before we understood ownership

[x] Monitoring removed for /v1/facilities/*
[x] Monitor added for /facilities_api/v1/va if it does not exist
[x] Monitor added for /facilities_api/v1/ccp if it does not exist
[x] Monitor added for /facilities_api/v2/va
[x] Monitor added for /facilities_api/v2/ccp
[x] Add rack_attack limits for /facilities_api/v2/va (30/min) and /facilities_api/v2/ccp (8/min)

New ACs now that we own /v1/facilities/va

[ ] Recreate / restore: Vets-API VA Facilities Controller: 500 errors to ensure we have monitoring for /v1/facilities/va
[ ] Consider other API monitors that we might need, to cover /v1/facilities/va . (rack_attack limits?)

eselkin commented 6 months ago

The 500s should probably not get an alert, unless it's really ridiculous. Because we always get a 500 on Vet Center CAPs (because they are not in the LH system)

jilladams commented 5 months ago

So... what had happened was:

We were confused about the usage / ownership of /v#/facilities/va, so we started removing / changing Facilities monitors of that endpoint to instead monitor the endpoint we knew we owned (/facilities_api/v#/va). This included removing monitoring for /v1/facilities/*
In the meantime: last week we figured out that we actually DO own /v#/facilities/va https://dsva.slack.com/archives/C0FQSS30V/p1718747443195809 AND
That the mobile app uses it. Which means: it really needs its own monitoring, for that reason. And we should own those monitors.

Meaning: we need to revise this ticket to cover that situation, and revise what we did / didn't do in Datadog. I've taken a pass at updating the ACs in the ticket body to add a 2nd batch in the revised world of ownership we understand now.

cc @Agile6MSkinner @mmiddaugh @eselkin

jilladams commented 5 months ago

Also as far as the blockers: we discussed the CAP related issues in this thread: https://dsva.slack.com/archives/C05UCL10WH4/p1717608517344489

Tl;dr: Vet centers make API requests related to CAPS, but LH doesn't have CAPS, so returns a 500 error. #9727 is the work for LH to be able to receive CAP data. We'd like to stop making the FE call for CAPs til LH can return them and/or make them work, reflected in #15656.

But in the meantime, we can adjust monitoring alarm limits, which Eli already did: https://dsva.slack.com/archives/C05UCL10WH4/p1717613241041999?thread_ts=1717608517.344489&cid=C05UCL10WH4

We can adjust our monitor not to alarm for #s that are the baseline for how many CAP 500s we get on average.
Tradeoff: any other facility type that started throwing 500s would probably be below the bar and could get missed.
FE change to stop making this call would be "if a CAP, don't call the API" 2-3 pointer + testing
Risk vs. Reward: we've never seen LH 500 errors besides this. The risk is low to just adjusting the monitor.

So I do not think this ticket is actually blocked by CAP-related things in real life anymore. Removing that status.

jilladams commented 4 months ago

Also noting: we removed a monitor in https://github.com/department-of-veterans-affairs/va.gov-cms/issues/17791 that may have pertained to the Legacy API client, and that we may want to reinstate when we pick up this ticket.

department-of-veterans-affairs / va.gov-cms