jilladams
commented
2 days ago Pulling this in, based on conversation in #18736 that we can create this monitor now for use as we roll out the new endpoint. We still don't know what our rate limits are -- if that's a hard blocker, holler.
[2024-10-04] Blocked on confirming our rate limit with Search.gov and completing cutover to the new API endpoint
Description
Today when we moved to the new search API endpoint (#18736) , we hit a rate limit, but we didn't get a forewarning. We learned it when the VA.gov - Search Browser Test alarmed, showing that Search calls in the browser were failing.
We don't have an alarm that explicitly goes off related to Search rate limiting and we probably should.
We do this for VA Forms: https://vagov.ddog-gov.com/monitors/160407, based on the rate limit for LH API user key. The monitor counts calls to the API every minute, warning at 400, alarming at 450.
We already know we have weird rate limiting problems with the search_click_tracking endpoint (#14819), and that doesn't appear to affect traffic to the actual search endpoint / get it rate limited. Maybe our rate limits are per endpoint? Or if they're per API key, we are using different keys per endpoint?
Either way, there's some kind of current firewall between v0/search and search_click_tracking already, so once Search.gov verifies our rate limits for v0/search to the new gsa endpoint, we could set up a monitor similar to the Forms one.
ACs