Increase HPA Scaling Threshold

ericboehs commented 1 month ago

With the readiness probe, our pods are not constrained by round-robin busyiness, our HPA threshold can go up which will decrease our overprovisionedness.

Tasks

[x] Increase HPA threshold (the used capacity threshold in which we scale up)
[x] Monitor performance characteristics
[x] Identify and update Datadog dashboards that display the threshold

Acceptance Criteria

[x] Vets API (all envs) is less overprovisioned by scaling later
[x] Datadog dashboards reflect new HPA value

Notes

Extracted from https://github.com/department-of-veterans-affairs/va.gov-team/issues/87124.

ericboehs commented 1 month ago

I've increased it to 40 (from 30). Going to test out 50 today.

ericboehs commented 1 month ago

We're now at 60. I'm hesitant to go to 70 until we've got some other scaling values tuned.

ericboehs commented 1 month ago

We're pretty happy with our HPA at 60. I just lack updating a few charts in DD and this is done.

ericboehs commented 1 month ago

Updated the charts I could find. This is pretty easy to fix though if I missed any.

JoeTice commented 1 month ago

SPRINT 6 UPDATE: DONE - This ticket was completed on 7/31, but didn't get marked as closed

department-of-veterans-affairs / va.gov-team