Load testing

[laineymajor]

PROBLEM STATEMENT Load testing needs to be completed to check how systems function while a heavy volume of concurrent virtual users perform transactions over a period of time. We will run a Load Test to:

• Assess the current performance of our system under typical and peak loads.
• Make sure we continue to meet the performance standards as we make changes to our system (code and infrastructure).

ACTION STEPS/TASKS

• [x] Igor to write technical instructions for VFS teams to follow in order to complete load testing
• [x] Determine peak loads and times (Mondays just before noon ET)
• [x] Work with 2 VFS teams with existing load testing scripts to prep for run of load tests on March 20 from 11:30-12:30ET
• [ ] Work with 2 VFS teams with existing load testing scripts to run load tests on March 20 from 11:30-12:30ET
• [x] Lainey to communicate with VFS teams
• [x] Igor will run load tests for platform on March 20th from 11:30-12:30ET; enough to simulate the peak load
• [x] Load tests will be run
• [x] VFS teams, including Tech Team 1, will record load test results

DEFINITION OF DONE

[laineymajor]

@laineymajor to send out communications to VFS teams about upcoming load testing.

[laineymajor]

@laineymajor drafting new MVP load testing plan and will include in this ticket

[LindseySaari]

Loadtesting dashboard in Datadog

[laineymajor]

@laineymajor update with MVP plan

[laineymajor]

grafana (or jmeter) current request rate dashboar - lindsey to add

[laineymajor]

I've updated the action steps to match the new MVP load testing plan

[laineymajor]

This ticket will carryover to next sprint as we needed to give VFS teams some lead time to prep for load testing. We will be completing load testing together on March 20.

[laineymajor]

TEAMS TO TEST

1. checkin team
2. lighthouse team
3. Platform team (internally done)

[laineymajor]

What needs to be done BEFORE load testing on Monday:

• [ ] what are the metrics? what is the normal load we are looking to simulate? - [ ] Q for VFS teams: Are any of the tests being run, kicking off sidekiq jobs **If no, we will need to generate some load to kick off batches of jobs
• [ ] Rachal: Adjust the pod min/max
• [ ] Rachal to prep to kickoff sidekiq jobs
• [ ] sign in team and lighthouse team to run scripts that they've written (ensure they have scripts to run)

[LindseySaari]

Potential boards to pay attention to: Load testing dashboards:

• https://vagov.ddog-gov.com/dashboard/2ak-msa-hs4/eks-hpa-vs-brd-asg?from_ts=1678388113312&to_ts=1678992913312&live=true
• https://vagov.ddog-gov.com/dashboard/b2i-kmn-jr7/eks-deployment-data?from_ts=1678820232140&to_ts=1678993032140&live=true
• Pods overview: https://vagov.ddog-gov.com/dash/integration/24/kubernetes-pods-overview?from_ts=1678978664273&to_ts=1678993064273&live=true
• See EKS USE and RED sections: https://vagov.ddog-gov.com/dashboard/9dk-y7n-6vf/platform-monitors-alerts?from_ts=1678979069356&to_ts=1678993469356&live=true

[LindseySaari]

Load testing today at 12:30PM

[LindseySaari]

Board for pods in the vets-api-staging vets-api namespace

Deployment dashboard

[LindseySaari]

We had an issue with the check-in team tests. 500 errors were being returned, but we pushed a fix. Thread here

[LindseySaari]

After the pushed the fix, load tests ran at 100% success.

@considerable Please type up a summary of the outcome and the issues we ran into. Thread here

[LindseySaari]

Link for search load test script

[LindseySaari]

Adjusted liveness probes yesterday. Requests are only going to one pod, so we need to re-run the tests. The script may be mocked and requests are for some reason going to 1 pod.

Next step: scale down and re-run tests @oseasmoran73, @considerable and Kanchana will need to re-run the scripts. We will scale down to 2 pods or decrease the targetAverageValue.

[LindseySaari]

Console commands:

git clone git@github.com:department-of-veterans-affairs/vets-api-loadtest.git

cd vets-api-loadtest

docker run --rm -v `pwd`/loadtest:/loadtest -i locustio/locust:2.14.2 \
  -u 40 -r 5 -t 30m --headless --only-summary -H https://staging-api.va.gov \
  -f /loadtest/search/search_locust.py

[LindseySaari]

At 2:25 on 3/23, the team ran the loadtest script and scaling is working

[LindseySaari]

After the load test completed, pods scaled down properly as well!

[LindseySaari]

Once the loadtest summary is updated, we can close this ticket @considerable

[laineymajor]

@considerable did you complete load testing on Monday with the VFS teams? Please provide a detailed summary on the work you did with the VFS teams.

[laineymajor]

@considerable to provide summary, then close ticket (before COB 3.27).

[considerable]

A. Load test for smoke-testing

Note: Amazon blog Load testing your workload running on Amazon EKS with Locust explains load testing for the performance and reliability of a workload by generating artificial loads that mimics real-world traffic.

Reality check with actual load test ran 3/20/2023 against staging-api.va.gov:

• During our 1 hour of coordinated load testing, we produced 200 hits hourly rollup while production had 270 hits hourly rollup, i.e. we produced a load comparable to production.
• Out of the many possible test scenarios we ended up with 2 simultaneous /v0/search endpoint test-running machines in different locations with 40 users each, that was 80 simultaneous users in total.
• The system metrics at staging-avi.va.gov such as the number of instances required for the run gave us no surprises, everything occurred working in a “business as usual” state.

Lesson learned:

• The hourly rollup visualization turned out to be a useful tool to compare the load against normal prod.
• The perfect test run requires some practice, tuning, coordination, and likely a code change or IP whitelisting so that it might be practical for individual teams or individual endpoints to run a separate load test while comparing the load they generate with the normal production load, which currently reads about 270 hits for hourly rollup.

So, what the load test said?
Has the EKS infrastructure enough capacity to run the code? - YES:

• The code and the infrastructure currently are in balance for deployment in production.

---

B. Load test for the auto-scaling configuration

Note: Amazon recommends performing load tests to choose an automatic scaling configuration that works the way you want. See Load testing your auto-scaling configuration.

Important notes:

1. Initially, with the load of 500 req/min to /v2/check_in for 30 mins, vets-api had only scaled up by two pods. Therefore, vets-api got to be scaled down from 16 pods to a low number, so we can see how the pods do scale with load.
2. Upgrade to ruby 3 might get in the mix.
3. Researching log, distinguish those entries showing instance IDs instead of pod IDs.
4. Maybe we caused a breaker outage with the load.
5. ClamAV was not able to connect. Shortly after the limit was reached, the pod was taken out of service. ClamAV memory was increased to 3Gi.
6. Enable the mocking for load tests to work.
7. Kudos to Eric and Kanchana w/o whose help we wouldn’t have been able to find those errors. We really appreciate it!