Load test locally 150 users and 100,000 records

[mjtravers]

Benchmark current capabilities while running the locust script with 150 users and 100000 records. Profile with Silk to identify request durations and timing within each request.

Create tickets per performance issue discovered.

Load test all schedules (locust currently only set up for schedule A)

There is a PR in progress for this ticket: https://github.com/fecgov/fecfile-web-api/pull/882

Here is a spreadsheet example 🔒 Locust testing spreadsheet 🔒

QA Notes

null

DEV Notes

null

Design

null

FECFILE-190

[mjtravers]

Moving ticket back to the backlog to be worked on later in the sprint. Ticket #843 takes precedence.

Current work captured in this branch: https://github.com/fecgov/fecfile-web-api/tree/feature/819

[Elaine-Krauss-TCG]

Successfully imported 99,999 records (+1 record for the testing committee account that already exists) using a fixture generated by my new script.

[Elaine-Krauss-TCG]

Surveying Silk profiling results has shown a few key opportunities for improvements:

• The report to_representation method is taking 1,000+ms to calculate during heavy load
  • Calculating whether or not a report can be deleted is primarily responsible for this (report serializer, can_delete method)
• Flagging reports for recalculation adds considerable time to creating/updating transactions
• Summary page calculation never completed in upwards of 10 minutes of calculating
• All requests suffer in performance with 150 authenticated users making 10 requests per second. As an example: even with thousands of contacts, loading the contacts page takes <100ms in isolation, but it was taking upwards of 4,000ms during the height of locust testing.

Requests for lists of transactions are also extremely taxing with response times upwards of 8,000ms for a list of 10 transactions. This is primarily due to just-in-time aggregate calculations. Thankfully, this is already being addressed in another ticket.

Transaction schedules A, B, C, and E were included in testing. Including Schedule D in the generated bulk data fixture was consistently causing importing the fixture to fail.

For future silk profiling: devs can merge in the silk-profiling-base branch to quickly set up the API for silk testing.

You can find a spreadsheet with Locust and Silk testing data here

[Elaine-Krauss-TCG]

New tickets were created in the aftermath of this ticket:

• Refactor Report Recalculation Flagging
• Investigate non-terminating report summary calculation
• Refactor Report Serialization

[mjtravers]

Passes CR. Sending to QA.

Notes for QA: The outcome of this ticket are the notes in the comments above the the 3 new performance-related tickets listed above as well.

[WiseQA]

QA review verified per visual inspection the DEV screenshot above showing the successfully imported 99,999 records (+1 record for the testing committee account that already exists) using a fixture generated by my new script.

QA Review Completed. Moved to Stage Ready.

[exalate-issue-sync[bot]]

akhorsand commented: Accepted during PI Planning Sprint Review on 7/8/2024