Performance test celery .fec file creation with large filings

[dheitzer]

The idea is to create a large .fec file and then stress the cloud.gov resources as they generate the files.

For this ticket, Locust should hit the "download .fec" api endpoint.

We need to set up and run performance testing on the fecfile-web-services celery filing functionality using large payloads. To do this, we need to test the submit functionality (not against testefo however) with large committees or committees with lots of data. During these tests, we will want to be monitoring the speed and resource utilization of the application.

• [x] Create a test for requesting a .fec file with 1,000 transactions to start
• [x] If 1,000 transaction successful, ramp up number of transactions incrementally until the celery services experience load issues.

Note: Please do not submit to testefo for this. The key here is that we're trying to performance test the .fec generation for committees with lots of data.

QA Notes

QA will receive a report on server performance.

DEV Notes

Do not perform test on Cloud.gov resources, just locally.

Design

null

FECFILE-171

[exalate-issue-sync[bot]]

Matt Travers commented: Ticket on hold while we sort out whether we want the Celery testing to make changes to the database.

[exalate-issue-sync[bot]]

Matt Travers commented: Passes CR. Sending to QA.

Ticket taken off hold. No performance tests will be made in the [cloud.gov|http://cloud.gov] space using this method until deemed appropriate.

[exalate-issue-sync[bot]]

Sasha Dresden commented: Created new TEST_RUNNER which silos tests marked with the performance tag. To use run: python manage.py test --tag=performance

All tests marked +@tag+|https://github.com/tag will be run (currently only the one). By default python runs all tests, so I had to modify the test_runner so performance tagged tests wouldn't be run when no tags were provided.

Currently this can only be run locally, however a future ticket could look into conditionally running on other environments.

Performance Report I ran this test multiple times with an increasingly large number of transactions. Results below. 2k took 8.3 seconds 5k took 17.5 seconds 10k took 35.8 seconds 25k took 85.1 seconds

From these results it appears there might be some initial setup cost, but for the most part this is a linear process and higher numbers can be extrapolated. It seems to be able to process around 285 per second, however the greater the number of transactions the faster the rate, so I'm guessing it might be closer to 300 per second, so based off these estimates 1 million transactions would take a little under 1 hour.

However, I didn't want to test further than 25k as the setup time was getting quite long. Creating 25k transactions took me 39 mins, however this was after I applied a performance improvement to the transaction aggregate trigger. Before this improvement 2k transactions took 20 minutes. I did not include this trigger performance enhancement as part of this ticket as I did not want to have this PR be dependent upon that change.

[exalate-issue-sync[bot]]

Matt Travers commented: Passes CR. Sending to QA.

Server performance report in ticket comments.

[exalate-issue-sync[bot]]

Shelly Wise commented: QA review verified with DEV Performance Report for this ticket was provided (see below comment)

QA Review Completed. Moved to Stage Ready.

[exalate-issue-sync[bot]]

akhorsand commented: Accepted by Paul Clark at 7/30/24 sprint review.