TangoYankee
commented
4 months ago @TylerMatteo I rushed the below summary so that I could move on to higher priority tickets. However, there should be enough here to move along the load testing discussion.
Tooling Landscape
The article 'best-load-testing' served as an outline for what to look for in load testers. There are many proprietary and open source services. As established in previous discussions, our team leans to open source solutions. Open source tooling of interest includes Auto Cannon, k6, apache Jmeter, and artillery. We also develop primarily in javascript and our tooling should support this. This is a mark against J meter because it is leverages Java. The tooling should be cloud agnostic to avoid vendor lock. This is a mark against Artillery because it targets AWS lambda and Fargate for its test runners. This leaves autocannon and k6. Autocannon appears to be a basic and adequate choice. However, K6 has more features in the both the test running and data analysis phase. Autocannon nominally is more mature- it is on v7 while k6 is on v 0.40. However, the k6 team is larger, part of a company (Grafana), and the team has been developing the tool since at least 2017. K6 is selected for a deeper dive.
K6 Experimentation
K6 is a Go application but reads javascript for its scripting language. It can be run in a Docker container. Several experiments were conducted for k6. These experiments involved test running, file building, and data display.
The first configuration created a basic test for tax lots. It is on the commit for Basic tax lot tests. The tests are run in Docker against the zoning api running directly on the computer's localhost. There are three npm scripts. The bundle command takes the typescript files and transpiles them into javascript to be run in K6. The testnpm script runs k6 in docker with the load test script. The go script runs 'bundle' and 'test' in series. K6 only runs one script at a time and organizing the tests is an open questions. But, each script can organize itself with each request assigned a tag and/or group. In a basic setup, the tests produce performance summary statistics and pass/fail outcomes for customizable thresholds. The next section explores more complex test result products/visualizations.
Grafana visualizations k6 is part of the Grafana ecosystem. Its results can be saved to a time series database and visualized through Grafana. The linked demo uses influxDB (based on Rust). There is also a Postgres extension called Timescale. However, the example timescale repo relies on installing Go natively. Finally, Azure does have a Managed Grafana service. These tools show how we could scale the data intake from k6. It offers a second-by-second display of app performance, reflecting how the app reacted during different stages. However, data visualization should probably be de-scoped from an MVP because it requires yet another service to host the Grafana service.
Auto-generated tests Finally, there is tooling around automatically generating tests from the open api documentation. The open api test file was semi-automatically generated from these instructions. However, it falls short in two ways. First, the generated file needs to be manually edited with key parameters before it can be run. These precludes automatic updates to the tests because the manual edits would be overridden. Second, the tests are grouped by domain. However, the groups documentation suggests using Behavior-Driven-Development; we would end up rewriting the auto-generated tests to regroup them.
Next Steps
- [ ] Make a repo specifically for load testing
- K6 requires its own webpack and docker configurations. We can centralize/reduce duplication by using the same setup for load tests across all services
- Load tests can be run against any url, making it easy to point the test scripts to any service regardless of where that service lives.
- We do not necessarily need load tests to run as part of its target applications CI/CD flow. Instead, we can run load tests on a separate cycle through their own pipeline. Though if we do want to couple application releases and load tests, GitHub does have a work-around to have one repo trigger events in a second repo.
- Placing load tests for all services in one repo may help us think about/design tests for/ the services interacting with each other. (In contrast, collocating load tests and their tested service may pigeon-hole into think only about "unit" load tests.)
- [ ] Make a script to target the basemap server and run it with GitHub actions (using basic summary stats and threshold checks)
- For the same reasons that basemap server is the first service we're attempting to move to Azure, it should be the first service we attempt load tests. Ie) it's an application we rely on in production and need to reinforce
- Because we will be putting the basemap server on Azure, it can actually help us test Azure as much as it helps us test the basemap server.
This Issue is a spike to research best practices and tooling for performing API load testing in the context of our back end tech stack.
Possible outcomes