Since most API's are heavily used by multiple systems, we need to have proper performance testing, in a well defined setting with proper test scenario's upon every (at least major and minor) release.
There are multiple goals to be achieved and not all of these goals align in terms of "what" to test.
See whether the new release is at least not slower than the previous release
Find the optimum configuration (infra/settings) for a given load
Locate bottlenecks and fix them
This issue is about 1.
For 2 there is a separate issue that focus specifically on getting the most performance
For 3 it's important that a developer can perform a ad-hoc performance test, identify bottlenecks or replicate reported (like observed in 1) bottlenecks, and fix them. Separate issue.
[x] Decide on what we're going to test
Different systems (cpu/memory)? No, fixed
Different number of nodes? No, fixed
Different number of uWSGI processes? No, fixed
Different amount of parallel requests? No, fixed
Different amount or type of data? No, fixed
[ ] Decide what want to see as result: Relative URL, Average, 95%tile, 99%tile, etc.
[ ] Create test dataset, ideally with a command: 1 million of everything or so
[ ] Create test scenario's to run: Ideally, a GET for every resource with and without one or more filters (POST can be done later)
[ ] Create a format to document the performance test results for each release
[ ] Make sure everything is repeatable and reproducable
[ ] Add first report in new format to docs
This should be done in a generic way, and then spread across all API projects.
Since most API's are heavily used by multiple systems, we need to have proper performance testing, in a well defined setting with proper test scenario's upon every (at least major and minor) release.
There are multiple goals to be achieved and not all of these goals align in terms of "what" to test.
This issue is about 1.
For 2 there is a separate issue that focus specifically on getting the most performance For 3 it's important that a developer can perform a ad-hoc performance test, identify bottlenecks or replicate reported (like observed in 1) bottlenecks, and fix them. Separate issue.
This should be done in a generic way, and then spread across all API projects.