peanball
commented
2 months ago There is also the option to compute various statistics, including percentiles, on the fly without retaining all results: https://github.com/influxdata/tdigest
This could either be an alternative mode that disables statistics that require all samples, or we could make the existing statistics approximate well enough while streaming without the need for any persistent storage.
CSV exports could also be directed to a file and streamed "one the fly" if needed.
Currently the code allocates a single channel with a fixed 1000000 results. This limits the usability when going beyond 10k req/s and running a longer test.
It would be beneficial to process the results in parallel as increasing the buffer would eventually cause memory issues. We have to be careful to not consume too much processing power to still be able to fire a lot of requests.