search

opensearch-project / opensearch-benchmark

OpenSearch Benchmark - a community driven, open source project to run performance tests for OpenSearch
https://opensearch.org/docs/latest/benchmark/
Apache License 2.0
111 stars 78 forks source link

[FEATURE] Add relative standard deviation (RSD) to the new aggregate feature #662

Closed OVI3D0 closed 3 weeks ago

OVI3D0 commented 1 month ago

Is your feature request related to a problem? Please describe

The new aggregate command returns the average of multiple test execution results. It would be helpful to users if they were able to see the relative standard deviation, so users can see the dispersion of their data.

Describe the solution you'd like

Add an output to the aggregate command to show users the relative standard deviation of key metrics. The RSD can be calculated by getting the mean μ and the standard deviation σ, then calculating the RSD as ( μ / σ ) * 100%

OVI3D0 commented 1 month ago

@IanHoang maybe it can look something like this?

|                                                 Min Throughput |   term |        9.99 |  ops/s |
|                                                Mean Throughput |   term |          10 |  ops/s |
|                                              Median Throughput |   term |          10 |  ops/s |
|                                                 Max Throughput |   term |          10 |  ops/s |
|                                              RSD of Throughput |   term |           x |  ops/s |

or it can be called with a flag like --relative-standard-deviation and return a separate table displaying the relative standard deviation of some key metrics?