Open axw opened 4 years ago
Similar to benchstat, it would be useful to be able to compare all of the benchmarks run for two different apm-server builds, and identify statistically significant changes.
The way benchstat works is by:
We could do this by creating two transforms, each grouping on apm-server build and benchmark name, which will:
Then given two apm-server builds, we can use the t_test aggregation for each benchmark/metric combination.
Similar to benchstat, it would be useful to be able to compare all of the benchmarks run for two different apm-server builds, and identify statistically significant changes.
The way benchstat works is by:
We could do this by creating two transforms, each grouping on apm-server build and benchmark name, which will:
Then given two apm-server builds, we can use the t_test aggregation for each benchmark/metric combination.