search

amazon-ion / ion-java-benchmark-cli

Apache License 2.0
7 stars 9 forks source link

Update the performance regression detection method to reduce false positives. #58

Closed linlin-s closed 1 year ago

linlin-s commented 1 year ago

Issue #, if available:

#53

Description of changes:

This PR optimized the method of detecting performance regression by making these changes:

  1. Removing the logic of calculating threshold value of regression detection. Instead, using two-sample t-test to compare the benchmark results before and after changes to see if regression happened. More details about how to use t-test method to detect regression is included in the comments of source code.
  2. Adding logic to remove outliers from raw data. Two-sample t-test requires normally distributed data. Given that some of our raw data deviates from this due to noise, we preprocess using the IQR (Interquartile range) method to filter out outliers. After this preprocessing, we verify data is normally distributed by using Shapiro–Wilk test.
  3. Adding unit tests for methods removeOutliers and detectRegression.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.