search

DK96-OS / MathTools

Mathematical Software Components. This library is actively maintained, and aims to stay updated. New feature proposals are welcome, but may not be included.
Apache License 2.0
2 stars 1 forks source link

Flaky Generator and Statistics Tests #99

Closed DK96-OS closed 1 year ago

DK96-OS commented 2 years ago

How to isolate these tests that occasionally fail due to statistical fluctuations, and use retry mechanics.

DK96-OS commented 1 year ago

This Test assertion failed recently: mathtools.generators.elements.ints.IntRangeElementGeneratedValuesTest.testRangeSize16(IntRangeElementGeneratedValuesTest.java:63)

The "Range Size 16" Test Case creates an IntRangeElement with an independent startValue, and dependent endValue that is 15 greater than startValue, such that there are 16 possible outcomes.

The input that fails is the one with input Parameter of Integer.MAX_VALUE (2147483632).

Aside Note: The duration of this test appears to vary greatly with input startValue, for this particular run, there is a range of 2 - 220 milliseconds.

The failed assertion is on the standard deviation of the generated values. There is an expected value of 10, and a tolerance of 4.5. These values were not determined mathematically, but through observation of many test runs.

This assertion has produced a high rate of failure following upgrade to Kotlin 1.8.0.

This is an indication that the underlying implementation of Random has changed.

DK96-OS commented 1 year ago

Consider that as long as a random number source is used in these tests, the standard deviation will fluctuate.

Recently, in #135 some tests were modified to check that the standard deviation is within some expected range without using an assertion. These tests will print to standard output when the standard deviation is outside of the expected limits. This should reduce the amount of interruption caused by these failing tests, without removing the ability to monitor changes in frequency of random outliers.