Should we be testing randomness?

[Tavio]

The randomLongDistributionTest test case in RandomFunctionTest.java generates 10 random longs between 85 and 86 and then expects at least one of the generated longs to equal 85 and at least one of the generated longs to equal 86. This is the equivalent of flipping a coin ten times and expecting at least one of the results to have been a head and at least one of the results to have been a tail.

This experiment has a quite small chance of failing, which can be calculated as the chance of all results being tails or all results being heads, that is (0.5 ^ 10) + (0.5 ^ 10) =~ 0.002 = 0.2%

What this means is that the randomLongDistributionTest test case fails about once every 500 times it runs. It failed, for instance, during this test run in Travis: https://travis-ci.org/six2six/fixture-factory/jobs/45108080. This is not good, as it may lead developers to believe the code they are adding to the repository is broken when it may not be.

I think it should be safe to assume that the randomness of Math.random() is correct. Seems to me that this test is testing things it should not be. What do you guys think?

[nykolaslima]

I agree with you. The test should not have chances to fail, even if they are quite small.

I believe that instead of test if the number is equal, we can test the range. What do you thing @Tavio?

[ahirata]

Yea, that's not good. I'm gonna remove that. Thanks!

[nykolaslima]

Thank you @ahirata