search

haf / expecto

A smooth testing lib for F#. APIs made for humans! Strong testing methodologies for everyone!
Apache License 2.0
663 stars 96 forks source link

[WIP] Add isFasterThanMedian using Mann_Whitney U Test #302

Closed AnthonyLloyd closed 5 years ago

AnthonyLloyd commented 5 years ago
AnthonyLloyd commented 5 years ago

This looks to be a dead end.

The test doesn't do well when comparing the same function. It goes a random way before the point it would call them equal. Also it is not equivalent to either the median or the mean. This can lead to results that are not consistent.

The equal stopping criteria has been improved though so I will keep that.

Possibly could use the median instead of the mean but it's statistically not correct and also more involved to implement.

Using median and MAD to eliminate outliers (http://anthonylloyd.github.io/blog/2016/10/21/MAD-Outliers) could be done. This could improve the resolution when differences are small but again more involved to implement.

Actually the outlier code can be copied in easily. I'll give it a try.

AnthonyLloyd commented 5 years ago

Median and MAD would be a lot of processing and it's not clear it would fit with the stopping critieria.

I've gone instead for using a s.d. test for outliers and keeping if they are within 99.99%. The code for this is simple and fits well so is fast.

In master now so closing this.