search

zalando / expan

Open-source Python library for statistical analysis of randomised control trials (A/B tests)
MIT License
335 stars 50 forks source link

Fix early stopping, by making it stricter #242

Open aaron-mcdaid-zalando opened 6 years ago

aaron-mcdaid-zalando commented 6 years ago

The implementation of "alpha spending" increased our Type I error rate from 5% to approximately 10%, as it didn't spend alpha 'piece-wise' but instead it spent it 'cumulatively'

Therefore, this PR makes the early stopping stricter by an arbitrary factor of 100. As a result, we will get an 'extra Type I error rate' of approximately 0.05% instead of 5%.

Then, the total Type I error rate will be 5.05% (= 5%+0.05%) instead of 10% (=5%+5%)

coveralls commented 6 years ago

Coverage Status

Coverage increased (+0.03%) to 92.412% when pulling a6e2523fad44aad6c57b72f12a7be9714030f472 on fix-early-stopping.Solution2 into 5fa3dd1a0833530fe6bf659544e98a0b9fe9f9b9 on master.

aaron-mcdaid-zalando commented 6 years ago

In practice though, we are unlikely to use this internally without further investigation of the implications. In the last few minutes we had a meeting about this option, and other options, for proceeding

Therefore, we won't immediately merge this