search

Matt52 / bayesian-testing

Bayesian A/B testing
MIT License
73 stars 8 forks source link

Minimum sample size #16

Closed abrunner94 closed 2 years ago

abrunner94 commented 2 years ago

First, this package is great! I wanted to know if the probability estimates rely on a minimum sample size or how one might go about determining minimum sample size for a Binary test, for example.

Matt52 commented 2 years ago

Hey, thank you, I am happy you like the package! Probability estimation does not rely on any explicit sample size parameter and this package is mainly about test evaluation. However, the result probabilities are affected by provided data (and its sizes). So if your data is just a few observation long, then the uncertainty will be reflected in the output probabilities. There are more methods to determine minimum sample size so you will have to do some research and pick one that fits your case. :)

Matt52 commented 2 years ago

BTW I know that many people do not calculate minimum sample size (when using Bayesian AB testing), and use custom stopping rules based on probabilities or expected loss values.

abrunner94 commented 2 years ago

Thanks! I'll check out this approach for calculating loss

Matt52 commented 2 years ago

Looks like a good example. 👍 I plan to add expected loss to this package too (when I am bored a bit more in my free time).

Matt52 commented 2 years ago

@abrunner94 I just added expected loss into evaluation output. Check out new release v0.3.0.

I know it is not exactly addressing the original question about sample size but as I mentioned, it is not in the scope for this package.