search

Matt52 / bayesian-testing

Bayesian A/B testing
MIT License
72 stars 9 forks source link

Credible intervals #37

Closed komatsuna4747 closed 1 week ago

komatsuna4747 commented 8 months ago

Thank you for the excellent package! I was wondering if this package could provide credible intervals for each variant when reporting A/B testing results.

Matt52 commented 8 months ago

Hi @komatsuna4747, I am happy you like the package! To be honest I was not really planning to add credible intervals but perhaps it is something that could be very simply added. I can see I could easily just use sampled values from posterior distributions (which already exist anyway for PBB and loss estimations) and estimate credible intervals based on empirical quantiles. Alternatively I could also just return the definition/parameters of the posterior distributions which could give you more flexibility.

Let me know what would you consider as a better option. I am also happy to look at the pull request in case you would like to contribute yourself. :)

Matt52 commented 1 week ago

New version v0.8.0 now contains credible intervals in the evaluate result. It is also possible to set any desired probability alpha (the default one is 95%).