Implement proper statistics

rlee287 / Halite-II-Gym

An enhanced Halite II gym for comparing bots

MIT License

0 stars 1 forks source link

Implement proper statistics #1

Open rlee287 opened 6 years ago

rlee287 commented 6 years ago

Implement proper statistics to help decide which version(s) are better.

[x] Decide on proper distribution to use for significance testing (use binomial model)
[ ] Calculate and display relevant statistics
- [x] Avoid using any external dependencies (installation of the scipy stack is nontrivial for Windows users)
- [x] ~~p-value~~ Confidence intervals
- [ ] Statistical power
- [x] Win ratio of each bot so far

rlee287 commented 6 years ago

Still trying to figure out statistical power, but it should be possible to model the gym results using a binomial process and using something like binom_cdf(result_win, num_trials, 1/num_bots) to calculate the p value.