search

julianje / Bunny

Hop around the experiment space.
MIT License
6 stars 0 forks source link

Bunny.Hop bug when ensuring a null signal wont' give a false positive #23

Closed julianje closed 9 years ago

julianje commented 9 years ago

Bunny.Hop uses binary search, which assumes that the power function p:SampleSize->[0,1] is increasing.

If you use a null model, the goal is to find a sample size where your test will only detect something less than (1-power)% of the time. So Hop won't work:

Bunny.Hop(Experiment,limit=100,power=0.05) Power: 0.05 Limit: 100 Replications per proposal: 10000 Searching for your sample size... Simulating with 50 participants per condition... Power=0.0577 Simulating with 25 participants per condition... Power=0.0631 Simulating with 13 participants per condition... Power=0.0812 Simulating with 7 participants per condition... Power=0.1178 Simulating with 4 participants per condition... Power=0.1809 Simulating with 2 participants per condition... Power=0.4001 Simulating with 1 participants per condition... Power=0.9058 [1, 0.9]

julianje commented 9 years ago

Can't fix this if we want Bunny to be able to do binary search. Instead, Bunny.Hop() now checks if it converged. If it didn't it suggests running Bunny.Explore() or increasing the sample size.