Development Research in Practice: The DIME Analytics Data Handbook. By Kristoffer Bjärkefur, Luíza Cardoso de Andrade, Benjamin Daniels, and Maria Jones
Think of an intuitive way to explain that most studies are small. Cannot trust that law of large numbers apply. More importantly, we no longer need to rely on law of large numbers because we have the computing power. Every command can now bootstrap and redraw.
I think we have a pretty poor idea of what is "large" and "small" for a lot of designs and distributions. My roughest guide is "large" kicks in around 10,000 randomization clusters, and small is anything less -- the real definition would be the limit around which the mean significant result is close to the true treatment effect. It's 10,000 obs if the treatment effect estimator is normally distributed given certain assumptions about the size of the treatment relative to the noise in the population...
bbdaniels
commented
4 years ago
Think of an intuitive way to explain that most studies are small. Cannot trust that law of large numbers apply. More importantly, we no longer need to rely on law of large numbers because we have the computing power. Every command can now bootstrap and redraw.
I think we have a pretty poor idea of what is "large" and "small" for a lot of designs and distributions. My roughest guide is "large" kicks in around 10,000 randomization clusters, and small is anything less -- the real definition would be the limit around which the mean significant result is close to the true treatment effect. It's 10,000 obs if the treatment effect estimator is normally distributed given certain assumptions about the size of the treatment relative to the noise in the population...
Originally posted by @bbdaniels in https://github.com/worldbank/d4di/diffs