chadadavis
commented
4 years ago Very nice article. Much emphasis is placed on the difference between areas of application of hypothesis tests, eg human drug trials versus eg a new discount system on an e-commerce platform. Granted, the costs of false positives and false negatives are assumed lower in the latter context, but that doesn't always make the difference negligible enough to assume that two small numbers are equivalent. In my experience, the source for the asymmetric cost is human bias, more than the area of application. From the article: "... so false negatives only delay discovery, rather than eliminating it." Experimenters often believe results that validate their goals, and rarely are incentivised to question their own conclusions after a successful experiment. However, results that don't (yet) validate their goals often result in the experimenters running followup experiments. I think this depends on social dynamics between experimenters that one is working with.
Following the discussion in this article: https://geoffruddock.com/ab-testing-with-a-symmetric-risk-profile/ it might be worthwhile to set the default alpha and beta to the same value (e.g. 10%).