A shiny app for analyzing AB tests
This shiny app is an example of an A/B test dashboard that uses both frequentist and bayesian methods.
The underlying data it uses is simulated to be "realistic", such that it could process production-level data.
Additionally, the underlying code/functions/logic is unit-tested and is designed to be extendable and maintainable.
This app uses "attribution windows" for all metrics, which means it only counts conversions toward each metric within a certain amount of days from the time the user joins the experiment. Here is an exploratory blog-post that shows the possible effects of using (or not using) attribution windows.
The simulated experiment traffic (and the sample size calculator) assumes that returning visitors will be included in experiments.
One of the datasets require is experiment-traffic which is used to calculate the bayesian prior dataset and prior alpha/beta. Rather than requiring that in the application, it could be calculated in a datamart, then the application would only require a much smaller bayesian prior dataset that was per experiment/metric.