Re-run statistical significance for the A/B test once we have a week of data at 50%

[mackhowell-nava]

• One thing we noticed from the first pass at statistical significance was the larger sample size for the control group (see spreadsheet)
• Re-run the statistical significance calculation after 1 week at 50% -- June 25th, 2024.
• See how the data compare.

[mackhowell-nava]

Done -- statistical significance was NOT MET after re-running the math after one week at 50%.

The sample size for the experiment vs. control groups looked more even (half and half), which is what we'd need. From reading more there is an implicit assumption with some of this math that there is a SET sample size for both groups. See this article: https://www.evanmiller.org/how-not-to-run-an-ab-test.html

Updated the spreadsheet here: https://docs.google.com/spreadsheets/d/13_kZX8_byEb-FoF1nJjsfVeucBRWDb0lmNkYjnAwkXk/edit?usp=sharing