search

tonyhb / require-abtest

Split testing for RequireJS
28 stars 1 forks source link

Expiring A/B tests #4

Open tonyhb opened 10 years ago

tonyhb commented 10 years ago

How do we expire A/B test cookies so that we can create another test?

Say we're testing /views/page.js via RequireJS. We run our tests like so:

define(['test!/views/page.js'], function(page) {
    // ...
});

If we stop running that test and want to run another test on page.js, the user will still have the cookie for the first test's cohort. We need a way to manage and expire tests.

tonyhb commented 10 years ago

A potential solution:

  1. Track the date (yyyy-mm-dd) of when a user is assigned a variation in the cookie.
  2. Add a "start date" to test definitions

If the start date is after the user's current variation assignment we can assume that we're running a new test and need to re-assign a variation.

tonyhb commented 10 years ago

After thinking about the solution above, it's a great first step but doesn't solve expiring A/B tests. This question should really focus on:

Regarding question two, we're doing tracking entirely in JS. The example tracking package implements Google Analytics and GA Experiments. Expiring A/B tests after significant results just might not work.