A/B test framework for app features

[jace]

Baseframe should provide an A/B test framework for new app features. In its simplest form, the app initalizes Baseframe with a list of feature flags currently being tested:

baseframe.init_app(app, ab=['feature1', 'feature2'])

For a high impact experimental feature, the app may want to limit the test to, say, 1 in 10 users:

baseframe.init_app(app, ab=[('feature1', 10), 'feature2'])

Baseframe then adds a before_request handler that examines session['ab'], creating a dictionary if required and adding all missing flags with a value of None (not part of A/B test), False (A group) or True (B group):

for flag, chance in self.ab_flags:  # Default chance is 1
    if flag not in session['ab']:
        session['ab'][flag] = random.choice([True, False]) if randint(1, chance) == 1 else None

An after_request handler runs through session['ab'] again and removes all keys that are not in self.ab_choices, to remove flags that the app is no longer testing.

Within the app, any transaction that is based on the feature can read the A/B flag status from session['ab'][flag]. It is not possible to read A/B flags from the client-side unless flags are stored in a separate, unencrypted (but still signed) non-HTTPOnly cookie.

[jace]

Ideally, Baseframe should take this further, also providing a mechanism to log events and linking A/B flags to event outcomes.