dannycoates commented 9 years ago

Experiment lifecycle

This is conceptually what an experiment lifecycle might look like (written in js just for fun)

Ideal scenario

This scenario demonstrates what happens when everything goes smoothly.

Definition

We define the parameters of the experiment and push it through a normal development cycle eventually being deployed to production.

parameters

hypothesis: a human description of the experiment
startDate: when the experiment should begin (ISO 8601 format)
subjectAttributes: attributes of the candidate test subject that will be used for determining eligibility and/or test group
independentVariables: the application configuration variables that will be controlled by this experiment
eligibilityFunction: a function that determines if a candidate should participate in this experiment or not
groupingFunction: maps participants to their values of independentVariables

module.exports = {
  hypothesis: 'a niceCaptcha does not negatively affect signups',
  startDate: '2014-12-15',
  subjectAttributes: ['sessionId'],
  independentVariables: ['niceCaptchaEnabled'],
  eligibilityFunction: function (subject) {
    // a random sampling of 10% of sessions will be in the experiment
    return this.bernoulliTrial(0.1, subject.sessionId)
  },
  groupingFunction: function (subject) {
    // 50% of participants will see a niceCaptcha the rest are the control group
    return this.bernoulliTrial(0.5, subject.sessionId)
  }
}

Analysis

Data analysis is outside of scope right now. Perhaps it can be defined here and automated at some point.

Conclusion

Analysis confirmed the hypothesis. The corresponding value for 'niceCaptchaEnabled' is set as the conclusion.

parameters

conclusion: the "winning" value(s) of the independentVariables.

module.exports = {
  hypothesis: 'a niceCaptcha does not negatively affect signups',
  conclusion: true,
  startDate: '2014-12-15',
  subjectAttributes: ['sessionId'],
  independentVariables: ['niceCaptchaEnabled'],
  eligibilityFunction: function (subject) {
    // a random sampling of 10% of traffic will be in the experiment
    return this.bernoulliTrial(0.1, subject.sessionId)
  },
  groupingFunction: function (subject) {
    // 50% of participants will see a niceCaptcha the rest are the control group
    return this.bernoulliTrial(0.5, subject.sessionId)
  }
}

General Release

Participants in the experimental group maintain their value (it won), while for everyone else (including the control group) conclusion becomes their default value according to the release schedule.

The default release schedule has a startDate and endDate where the pecentage of subjects having the conclusion is linear with where the current time is in that interval. For example, with a startDate of '2015-01-15' and endDate of '2015-01-16', at '2015-01-15T18:00' 75% of the population will have the conclusion.

parameters

release.startDate: when to begin releasing the conclusion to the general population
release.endDate: when 100% of the population should have the conclusion

module.exports = {
  hypothesis: 'a niceCaptcha does not negatively affect signups',
  conclusion: true,
  startDate: '2014-12-15',
  subjectAttributes: ['sessionId'],
  independentVariables: ['niceCaptchaEnabled'],
  eligibilityFunction: function (subject) {
    // a random sampling of 10% of traffic will be in the experiment
    return this.bernoulliTrial(0.1, subject.sessionId)
  },
  groupingFunction: function (subject) {
    // 50% of participants will see a niceCaptcha the rest are the control group
    return this.bernoulliTrial(0.5, subject.sessionId)
  }
  release: {
    startDate: '2015-01-05',
    endDate: '2015-01-07'
  }
}

Post Release

After release the default value should be changed in the configuration to match the conclusion. The experiment can either be deleted or kept for posterity.

ckarlof commented 9 years ago

Great start @dannycoates! We should also think about the process of transferring successful experiments to "that's just how the code works now". We don't a bunch of old experiments littering the codebase.

dannycoates commented 9 years ago

I've made a few changes as I've been implementing...

module.exports = {
  name: 'niceCaptchaAB',
  hypothesis: 'a niceCaptcha does not negatively affect signups',
  startDate: '2014-12-15',
  conclusion: {
    niceCaptchaEnabled: true
  },
  release: {
    startDate: '2015-01-05',
    endDate: '2015-01-07'
  },

  subjectAttributes: {
    sessionId: 'required'
  },
  independentVariables: {
    niceCaptchaEnabled: false
  },
  eligibilityFunction: function (subject) {
    // a random sampling of 10% of traffic will be in the experiment
    return this.bernoulliTrial(0.1, subject.sessionId)
  },
  groupingFunction: function (subject) {
    // 50% of participants will see a niceCaptcha the rest are the control group
    return this.bernoulliTrial(0.5, subject.sessionId)
  }
}

name was added as an identifier. Maybe this is a temporary implementation detail or maybe its useful to humans as well.

conclusion is an object in order to make it more clear what variables are being set. Its now easier to read just the hypothesis and conclusion to understand the experiment.

subjectAttributes became an object with the key as name and value as something else, possibly a validation schema along the lines of https://github.com/hapijs/joi but for now 'required' is the only option. This is the least thought out change.

independentVariables is also an object with the key as name and value as the default value for ineligible subjects.

dannycoates commented 9 years ago

For simple rollout of a feature:

{
  conclusion: {
    newFeatureEnabled: true
  },
  release: {
    startDate: '2015-01-05',
    endDate: '2015-01-06'
  }
}

It might be nice to have some synonyms for different use cases to make it more intuitive, conclusion looks a bit strange in this context.

dannycoates / abatar