API design and "context"

[dannycoates]

So we've got this notion of "context" that we need to deal with. Experiments need some attribute(s) to make their categorization.

The way to do this currently is to send whatever attributes are relevant in the call for a choice:

// var email = someTextField.value || whatever
if(AB.isEnabled('avatar', email)) {
  var text = AB.choose('avatarText', userId, sessionId, etc)
  // ...
}

Which is pretty nice from the client API side of things. Its clear what's being used to determine the choice and the syntax is nice and lightweight. However it makes it less flexible on the experiment side:

AB.Experiment.create(
  'avatar',
  function isMozillian(email) {  // The experiment is confined to *only* email
    return mozRegex.test(email)
  }
)

If a later experiment wanted to disable avatars based on some other criteria (like user agent version number) both the client and experiment code need to change. It seems like it would be nice if the client code didn't need to change. We could set the context elsewhere...

AB.subject = {
  userAgent: window.navigator.userAgent,
  email: user.email,
  userId: user.id,
  sessionId: sessionId,
  roll: localStorage.get('roll')
}

// meanwhile in some other file

if(AB.isEnabled('avatar')) {
  var text = AB.choose('avatarText')
}

// or more verbosely if AB.subject isn't set or we want to override an attribute

AB.isEnabled('avatar', { email: 'me@mozilla.com' })

Now an experiment has access to all of the subject attributes to work with:

AB.Experiment.create(
  'avatar',
  function (subject) {
    return isBeta(subject.userAgent)) || mozRegex.text(subject.email)
  }
)

The experiment API is a bit more verbose because we have to go through subject but it doesn't seem too much of a hassle.

I like the flexibility of the AB.subject approach, but is it necessary or meaningfully better?

f? @shane-tomlinson @kparlante

//updated to match #2

[rfk]

FWIW, I also the more flexible coupling of the AB.subject approach.

I guess this means we send a little bundle of user data to server every time we check for an experiment. Is there any risk of this becoming creepy without user consent, or is it all just stuff that's already on the wire anyway?

[dannycoates]

I guess this means we send a little bundle of user data to server

In fact, No! :smile:

The experiment code is run on the client so no user data needs to get sent to the server, however to get useful statistics some experiments might need to report more that just which choice was selected. (now I'm wondering if that would ever be the case?)

[rfk]

strongly :+1:

[eytan]

Hi guys, I recently stumbled across your project -- this is great stuff! I have recently been thinking about how we could improve client-side experimentation with PlanOut (particularly on mobile), so I'm curious to see how things progress. Would you guys mind if I provided some unsolicited feedback every so often?

FWIW, there is a short blog post that explains how Facebook currently does client-side testing and logging on mobile: https://code.facebook.com/posts/520580318041111/airlock-facebook-s-mobile-a-b-testing-framework/.

[dannycoates]

Hi @eytan! Thanks for your comments. Please keep them coming :)

PlanOut has been a huge inspiration. My goal with this project right now is to explore many of those ideas and learn ways to introduce them into our apps.

[eytan]

Passing in the context makes a lot of sense! You will definitely want to use "exposure logging" so that you only consider metrics for users who are in your experiment, and that requires calling home at some point in time with some kind of unique identifier. A nice thing about the client-side experimentation is that it lets you determine whether a user is eligible to be in the experiment without sending any user data over the network.

What is the motivation for doing the assignment logic client side, rather than server side (e.g., sending async requests to a parameter store endpoint with the subject blob and parameter namespace)? Cons for client-side include being much heavier weight as far as data you need to send down the wire (assignment code vs user blob), and less straightforward centralization, making it harder to deploy experiments via namespaces and test experiments. For example, with PlanOut, one could associate each (user account, namespace) pair with an overrides blob on the server side, so that you could make the user always be assigned to certain parameter values or look like they are on some version of the browser. This makes it a lot easier to test your experiments. FWIW, I recently added a bunch of new documentation on how overrides work with PlanOut at http://facebook.github.io/planout/docs/testing.html

One in between solution would be to serialize your experiment definitions (i.e., via PlanOut), and when users make a request to a namespace, it passes back the serialization associated with the user, as well as the overrides. This way all of the assignment could happen client-side and you'd still be able to have easy testability.