search

dannycoates / abatar

A/B testing minion for node and browsers
1 stars 2 forks source link

More due diligence on the key hash #3

Open dannycoates opened 9 years ago

dannycoates commented 9 years ago

I think we need to consider experiment versioning and possibly namespacing

eytan commented 9 years ago

It sounds like #4 describes namespaces. What do you mean by "versioning?" As we discuss in the PlanOut paper - changing an experiment definition mid-way through requires that you do some fancy stuff with your data analysis, and so as a best practice it's recommended that developers never change experiments, only launch new ones.

At Facebook, we require that all experiments get created within a namespace, and that when you want to make a change to experiment, you have to "clone" an experiment. When you clone an experiment, the system copies the experiment definition, allocates some random percentage of unallocated segments. Developers can then change their script. Because salts are tied to experiment names, the assignments for the clone will be orthogonal to the old version.