Add Experiment Definition (Fixes #23)

mozilla / nimbus-shared

Shared data and schemas for Project Nimbus

https://mozilla.github.io/nimbus-shared

Mozilla Public License 2.0

6 stars 15 forks source link

Add Experiment Definition (Fixes #23) #19

Closed k88hudson closed 4 years ago

k88hudson commented 4 years ago

This needs some discussion and is likely to change as we start thinking more broadly about mobile etc. but I think we should define what we think an Experiment will look like so we can stay coordinated across applications. Some notes:

We need to define what subset gets sychronized to client configs v.s. a broader set of data that is available to applications like Pensieve
Although we want to make this practical for right now, we should keep in mind that long term this should not be specific to a particular client (i.e., it should be suitable for Desktop + Mobile) or a particular feature implementation (e.g., Messaging).

Anyways, ready for initial feedback. Thoughts @jaredlockhart @tdsmith @mythmon @piatra?

tdsmith commented 4 years ago

Analysis also needs to know:

the design duration of the enrollment period (probably not the actual duration)
the contrasts to draw; "use branch C as a control branch" and "all pairwise comparisons" are probably the starting set. I guess we could express that as a nullable reference_branch field on the Experiment?

Do you have a use case for experiment_url?

k88hudson commented 4 years ago

Do you have a use case for experiment_url?

This seems useful to be able to look up the complete definition, no?

jaredlockhart commented 4 years ago

@k88hudson Where will we find the values of the Features, Audiences, and Triggers to put into the forms and recipes? For the forms to land this week we'll need those?

jaredlockhart commented 4 years ago

Do you have a use case for experiment_url?

This seems useful to be able to look up the complete definition, no?

Yeah especially if you're looking at the JSON in isolation in some random place and want to refer back to where it came from.

jaredlockhart commented 4 years ago

If this goes out to Firefox Experiment Manager, does it need to conform to this:

https://github.com/mozilla/messaging-system-inflight-assets/blob/master/messaging-experiments.json

to run without any current client changes?

k88hudson commented 4 years ago

Updated with a ExperimentRecipe type that is compatible with ExperimentManager

tdsmith commented 4 years ago

Flagging an earlier comment --

the contrasts to draw; "use branch C as a control branch" and "all pairwise comparisons" are probably the starting set. I guess we could express that as a nullable reference_branch field on the Experiment?

I think this is more flexible than a is_reference_branch flag on the Branch and it's a little easier to consume; how about it?

mythmon commented 4 years ago

Oh no, I'm sorry. This PR got closed because I changed the default branch to main, and this was targeting master. @k88hudson can you change the target branch of this PR? I think we can re-open it then.