Finish a/b test - Githubissues

assaf / vanity

Experiment Driven Development for Ruby

http://vanity.labnotes.org

MIT License

1.55k stars 269 forks source link

Finish a/b test #345

Open edenisn opened 6 years ago

edenisn commented 6 years ago

When we choose winner of a/b test and then click on link "make permanent" we confirm with, for example "Finish this experiment and assign all current and future participants to option B?". What I need to do if I don't want to assign all future participants to option B? I just want to close test without assign participants any more.

phillbaker commented 6 years ago

@edenisn if you want to end an experiment, by definition you're reducing the number of options to just one - users have to go into an alternative. I suppose the experiment file can also be deleted.

I'm not sure I understand the request?

edenisn commented 6 years ago

@phillbaker we don't want to delete experiment. We just want to finish the experiment and so that the number of participants to the winner option no longer increases.

phillbaker commented 6 years ago

Hm, still a little confused as to the desired behavior for users currently in the experiment and users entering the experiment. Can you clarify the behavior below:

While experiment is active:
- New users randomly assigned to alternatives (or assigned based on the strategy of the experiment)
- Existing users continue to see the alternative they were shown
When experiment is "finished" as requested in this issue:
- New users ___ ?
- Existing users continue to see the alternative they where shown?

edenisn commented 6 years ago

When experiment is "finished" as requested in this issue:

New users still assign and increases option B. The number of participants still growing (If option B is a winner of experiment)
Existing users continue to see the alternative they where shown? - Existing users continue to see option B (If option B is a winner of experiment)

phillbaker commented 6 years ago

Ah, I think this may be a dupe of https://github.com/assaf/vanity/issues/195 ?

edenisn commented 6 years ago

Yes, quite right

edenisn commented 6 years ago

In my opinion the best way out of this situation is to bring the statistics below the test at its end. For example:

Option A - 16 participants, 8 converted, 34% Option B - 34 participants, 27 converted, 76%

Thus, we fix the statistical data at the end of the test

phillbaker commented 6 years ago

to bring the statistics below the test at its end

Can you explain this? I don't understand the suggestion.

I think we might be able to update the logic here:

https://github.com/assaf/vanity/blob/6e532234678aa90e828f9c1c9afbc5b06bbfb178/lib/vanity/experiment/ab_test.rb#L189-L207

to include whether the experiment has an outcome, and if so to not save additional choices to the DB. How does that sound?

Alternatively, experiments has a completed_at timestamp and participants have a created_at timestamp. Particpants added after the completed at could be ignored for statistics purposes.

edenisn commented 6 years ago

I mean that we simply can add more statistical information to /admin/vanity page when experiment finished. Add information with participants, converted and % at the end of the experiment.

By the way, your suggestion is very good. It would be fine if we don't save info with additional choices to DB after finished experiment.

Thanks @phillbaker