search

assaf / vanity

Experiment Driven Development for Ruby
http://vanity.labnotes.org
MIT License
1.55k stars 269 forks source link

Progressive tests #355

Open nmilan opened 5 years ago

nmilan commented 5 years ago

Hello there, we have tried to set up progressive tests and it seems it's not possible? So what we need to do it's to start two experiments at the same time with the same alternative. For example: One experiment is: landing_page_cvr(alternatives: a,b), Second experiment is: landing_page_cta(alternatives: a,b) We should start both in the same time and it should be same alternative(for example a for both)

What we tried so far: 1) We use Vanity.playground.experiments[:experiment_name].chooses(alternative) and test start with chosen alternative. But when we have to finish that test like this Vanity.track!(:experiment_name) that didn't work. 2) We try to start the second test and after that, we try to update VanityParticipants.update_columns(seen: alternative_index) 3) Also, we try to add a variable in an experiment for alternatives.

We think the best solution will be if we can somehow pass alternative as a variable(ex: Vanity.ab_test(:experiment_name, :alternative)) on the starting experiment. Do you have any solutions for this?

phillbaker commented 5 years ago

Hi @nmilan and the others who +1'ed this - thanks for opening an issue. I'm not totally sure I follow the example, but do any of the following seem related?