search

gregdingle / genetify

An unobtrusive way to A/B test and optimize webpages
http://genetify.com
MIT License
370 stars 44 forks source link

Weight hogger #4

Open tczupryna opened 13 years ago

tczupryna commented 13 years ago

I ran a test to check out how genetify will work for me and ran into a bit of a problem that may prevent me from being able to use it more extensively. Scenario:

1 test with original and 'empty' variables. 'empty' is just hiding the section. One goal worth 144. My expected conversion on the page is around 1%.

What I saw was that the first variable to record a goal (original) started hogging all of the impressions right away. It took quite some time to give the second variable enough impressions to see that the results are essentially equal. The original is still getting the majority of the impressions even though the avg. of empty is about the same.

Please visit http://www.naymz.com and pull up the console to see the results of 'booster'.

My initial thought was that it would be nice to set a minimum number of impressions before weight is considered. Ideally this could be set on the test level somehow.

I'm sure I explained this clear as mud. Let me know what I'm missing.

gregdingle commented 13 years ago

I've had the same issue with some tests. Try modifying the weight function to make it optimize slower. Look at genetify.weight. https://github.com/gregdingle/genetify/blob/master/genetify/genetify.js#L1233