AB testing not evenly distributed

[phungleson]

Hi guys,

Recently we upgraded the our 2-year old version of vanity to latest version. We find that Vanity assign majority (90%) of traffic to first alternative (or default if we set default).

One to note though, we are lazy and we don't set metrics, we re-use the same metric for multiple experiments.

Everything works normally if we revert back to old version. I know it is hard to tell from you guys but is there any case that it would happen, any probability calculation can lead to this issue?

It baffled us for last few weeks.

Thanks in advance, Son.

[phillbaker]

Hi there! Sorry to hear you're having trouble. A bit more information would help to diagnosis this:

• What version of vanity were you on and to what version did you upgrade?
• What storage engine were you using?
• Are you registering participants with JS or server side?
• Can you post your (annotated) experiment and metric files?

[phungleson]

Hi @phillbaker

We are using vanity (2.2.7), the old one is around this sha https://github.com/assaf/vanity/commit/36f46b4e552ffa410f8ded05cc7445d426a83283

We are using aws redis, redis_version:2.8.24

We are running unicorn (5.0.1) and has this

after_fork do |server, worker|
  defined?(Vanity) and
    Vanity.playground.establish_connection
end

We register participants on server side, we tried both use_vanity and use_vanity :current_user

Some experiments we are running look like this

engine_aa.rb

ab_test 'Engine' do
  description 'Impact of engine on clicks'
  alternatives 0, 4
  metrics :clicks
end

engine_bb.rb

ab_test 'Engine' do
  description 'Impact of engine on clicks'
  alternatives 4, 6
  metrics :clicks
end

metrics/clicks.rb

metric "Clicks" do
  description "Measure clicks"
end

One thing we found yesterday is that

Vanity.playground.participant_info(Vanity.context.vanity_identity) doesn't contain the experiment we are expecting.

Thanks for looking into this, Son.

[phillbaker]

Thanks, that's helpful. A few more questions may help clear this up.

• what were you passing to use_vanity in the original version?
• can you reproduce the problem locally in development or only in production?
• The Vanity.playground.participant_info(Vanity.context.vanity_identity) missing info is a good clue - what did it return in the original version?
• It looks like that version is 1.9.1, which is right before the breaking changes in 2.0. Were you able to check the release notes for 2.0 and the further versions to see if any changes in integration needed to be made?

[phungleson]

Thanks @phillbaker looks like we found the issue, looks like in newer version of vanity, you guys serve the default option to bots.

[phillbaker]

Ah, yup. Any way we could make this clearer or or explicit in the docs? Sounds like it was a bit unexpected.