Weighting does not work as expected.

[robhuzzey]

Expected Behaviour

We were expecting our experiments to reflect the weighting defined in the variantWeights

Current Behaviour

We've been using the weighting:

emitter.defineVariants("My Example", ["A", "B", "C"], [25, 25, 50]);

in our split tests and seeing results of

A = 40% B = 10% C = 20%

Assumptions

Looking over how this works, I can't see how the code would know we had n visitors & know how to weight appropriately so I can only assume it's random.

With that in mind, how does weighting work?

Help needed

Without using a 3rd party service (like mixpanel), what is the correct way to ensure our splits are as close to the weighting as possible?

[wehriam]

I documented the code to provide some insight on what's happening: https://github.com/pushtell/react-ab-test/blob/master/src/Experiment.jsx#L66-L106

For your results, 40% + 10% + 20% = 70%

40 of 70 = 57% 10 of 70 = 14% 20 of 70 = 28%

I'm not sure how many iterations you're running, but you can see the test that confirms the weighted choices within a threshold here: https://github.com/pushtell/react-ab-test/blob/master/test/browser/weighted.test.jsx#L54-L100

[robhuzzey]

Thank you.... I've just looked into this some more & we have markup like this:

<Experiment name='My Example'>
  <Variant name='B'>
    // Some components
  </Variant>
</Experiment>

in the top of our page...

and..

<Experiment name='My Example'>
  <Variant name='A'>
    // Some components
  </Variant>
  <Variant name='B'>
    // Some components
  </Variant>
  <Variant name='C'>
    // Some components
  </Variant>
</Experiment>

in the bottom.

I think it's possible our unusual 70% number is coming from the fact we haven't got Variant's A & C in the markup at the top of the page?

I'm thinking of changing the markup at the top of the page to:

<Experiment name='My Example'>
  <Variant name='A' />
  <Variant name='B'>
    // Some components
  </Variant>
  <Variant name='C' />
</Experiment>

As this should give us the correct 100% number & therefore expected weighting results.

Does that make sense so far?

[wehriam]

You may want to check the documentation on coordinating multiple components.

[robhuzzey]

Thank you... I have looked at that which is why we decided to not use the variants where we did not want them to display.

The issue I am facing is that we don't have 100% after 8854 visits. A = 3606 (40.73%) B = 942 (10.64%) C = 1832 (20.69%)

I wasn't sure as to what was causing that as we've followed the instructions accordingly.

Any help is greatly appreciated :)

[wehriam]

What is being displayed the other 30% of the time?

[robhuzzey]

That's where we are confused... there should never be another 30% of the time? Our weightings add up to 100% emitter.defineVariants("My Example", ["A", "B", "C"], [25, 25, 50]);

[robhuzzey]

@wehriam thanks for all your input on this... I'm thinking I can close this issue if I've mis-understood the way weightings work?

My assumption is that the array sum should equal 100 (relating to 100%) and each entry in the array is the percentage you wish to display so for example a 60/40 split would be [60,40].

We are currently investigating to see if the emitter.addplaylistener is triggering every time as this may be the cause of our missing 30%?

[wehriam]

The array sum does not need to add up to 100. Please refer to the documented code.

[robhuzzey]

@wehriam I am sorry, but I have referred to the documented code several times & it's still not clear to me how it works.

To keep things simple....

We need a way to split A, B, C with a 25%, 25%, 50% respectively.

Can you advise how we would achieve that with weightings please? Or is it not achievable?

[wehriam]

Weighting is achievable, tested, and documented. I would double check the code you're using to measure the results.

[robhuzzey]

@wehriam yes, that's what we are currently doing, just gathering data now on that however if our fundamental understanding of how weighting works is flawed, it's pointless attempting a fix.

Could you advise of what numbers you would add to the weightings array to achieve a 25%, 25%, 50% split for 3 variants please?

According to the documentation & code, it should be [25,25,50] based on my understanding... is my understanding incorrect?

[wehriam]

Yes,[25,25,50] would provide a weighting of 25%, 25%, 50%.

[robhuzzey]

@wehriam thank you... I will close this now as logically is must be something in our implementation that is flawed.