search

wingify / vwo-java-sdk

VWO Java SDK for server-side A/B Testing
https://developers.vwo.com/docs/fullstack-overview
Apache License 2.0
9 stars 4 forks source link

Whether vwo experiment can keep consistent user experience when change experiment group ratio? #10

Closed laocu closed 1 year ago

laocu commented 1 year ago

I read the SDK code and found that the traffic ratio uses the 'weight' field to identify the traffic ratio of group. User experience may change when change the exp group ratio during experiment in operation. For excample: iteration1 control:weight 10.0000 treatment:weight 10.0000 iteration2 control:weight 20.0000 treatment:weight 20.0000 The treatment group users in iteration 1 became the control group in iteration 2.

Do the VWO experiment have this problem？

rohitesh-wingify commented 1 year ago

Hey lacou,

Appreciate your going through the codebase and raising an issue here.

Your hypothesis is correct, but there is more to it. Let me explain.

Yes, it is possible for a user who was part of treatment group in iteration 1, to became part of control group in iteration 2, when the treatment/control weight is changed. We do guarantee that a user will be part of the same group everytime, as long as variation traffic distribution (control/treatment group weights) are not changed.

However, to stop this from happening, you can use a UserStorage with our product, and it stores the variation that a user was made a part of. That way, in iteration 2, we will find the user was part of treatment group, and return that instead.

Does that answer your query?

laocu commented 1 year ago

Thank you for your patient reply. I will learn this function.