search

cfpb / django-flags

Feature flags for Django projects
https://cfpb.github.io/django-flags/
Creative Commons Zero v1.0 Universal
256 stars 31 forks source link

[QUESTION] How do I set up a simple A/B-test? #75

Closed DataGreed closed 3 years ago

DataGreed commented 3 years ago

The documentation mentions that A/B tests can be ran using this package, but the documentation does not suggest any way to do it.

How do I set up a simple a/b test? with 50/50% distribution of a boolean value (50% of users get True and 50% get False)?

Scotchester commented 3 years ago

Hi @DataGreed, thanks for the question! I'm sorry to say that that statement in the docs may be a bit misleading.

Django flags can be used to turn things on and off in a template based on which group users are in, but it doesn't have a built-in mechanism for actually doing the segmenting of users into groups. Something else in your back-end code would need to do that, particularly if it's important to ensure that the groups are as even as possible – you'd probably need to keep a record of how many people have seen each and alter your segmentation based on that.

If you're willing to have the balance of segmentation not be perfect, I would forgo flags and just implement a coin-flip sort of check in your template. Heads, the user gets option A, tails the user gets option B. Given enough data, it would get close to 50/50.

This actually got us thinking. though, that a randomizer flag condition could be useful. Then if you're already using flags, you don't need to write your own randomizer. It could maybe even have a parameter for what percentage of the time to activate it, vs. assuming 50/50. What do you think about that?

DataGreed commented 3 years ago

Hi, that's a really good idea! Seems like django packages are lacking on a simple solution that will help to bootstrap a/b testing quickly.

Although, I have already implemented my own tailored A/B tests solution for my project, so I won't be able to contribute 🤷