search

balena-io-modules / analytics-client

Client part of analytics services used at balena
5 stars 0 forks source link

Extend Experiment functionality to accept A/B test configuration #32

Open gelbal opened 3 years ago

gelbal commented 3 years ago

Currently the components using analytics-client need to type code to define each experiment's configuration as seen in the UI repository. We would like analytics-client to eliminate such cumbersome and redundant work. Instead, analytics-client could accept configurations to setup the experiments and expose convenient methods to manage them.

The provided configuration would define the experiment name and cohorts (name, % distribution). Potentially analytics-client could support other constraints on which user is qualified for a cohort / experiment (e.g. using a specific browser, having a certain user property, etc).

gelbal commented 3 years ago

We could consider integrating open source A/B frameworks to analytics-client not to repeat effort. At first search I found this library that looks promising: https://github.com/appannie/ab-testing

laziob commented 3 years ago

I still havent give this a in depth look, but to take into account there are also this other open source tools: Plan Out (by Facebook) : https://github.com/rawls238/PlanOut.js or https://facebook.github.io/planout/ Wasabi (by Intuit though it seems to be outdated but functional): https://github.com/intuit/wasabi Mojito: https://mojito.mx/docs/home

All this assuming we go for a build instead of buy route. If for some reason we look into paid services, I recommend Conductrics: https://conductrics.com/ I "know" the founder, we shared conversations regarding testing methods and the like. He knows tons about statistics, ML and really advance stuff which tries to knowledgly put on its platform. Can give better support related to testing than any other Account Rep of the big enterprise tools I've ever got to talk to. Also has server side testing which is nice for the long term.

Analytics-toolkit: https://www.analytics-toolkit.com/ The founder is also the author of a book an Online AB testing, and also shares lots of knowledge about AB testing on the site blog. I've also shared some conversations about testing with him. His platform is not as feature heavy as Conductrics I believe, but has some really interesting option for doing frequentist testing without fixed sample sizes which other tools dont have.

gelbal commented 3 years ago

Thanks for sharing these Ezequiel. As we start working on this task, it would be good to look at our options and understand the trade-offs.

gelbal commented 3 years ago

Additionally, analytics-client should give clear instructions on how to handle A/B experiments in various scenarios, including the analytics opt-out users. We already experienced a bug where the opt-out users didn't receive a Etcher Success banner. See FD thread.