YahooArchive / mendel

A build toolchain for experimentation on isomorphic web applications with tree-inheritance and multivariate support.
MIT License
88 stars 25 forks source link

Assigning experiments #141

Closed skbolton closed 6 years ago

skbolton commented 7 years ago

I am intrigued by this project and it could be an awesome fit for me. Is the idea that mendel would generate multiple bundles and I can then deal with the logic of what bundle gets served to which user or is that figured into the library too?

irae commented 7 years ago

To the point:

Building different versions of bundles correctly, serving them in production very fast without incurring extra payload or other performance drawbacks are in itself a gigantic task. Mendel is focused on this task.

The task of picking each bundle is served to each user is not part of Mendel itself, not because we couldn't, but because having this part pluggable is more flexible and a better architecture. You can use Facebook's Planout for this task, and we have a working example with mendel+planout here:

https://github.com/yahoo/mendel/tree/master/examples/planout-example

Please keep in mind that this is a multi layer example, the most complex we could have for a planout use. Planout implementation can be simpler than that if you so choose.

Also interesting to note:

Mendel is also architected in a way that enforces you do do different bundles "the right way" without creating maintainability problems and completely abstracting away the idea that different users will have different runtime code paths. Instead, your code should look, feel and be written as only one variation or codepath existed - only directory structure is different for variations, which is great even before you have to serve different bundles to actual users.

Currently I work on a small startup and we don't do A/B testing yet, so our implementation of "which bundle to serve" is only based on query string. We do http://ourexmapleapp.com/ourroute?variations=expandable-cards (where expandable cards is something we are working internally and not ready for our users) and we can have a better development cycle anyway, avoiding long living branches.

You can see how this is implemented in other example (it is just simple as that):

https://github.com/yahoo/mendel/blob/master/examples/full-example/app.js#L27-L33

irae commented 7 years ago

I updated the above comment to include a link to Multilayer and Multivariate test explanation on Mendel

https://github.com/yahoo/mendel/blob/master/docs/Design.md#multivariate-and-multi-layer-testing