Show conversion rates for multiple metrics

[rsslldnphy]

Vanity appears to support collecting multiple metrics for each test, but when viewing the experiments on the dashboard conversion is shown only for one of them (or the union of them?). Is there a way to show/get conversion % for each experiment for each individual metric?

To give a concrete example, we're testing what effect a given button text has on people clicking on it, but we also want to test the effect that text has on the number of people who then go on to buy. At the moment, the clicks show as the "conversion %" on the dashboard but there's no way I can see to see the % of sale conversions there.

[phillbaker]

@rsslldnphy great questions.

I think there's some history here. It started back in 1.4: http://vanity.labnotes.org/experimental.html. There's also some support for better collection in PRs: https://github.com/assaf/vanity/pull/121 has work for multiple metrics, but it's diverged quite a bit. (Separate work to cherry pick just that bit is here: https://github.com/ViliusLuneckas/vanity/commit/2d6fae16a6359b0d2672b4da47cbf4e5868044cf.)

I'd have to look at the code to figure out whether just one metric or the union is being used for conversion. Regardless of what's done now, it makes sense to me to show conversion for each metric explicitly.

[rsslldnphy]

Oh, are the changes described in "experimental.html" separate to defining multiple simple metrics for a test like this - metrics :clickout, :sale - in the experiment definition, or is it part of the same thing?

Will have a look at the code myself and see if there's anything I can contribute. Thanks for the quick response!

[sdhull]

Sorry to revive such an old thread but having better insight into multiple metrics per experiment would be extremely useful.

It sounds like you can specify multiple metrics already, but it's not clear what is displayed on the dashboard when you do, or how that affects the calculations of which alternative is performing better.

In a funnel (as described above, "How many click, but then how many go on to purchase?"), you typically care more about the later metric, so maybe that should determine the winning alternative, however conversions for later steps in the funnel are typically more scarce, so it might take longer for experiments to converge. So maybe it would just be good to have some insight...

EDIT: I've dug through the code and it looks like for each metric, when there's a conversion, it adds that participant id to a set of "converted" for that experiment, and increments a "conversions" counter, regardless of which metric it is. For our use-case, that's OK, but I can imagine other people having a problem with that strategy (eg, you're testing images for a particular product and you have metric for adding it to the cart, and another metric for checkout, you wouldn't want people being counted as a conversion for your experiment if they didn't add that product to their cart prior to checking out).

I'll work on preparing a PR to (additionally) track conversions for experiments per metric_id, and maybe update the dashboard as well.

[phillbaker]

@sdhull sorry for the slow response, but a PR to that effect would be much appreciated. Thanks!

[sdhull]

@phillbaker having all the different storage adapters is a real stumbling block (and let's not even get into the excessive & unnecessary metaprogramming).

If I have to change the activerecord schema for that adapter that there be some sort of migration or migration guide or ... ?

[phillbaker]

@sdhull yup, adding a migration would be best thing to do. Here's an example of how doorkeeper did it: https://github.com/doorkeeper-gem/doorkeeper/blob/master/lib/generators/doorkeeper/templates/add_owner_to_application_migration.rb.

[sdhull]

Very helpful, thanks.

[sdhull]

OK I apologize it's incomplete but check out #318