mint-metrics / mojito-r-analytics

Reporting & analytics tools for the Mojito split testing framework
https://mojito.mx/docs/r-analytics-intro
BSD 3-Clause "New" or "Revised" License
10 stars 2 forks source link

How can we show conversion metrics for variants that are not comparable #4

Open kingo55 opened 4 years ago

kingo55 commented 4 years ago

We often run experiments that implement a new feature that does not exist on the control group.

We want to see how many users are interacting with a particular feature, however we can't compare it against the control. This leads us to data like:

This makes our summary table hard to read and the metric plots somewhat misleading.

dapperdrop commented 4 years ago

Agreed, we need to treat these conversion points (goals) separately.

In the summary section, we can have another table, e.g. :

Goal Recipe Cvr
Some Goal Treatment 10%

If there are multiple treatments, perhaps we can conditionally format the cvr column to highlight the best performers.

Maybe the best way to flag these conversion points is by another attribute in goalList?

e.g.

list(
    title="Some Goal",
    goal="some_goal",
    operand="like",
    noControl = true
  )