Set up A/B testing infrastructure

[mathjazz]

This issue was created automatically by a script.

Bug 1695961

Bug Reporter: @mathjazz CC: @Pike, @flodolo Depends on: Bug 1701631, Bug 1701632

As part of the Improve mechanisms for one-directional community engagement L10n team objective, we'd like to experiment with design changes in Pontoon to increase the number of users reading notifications by 2x (KR).

We need to create a reusable A/B testing infrastructure that will allow us to:

• serve different experiment to different subset of users
• measure relevant metrics for each experiment

Actions we want to measure for each experiment:

• Number of times we display the red bell to users
• Number of clicks on the bell
• Number of clicks on the “See all notifications” link in the dropdown
• Number of clicks on the “Mark all Notifications as read” link in the add-on

Data we could store for each action:

• Action name
• Experiment name (could be used for a/b testing)
• Datetime
• User
• Additional data (e.g. Number or items marked as read)

Open questions:

• Is it sufficient to use an external analytics tool vs. storing metrics internally in the DB?

Resources:

• https://waffle.readthedocs.io/en/stable/
• https://github.com/mozilla/bedrock/blob/master/docs/abtest.rst

[mathjazz]

Comment Author: @flodolo

I don't think we should introduce external libraries. They come with technical challenges (e.g. users blocking them), and potential security/privacy concerns.

Take for example Bedrock, they explicitly call it out

Security (keeping the experimentation surface as small as required).

In Pontoon we have login information on top of normal data visualized.

[mathjazz]

Comment Author: @Pike

I've been thinking a bit about this, and I'd add cohorts to the picture.

• Experiment
  • Cohort A
  • Probability (intended size of the population)
  • Users
  • Cohort B
  • Cohort C

On top of that, cohorts change over time. Not really sure how to model that. Maybe actionlog, and keep cohort db entries if they change, or recreate the cohort set whenver the cohorts change, and then the history would be done by adding start-end datetimes.

Data pings could then be:

• Cohort (implies experiment)
• Datetime
• datablob

I'd not record the user with the data ping. Action name could be part of the datablob, or an extra column. Really depends on how easy the queries for things inside the datablob are, I don't have much experience with json fields.