Add experiment group to telemetry

athomasmoz commented 4 years ago

Why/User Benefit/User Problem

As a Product Manager, I want to understand how different variations of the experiment impact our metrics including retention.

Acceptance Criteria (how do I know when I’m done?)

Amplitude & telemetry include metadata on experiment name and variation for the duration of enrollment.
Change has been confirmed by data stewards and complies with our privacy policy
Data Science (Marissa) has confirmed the change gives her the information required for analysis

nbhasin2 commented 4 years ago

a) Find out if we can add another property (Experiment Identifier) on the all / some events we send to Leanplum or any other platform ? b) Find out what Temporary Device Identifier we can attach for running the experiment on Leanplum or any other platform. c) If a) and b) is possible then can we funnel users those who participated in a specific Experiment

Ref Ticket: https://github.com/mozilla-mobile/firefox-ios/issues/6242

nbhasin2 commented 4 years ago

a) Find out if we can add another property (Experiment Identifier) on the all / some events we send to Leanplum or any other platform ? b) Find out what Temporary Device Identifier we can attach for running the experiment on Leanplum or any other platform. c) If a) and b) is possible then can we funnel users those who participated in a specific Experiment

Answers:

a) There are three ways to do this - Click here for more info

    i. Find the users that are a part of the A/B test and then run a campaign on leanplum through that. 

    ii. Add a parameter to all or some events with experiment name 

    iii. Set the experiment name as one of the user attribute

    A user attribute is any piece of data associated with a user. Each session has its own user attributes, but they get copied from one session to the next. This is in contrast to event parameters, which may take on different values per event. For this reason, you generally use user attributes for things that do not change much within the session, or with which you want the entire session associated.

    Uses:

    - Personalizing content (variables, messages, resources, and interfaces) to different types of users.
    - Targeting an A/B test.
    - Filtering reports by a particular user attribute, like only looking at data for "whales".
    - Grouping reports (constructing a bar graph or histogram), by different attribute values. E.g. Create a histogram of average session length by number of
    friends.

    Examples:

    - Gender
    - Age
    - Number of friends
    - User interests

b) Temporary Device ID that is used by LP is basically as follows:

Device ID is set as follows:
- Default Value is - identifierForVendor
- If it cannot get the identifier for vendor, it will then fallback to randomly generated id

Note: IDFA can also be used but users have full right to disable IDFA and it'll then go back to identifierForVendor

c). Yes we can funnel the users who were a part of the A/B test and run further campaign. See subsection i) of A

nbhasin2 commented 4 years ago

@athomasmoz We can discuss in detail on what approach works best for adding Experiment group to telemetry.

nbhasin2 commented 4 years ago

Going with option iii) by adding the following as key value pair in the attributes. This will show in Leanplum. Experiment name - Experiment Name Experiment variant - Experiment Variant Experiment id - 123456789

nbhasin2 commented 4 years ago

Updated - https://github.com/mozilla-mobile/firefox-ios/wiki/Leanplum-A-B-test-setup

nbhasin2 commented 4 years ago

Data science has has confirmed that above information will give them what they want. Next steps are to send the values through Unified Telemetry / Glean.

Already started sending values through LP

mozilla-mobile / firefox-ios