Add end of study surveys

[biancadanforth]

This issue is for discussing survey design and integration with the study code.

Since we have 2 controls and 2 experimental branches, this study will have 4 different URLs for when a user from each of these 4 branches completes the study at the end of two weeks.

There may also be a fifth URL for those users who end the study early (i.e. by toggling tracking protection).

Per rrayborn (GitHub rayborn):

It will just be <url>?branch=<branchid>&state=<stateid) if that works for you. State being 'complete' or 'early-termination' (or something like that). I don't think we'll need other URL params.

[biancadanforth]

@rayborn, If there is a survey for users who end the study by toggling tracking protection on/off (the fifth URL mentioned above), we should be sure not to add anything at the end of this survey about resetting TP. See Issue #21 for more details.

[biancadanforth]

URL format: <url>?branch=<branchID>&reason=<reason>

Disable survey reason codes

Standard endings

• reason = "user-disable": User has uninstalled the addon from about:addons

Study-specific endings

• reason= "user-disabled-builtin-tracking-protection": User has lowered the level of Tracking Protection compared to their treatment branch from about:preferences or about:config.
• reason = "user-enabled-builtin-tracking-protection": User has raised the level of Tracking Protection compared to their treatment branch from about:preferences or about:config.
• reason = "introduction-confirmation-leave-study": User has opted to disable Tracking Protection from the Introduction panel confirmation screen.
• reason = "page-action-confirmation-leave-study": User has opted to disable Tracking Protection from the page action panel confirmation screen.

End of Study survey reason codes

Standard endings

• reason = "expired": study has expired

Study endings that won't bring the user to a survey:

Standard endings

• reason = "ineligible": User is ineligible for the study as determined by the Config.study.isEligible().

[biancadanforth]

We currently have two survey links: one to a "Disable" survey and one to an "End of Study" survey.

Test Links: Disable - only difference is a write in question End of Study Production Links: Disable End of Study

The only difference between the Disable and End of Study surveys is a write in question. That write-in question is: "Why did you disable Tracking Protection in Firefox?"

There are study endings listed above in which the user can end the study before it expires without having disabled Tracking Protection:

• reason = "user-enabled-builtin-tracking-protection": User has raised the level of Tracking Protection compared to their treatment branch from about:preferences or about:config.
• reason = "user-disable": User has uninstalled the addon from about:addons

@rrayborn, what are our options here for adjusting the survey?

---

Also, we wanted to tell users when the study ends that Tracking Protection has been reset to its default value (enabled in Private Browsing mode only), and point them to about:preferences#privacy if they would like to change it. I don't see that at the end of either the Disable or End of Study surveys -- can you add that @rrayborn?

[ghost]

I don't see many points where the survey itself would need altered. I've modified the logic as follows:

• Treat these users as generic disables. I don't think we need different flows. Crosstabs will be of interest
  • reason=user-disable
  • reason=user-disabled-builtin-tracking-protection
  • reason=page-action-confirmation-leave-study
• Treat these users as generic completes. I don't think we need different flows. Crosstabs will be of interest.
  • reason=user-enabled-builtin-tracking-protection
  • reason=expired
• I'll follow up with the stakeholders about what we want to ask users who immediately opt-out. I'm not certain we'll even want to survey these users, but it might be interesting to get an idea of their motivations
  • reason=introduction-confirmation-leave-study

[biancadanforth]

TODO rrayborn:

• [x] Add a note at the end of the survey for reason code expired that Tracking Protection has been reset. Fixes #85 .
• [x] Send Bianca updated Survey base URLs with appended reason codes as described below.
• [ ] Modify survey accordingly for introduction-confirmation-leave-study per pdol's preference.

TODO bdanforth:

• [x] Modify the baseURL in Config.jsm for introduction-confirmation-leave-study to use the generic disable baseURL.
• [x] After receiving updated survey URLs from Rob, check for weird questions that don't match how the user arrived at the survey.
• [x] Check if Kamyar's new telemetry structure/edits affect end study reason code strings. If so, update strings and send to Rob to change. Update: They are the same.

Updated my endings/buckets below:

Generic disables

Standard endings

• reason = "user-disable": User has uninstalled the addon from about:addons

Study-specific endings

• reason= "user-disabled-builtin-tracking-protection": User has lowered the level of Tracking Protection compared to their treatment branch from about:preferences or about:config.
• reason = "page-action-confirmation-leave-study": User has opted to disable Tracking Protection from the page action panel confirmation screen.

Generic completes

Standard endings

• reason = "expired": study has expired

Study-specific endings

• reason = "user-enabled-builtin-tracking-protection": User has raised the level of Tracking Protection compared to their treatment branch from about:preferences or about:config.

No survey

Standard endings

• reason = "ineligible": User is ineligible for the study as determined by the Config.study.isEligible().

TBD await word from pdol/cmore

Study-specific endings

• reason = "introduction-confirmation-leave-study": User has opted to disable Tracking Protection from the Introduction panel confirmation screen.

[biancadanforth]

From rayborn; here are updated TESTING links:

Alright, added tracking protection info to the end of the "expired" survey (and added a bit more of an explanation to the end of the other reasons too). https://qsurvey.mozilla.com/collab/tp-perception?reason=user-disable https://qsurvey.mozilla.com/collab/tp-perception?reason=user-disabled-builtin-tracking-protection https://qsurvey.mozilla.com/collab/tp-perception?reason=page-action-confirmation-leave-study https://qsurvey.mozilla.com/collab/tp-perception?reason=user-enabled-builtin-tracking-protection https://qsurvey.mozilla.com/collab/tp-perception?reason=expired

https://qsurvey.mozilla.com/collab/tp-perception?reason=introduction-confirmation-leave-study

For production links:

just replace /collab/ with /s3/, e.g.: https://qsurvey.mozilla.com/s3/tp-perception?reason=expired

[biancadanforth]

@rayborn : Thanks for sending me updated survey links! I have added them to the study. I took a look at each survey to make sure the questions made sense based on the exit condition. I found a couple discrepancies/issues. Can you fix these?

• The user-disable survey asks users why they disabled Tracking Protection. Since users arrive at this survey by manually uninstalling the addon in about:addons, I think this question should be removed.
• While the page-action-confirmation-leave-study survey asks the users why they disabled Tracking Protection, the introduction-confirmation-leave-study survey does not. It should!
• There's a misalignment between a click/focus effect and the stars on the star rating question. See screenshot below:
• pdol/cmore would like us to survey users on introduction-confirmation-leave-study, so can you make sure that survey is modified appropriately for that exit condition?

[ghost]

• Fixed the 'user-disable' explicit question about disabling
• [DIFFERENT THAN THE ABOVE AFTER SLACK CONVERSATION] Created a very simple survey for 'introduction-confirmation-leave-study' users that asks them why they aren't participating and gets a general star rating for comparison against other branches
• Fixed, thanks SG changed the Star CSS which caused an issue with singular star grids
• I think this is covered by the new very simple path we created

I'm retesting all flows once more, but I think we're almost ready to launch (will ping)

[ghost]

W also added:

• "user-enabled-auto-private-browsing" - Basically, if at any point they change a pref that turns all windows into private windows, we end the study.
• "user-installed-ad-blocker" - user installed an ad blocker which would confound our results.