When users agree to sign up for sync, we should end the study immediately

[Osmose]

If we successfully convince a user to sign up for sync, instead of the current behavior of opening a tab to about:accounts, we should instead end the study immediately, and redirect them to the post-study-survey. We should also include a parameter in the survey URL indicating that the user ended the study early due to conversion, as opposed to ending the study after the 2-week duration.

[Osmose]

@rayborn @gregglind @MattGrimes I've worked a bit on implementing this, and I've found that some of the potential ways users can exit the study aren't logging consistently, so I want to confirm the list of possibilities with ya'll before applying fixes. Here's the list of end states I can think of, and what they should log (along with the study_name and branch, like how the study template logs currently) and whether we should survey them:

• User sets sync up while the study is running, making them inelligible.
  • Log to telemetry with { study_state: 'sync-installed-inelligible'}
  • Survey with reason=sync-installed-inelligible in the querystring.
• User uninstalls the study
  • Log to telemetry with { study_state: 'user-ended-study' }
  • Survey with reason=user-ended-study
• Study expires
  • Log to telemetry with { study_state: 'end-of-study' }
  • Survey with reason=end-of-study
• User clicks the prompt in the advisor popup to install Sync without the feature offer
  • Log to telemetry with { study_state: 'user-converted-no-offer' }
  • Survey with reason=user-converted-no-offer
• User clicks the prompt in the advisor popup to install Sync with the special feature offer
  • Log to telemetry with { study_state: 'user-converted-feature-offer' }
  • Survey with reason=user-converted-feature-offer
• User clicks the link to disable all future warnings
  • Log to telemetry with { study_state: 'user-disabled-all-warnings' }
  • Survey with reason=user-disabled-all-warnings

Does that sound right?

[MattGrimes]

That looks correct to me. @rayborn when we see reason=user-converted-feature-offer we need to show some special language in the survey explaining to the user that this feature isn't currently available. I don't think anything special is needed for reason=user-converted-no-offer.

[ghost]

Looks comprehensive to me. I'll start modifying the survey accordingly.

[ghost]

@MattGrimes - How does this sound:

• No/Minor survey changes:
  • Survey with reason=sync-installed-inelligible -> same as "end-of-study" survey
  • Survey with reason=user-ended-study -> no change
  • Survey with reason=end-of-study -> no change
• New survey flows:
  • Survey with reason=user-converted-no-offer -> new survey branch. What do we care about? Motivation question? Feature expectations question? At the very least just some text about why they aren't on an actual installation page.
  • Survey with reason=user-converted-feature-offer -> new survey branch. What do we care about? Motivation question? Feature expectations question? At the very least just some text about why they aren't on an actual installation page.
  • Survey with reason=user-disabled-all-warnings -> survey questions like the "Give Feedback" survey in our Mobile/Foundation tests about user annoyance about prompts? Alternatively we could not prompt a survey on this branch (code change :( ).

[MattGrimes]

Here's my thoughs:

Survey with reason=user-converted-no-offer: I'd want to know why they decided to sign up for an account. Was it that we offered sync? Was it because they THOUGHT it had something to do with the password notification idea? Had they heard of accounts before?

Survey with reason=user-converted-feature-offer: Same as above. They signed up, we assume, because we promised that they would get password notifications. Would be good to validate that. Maybe a maxdiff sort of thing with our current account features + password notifications. Could throw vpn in there too if we wanted just for kicks.

Survey with reason=user-disabled-all-warnings: I want to know if they thought it was creepy. Are they worried about us knowing their passwords? Maybe they don't believe it's real?

[ghost]

Thanks for the ideas. I didn't add a maxdiff. I'm concerned about having a lot of questions and max diff is pretty time consuming. Instead I just have a general "Access to advanced services such as a VPN to keep your data private on public networks" as a checkbox option for Accounts expectations. I'm open to alternatives.

@MattGrimes - please review the following links:

• https://qsurvey.mozilla.com/collab/security-advisor?reason=end-of-study
• [New; same as above] https://qsurvey.mozilla.com/collab/security-advisor?reason=sync-installed-inelligible

• https://qsurvey.mozilla.com/collab/security-advisor?reason=user-ended-study
• [New; Similar to user-ended-study] https://qsurvey.mozilla.com/collab/security-advisor?reason=user-disabled-all-warnings

• [New] https://qsurvey.mozilla.com/collab/security-advisor?reason=user-converted-no-offer
• [New; same as above; we can do crosstabs for analysis] https://qsurvey.mozilla.com/collab/security-advisor?reason=user-converted-feature-offer

[MattGrimes]

I put some comments in the surveys, but here's my feedback:

1. I don't know if we even need to survey users that are inelligable. They get DQ'd the moment they install the add-on, so they have nothing to tell us.
2. I'd like to find out if users think notification in the browser, notification by email, or both is the preferred way of being told about password breeches.
3. Do users prefer to only be told about password issues on sites they have an account saved for. This test does not look at saved passwords.
4. If you end the study or disable all the warnings, I'd still like the user to answer questions like "I believe mozilla has my best interest in mind" etc.
5. For the user converted surveys, I'd also like them to answer the remainder of the questions. I assume (bold claim) that lots of people will end the study early by clicking on that sign up button. Even if they only saw one password notification I'd like to know if they thought it was creepy, helpful, etc.

[ghost]

I don't know if we even need to survey users that are inelligable. They get DQ'd the moment they install the add-on, so they have nothing to tell us.

Commenting on this particularly since this needs resolved ASAP since it may be a code change. I thought that this was for people who enabled sync mid experiment, not people immediately DQed. @MattGrimes, @Osmose - am I confused?

I'll address the other comments shortly.

[Osmose]

Mentioned this during the meeting, but @rayborn 's explanation is correct. Only users who enable sync mid-experiment are surveyed.

[ghost]

@MattGrimes - please review the following links:

==End of Study==

• General Notes:
  • Split the notification question into 2 dimensions: 1) Notification method (email, in-browser, etc), 2) Notification aggression (heuristic definition)
• End of Study
  • Mostly unchanged
  • observe-only, passive-panel, auto-open-panel
• Sync Installed Inelligible
  • New; same as above
  • observe-only, passive-panel, auto-open-panel

==Sync Converted==

• General Notes:
  • These users also get all of the "end of study" questions
• User Converted No Offer
  • [New]
  • observe-only, passive-panel, auto-open-panel
• User Converted Feature Offer
  • [New; same as above; we can do crosstabs for analysis]
  • observe-only, passive-panel, auto-open-panel

==User Disabled==

• User Ended Study
• User Disabled All Warnings
  • New; as of 11/08 Identical to user-ended-study except the language

Per Kamyar’s feedback

• I added a "I didn’t know I was in a study" and "I don't want to be in any Firefox studies" to the opt-out surveys
• "What features did you expect to use" -> "What features did you expect"
• Added "Save web sites for later viewing" as a feature option

[ghost]

PS sorry for commenting on a closed issue, but the discussion started here, so I want to continue it here.

[MattGrimes]

Looks good to me. I think we're ready to launch tomorrow morning.