Proposed modifications to our login amplitude metrics

[irrationalagent]

The scenerios/flows that a user can go through during the login/signin process are multiplying quickly, much to the benefit of the user (e.g. https://github.com/mozilla/fxa/issues/1756). Rather than playing whack-a-mole by patching our metrics incrementally to take these changes into account, I’d like to propose a couple more systematic modifications that would make things more easily extensible in the future.

These suggestions should all be non-breaking backward-compatible changes, i.e. they should not modify the behavior or meaning of existing events (unless the behavior of the event in question is broken). However, they would change the way we do analyses moving forward. I don’t at all consider this a “final proposal” and appreciate any feedback.

Change 1: New fxa - flow_begin amplitude event

tldr: let’s fire a new amplitude event at the beginning of every flow, if possible with an event property that indicates what flow is beginning

This event should fire at the very beginning of every registration and login flow. The purpose of adding this event is

1. To properly capture the impact of (future) auto-login events to our metrics. If we just counted auto-logins as “login complete” events without any corresponding top-of-funnel/”start” event (such as the fxa_login - view event that we currently use), we would be adding to the numerator in our conversion rates without adding to the denominator - in other words, they would become over-inflated.
2. In the future there may be scenarios in which the first event in a login/registration flow is not one of fxa_login - view or fxa_reg - view (the QR flow is one such example today). Having a generic “flow starts here” event for all flows will allow us to change the first steps in the login/reg funnels without worrying about breaking conversion rates.

The good news is that we already have a flow.begin event in our re:dash metrics, so the solution may be a simple as mapping this to an amplitude event. But before doing so we should consider:

1. Whether adding this event would cause us to overspend our amplitude budget (i.e. we should check the current event volume of flow.begin).
2. flow.begin currently fires in more places than just the beginning of the login and reg flows, e.g. when a user opens prefs, etc. It also fires on RP top of funnel events fxa_email_first - view and the new fxa_rp_button - view. Maybe that’s OK
3. We could consider attaching a new event property to this event, e.g. flow_type. The tricky thing here is we don’t always know the ultimate “destination flow” at the time that flow.begin fires - e.g. when the start of the flow is email-first. Maybe that’s would be OK and we just have an amplitude event property flow_type that could be login, reg, email-first, prefs, qr, button etc. This event property would allow easy segmentation of users starting particular flows.

Change 2: add a new event property login_type to login amplitude events

tldr we should add a new event property to all fxa_login events that denotes which version of the login flow a user is in

Without adding or removing existing events in the login/signin flow, we could add an event property to these events that denotes the type of login flow the user is engaged in. For example, we could have the property denote the most demanding action that the user needs to take to complete the flow. See the chart below for an example, ordered in descending order of user demand:

Value of Event Property	Description
qr	User is engage in the qr flow. NB perhaps this flow is sufficiently different from the others that maybe it just needs its own set of events.
confirm	User is required to confirm their login via email. This also implies that they have to enter their password.
password	User enters their password only, no email confirmation. This also implies that the user had to click a button to submit their password.
click	No password required, user clicks a button to login. This implies that the user had to at least take action to login.
auto	User is logged in automatically, no action required.

As noted above, one wrinkle here is that we sometimes do not “know” which version of the login flow a user will get until they complete an initial step, e.g. email-first. This means that we cannot attach this event property (in all cases) to the fxa - flow_begin event proposed above. Thus, we should (1) always attach it to the first event of the category fxa_login - * - (usually this will mean fxa_login - view), as well as (2) to fxa - flow_begin where possible.

All told, these two changes will allow us to

1. have better transparency into the login flows that our users go through, and will more easily allow us to understand their relative performance.
2. accurately incorporate future auto-login flows into our login success measures
3. More easily allow us to determine how often users are logging in to services using cached credentials

Thoughts? CC @davismtl

┆Issue is synchronized with this Jira Task ┆Issue Number: FXA-333

[rfk]

User is engage in the qr flow. NB perhaps this flow is sufficiently different from the others that maybe it just needs its own set of events.

FWIW this one feels different to me, and kind of orthogonal. We only have one pairing flow right now, but I can easily imagine a future where the user does a QR-code based pairing flow, but still as to enter their password in order to complete it (say because they need additional encryption keys, or because they haven't authenticated in a very long time). Even today, some users in the QR-code flow may need to enter a TOTP verification code.

[vbudhram]

from mtg: @irrationalagent Is there anything else to discuss? What train do you think this would land in if so?

[davismtl]

User is engage in the qr flow. NB perhaps this flow is sufficiently different from the others that maybe it just needs its own set of events.

FWIW this one feels different to me, and kind of orthogonal. We only have one pairing flow right now, but I can easily imagine a future where the user does a QR-code based pairing flow, but still as to enter their password in order to complete it (say because they need additional encryption keys, or because they haven't authenticated in a very long time). Even today, some users in the QR-code flow may need to enter a TOTP verification code.

At the very least, I'd say we want login_type=qr for flow_begin and login_complete. But you're right that most other steps in between might be entirely different.

[irrationalagent]

Maybe this just needs someone (probably me, I guess) to deliver a final spec here? Anyone else want to weigh in?

[davismtl]

• Per #2225 , I think we need the flow begin event.
• I think we'll like also only need the event property for fxa_* - complete and fxa_* - submit events based on how we plan to break out specific events for pairing: https://docs.google.com/document/d/1mYtPTswtzawcn-fwVIbJr1N2J-eiLYd6o1ymlB2DnKM/edit#

[jaredhirsch]

oh look, I'm assigned to this bug! ping me if this is high priority and I'll fit it in soon.

[data-sync-user]

➤ Wil Clouser commented:

We never had a final spec out of this and I'm not sure it's still relevant now that Looker is in play.  If there is work to do here, let's revisit with that spec.