search

mozilla / fxa-content-server

DEPRECATED - Migrated to https://github.com/mozilla/fxa
Mozilla Public License 2.0
163 stars 120 forks source link

Differentiate between different services/entrypoints in flow metrics graphs #3002

Closed ryanfeeley closed 8 years ago

ryanfeeley commented 9 years ago

Now that we are showing the registrations screen at the end of first run, our failure rate is going through the roof. I would argue that simply viewing the page does not indicate intent to register. Instead we should be looking for users who at least enter a valid-looking email address into th email field.

shane-tomlinson commented 9 years ago

Updating the title to change iframed to firstrun.

shane-tomlinson commented 9 years ago

@stomlinson and @vladikoff to chat about how to tackle this.

ckarlof commented 9 years ago

In triage, we discussed changing our signup/signin metrics to be a more conventional funnel, e.g., capture events like:

I probably missed some. These are ideally tagged with entry point, user session id, and relier so we can ask questions like: "which entry points are most effective in getting users to sign up" and "which reliers are most reliable in getting users to successfully signed up".

ckarlof commented 9 years ago

We also discussed sending these funnel metrics into GA, because they have cool tools for visualizing.

vladikoff commented 9 years ago

image

shane-tomlinson commented 9 years ago

Here is what @vladikoff and I are thinking. A funnel is useful for optimizing the flow, but this is not our most pressing need on a high level dashboard. For a high level dashboard, we still want to know high level success and failure rates. We currently show a aggregated success/failure rate for all sources of traffic. The firstrun flow has such high traffic that it totally overshadows all other sources of traffic.

Until the firstrun flow was introduced, all users had to perform some action to arrive at FxA; usually via a button click. This button click is a strong indication that users were at least somewhat interested and acted as a natural filter to for our stats.

With the firstrun flow, all new users are sent to FxA; no user provided signal is given, all filters are removed.

Instead of showing a single aggregate value for success/failure, it makes more sense to break the success/failure rate out by relier (for OAuth flows) and entrypoint (for Sync). Perhaps we could show an aggregate success rate for all OAuth flows as well.

This would allow us to see how successful various services are at driving signups, and whether certain entrypoints or services provide less value than others. Separating out by relier/entrypoints would allow us to tweak the flow for each individual relier/entrypoint without affecting the stats of the others.

vladikoff commented 9 years ago

To fix @ryanfeeley's problem we were thinking about adding fxa.content.screen.iframe.signup with entrypoint firstrun, or just fxa.content.screen.firstrun.signup. Probably the first one because it makes more sense.

rfk commented 9 years ago

it makes more sense to break the success/failure rate out by relier (for OAuth flows) and entrypoint (for Sync)

:+1:

It would still be useful to see a high-level aggregate though. Perhaps we can start it at a slightly later point in the funnel, e.g. "user brought focus to some element in the page" rather than just "user saw the page".

rfk commented 9 years ago

e.g. "user brought focus to some element in the page"

@shane-tomlinson says nope; the first-run flow auto-focusses on the email field when it displays the form.

vladikoff commented 9 years ago

@shane-tomlinson one thing with fxa.content.screen.iframe.signup, if we add that then we would have to modify our firstrun dashboard to have a larger query (if we want to compare) signup screens based on entrypoint.

shane-tomlinson commented 9 years ago

firstrun dashboard to have a larger query (if we want to compare) signup screens based on entrypoint.

Is the firstrun dashboard limited to firstrun data, or does it look at all Sync data? If just the firstrun, it's entrypoint is firstrun, so shouldn't have clutter from other entrypoints?

vladikoff commented 9 years ago

@shane-tomlinson example, currently we have:

{
  "viz": "timeseries",
  "requests": [
    {
      "q": "sum:fxa.content.signup.success{$env,entrypoint:uitour}.as_count()",
      "style": {
        "palette": "purple"
      },
      "type": "line"
    },
    {
      "q": "sum:fxa.content.signup.success{$env,entrypoint:firstrun}.as_count()",
      "style": {
        "palette": "orange"
      },
      "type": "line"
    }
  ]
}

we should have to change the graphs to include both:

"q": "sum:fxa.content.signup.success{$env,entrypoint:firstrun}.as_count() + sum:fxa.content.iframe.signup.success{$env,entrypoint:firstrun}.as_count()",

To include old data and new data

ckarlof commented 9 years ago

What are the deliverables of this work? I.e., what graphs/visualizations/reports are you going to produce?

FWIW, the ones here aren't working for me: https://metrics.services.mozilla.com/accounts-dashboard/flow/

Some of my issues:

rfk commented 9 years ago

If there are enough individual requiements/deliverables involved in the cleanup here, it may be worthwhile escalating to a "signin funnel metrics" feature in Aha for planning purposes, and breaking them up into individual issues.

https://metrics.services.mozilla.com/accounts-dashboard/flow/

I also wonder if datadog is better suited to doing the visualisations on that page, rather than having to go through the whole metrics pipeline.

philbooth commented 9 years ago

As a first step for this issue, to answer the two questions:

...I've created the following dashboard:

philbooth commented 9 years ago

The second part of this issue concerns the flow dashboard. From the comments above, it's not clear whether it provides much value because it lacks the granularity of insight by entrypoint/relier that the datadog charts have.

So we could try to fix it as per @vladikoff's fxa.content.screen.iframe.signup recommendation or we could just delete it. Anyone?

shane-tomlinson commented 9 years ago

The flow dashboard is used by management to keep tabs on various services, nixing graphs would probably cause some consternation.

Maybe the valuable information for the flow dashboard is per-service, with all sync entrypoints counted as one bucket?

vladikoff commented 9 years ago

from mtg: check status of this... cc @philbooth

philbooth commented 9 years ago

Status:

philbooth commented 9 years ago

@shane-tomlinson, is your suggestion for proceeding with this covered by https://github.com/mozilla/accounts-dashboard/issues/27?

shane-tomlinson commented 9 years ago

@philbooth - yes, yes it is.

rfk commented 9 years ago

Apologies for neglecting this issue @philbooth, a couple of thoughts below.

I was hoping there might be some comment on the new datadog dashboard, as to whether it answers the two questions indicated.

LGTM, and I think it highlights that these are useful axes to break down the data by - entrypoints for sync, reliers for oauth. Unfortunately I think that could be a terrifically complicated graph if we try to break down a multi-stage funnel across those axes.

The flow dashboard is used by management to keep tabs on various services, nixing graphs would probably cause some consternation.

@shane-tomlinson I think we discussed this at one point but I don't remember the details - did you have anyone specific in mind with "management" here? I can reach out and see about expectations.

Maybe the valuable information for the flow dashboard is per-service, with all sync entrypoints counted as one bucket?

If I understand correctly, the proposal here is to:

In which case, we need to dig into the issues @ckarlof raised in https://github.com/mozilla/fxa-content-server/issues/3002#issuecomment-137189874 - what exactly defines "success", "failure" and "bounce", and should they add up to 100%?

rfk commented 9 years ago

(Also we seem to have moved a fair way from the original bug title, so I updated it)

philbooth commented 9 years ago

In which case, we need to dig into the issues @ckarlof raised in https://github.com/mozilla/fxa-content-server/issues/3002#issuecomment-137189874 - what exactly defines "success", "failure" and "bounce", and should they add up to 100%?

The calculations are here. The discrepancy is because the failure rate is expressed as a percentage of engaged users rather than users as a whole. I've opened https://github.com/mozilla/accounts-dashboard/pull/30 to fix it.

rfk commented 9 years ago

should they add up to 100%

FWIW, I'm open to the suggestion that they shouldn't add up to 100%, but the presentation of the graphs kind of encourages one to think that they should.

The discrepancy is because the failure rate is expressed as a percentage of engaged users rather than users as a whole

Ah, thanks, interesting. I think I can see what it would be done that way - in isolation, failure as a percentage of users who tried seems more interesting than as a percentage of users who happened to find themselves on that page. But in context of the other graphs, it's confusing, so your linked fix looks like a great initial cleanup.

Having stared at these graphs for a while now, I think we should consider simplifying them down to just three views, answering the three most important high-level flow questions:

1) How many users are entering the flow? This is the red line on the "account signup flow" graph, but broken down by entrypoint/service.

2) Of the users that enter the flow, what percentage of them engage? This is the inverse of the current "bounce rate" graph, but again broken down by entrypoint/service.

3) Of the users that engage, what percentage of them succeed? IIUC this is the inverse of the current "failure rate" graph, and is a more interesting view than doing it as a percentage of everyone who landed on the page. A high bounce rate could otherwise mask a pretty bad non-success rate.

Splitting by sync entrypoint and by service, this would be a two-by-three matrixs of graph like this:

                                       +--------------------+       +------------+
   Users entering the flow:            | by sync entrypoint |       | by service |
                                       +--------------------+       +------------+
                                       +--------------------+       +------------+
   Engagement rate:                    | by sync entrypoint |       | by service |
                                       +--------------------+       +------------+
                                       +--------------------+       +------------+
   Success rate:                       | by sync entrypoint |       | by service |
                                       +--------------------+       +------------+

(This also frames all three as "positive" metrics, where graphs going up and to the right represent success rather than failure; that seems nice from a motivational standpoint)

We can follow up on more detailed funnel-based metrics in future work, but those seem like the broadest strokes of what's useful in the current view, and we have all the metrics necessary already, it's just a matter of changing how we present them.

@philbooth thoughts? @shane-tomlinson does this mesh with the breaking-things-down action you laid out in https://github.com/mozilla/fxa-content-server/issues/3002#issuecomment-136735596?

To @ckarlof's point about "success" versus "verification", we could replace the "account verification flow" graphs with a fourth row showing by-entrypoint rate of verifications, but I think would require new metrics logging and should be split out into a separate bug.

philbooth commented 9 years ago

I think we should consider simplifying them down to just three views, answering the three most important high-level flow questions...

This sounds really good to me @rfk. I'm not sure what I need to do to get the entrypoint categorisation into these figures yet, but I'll dig into it today.

shane-tomlinson commented 9 years ago

does this mesh with the breaking-things-down action you laid out in #3002 (comment)?

@rfk - indeed it does, and converting to all success is a nice modification.

philbooth commented 9 years ago

Have opened a PR for a new Heka filter that (I hope) enables this work in the dashboard:

https://github.com/mozilla-services/puppet-config/pull/1612

vladikoff commented 9 years ago

Blocked by https://github.com/mozilla-services/puppet-config/pull/1612#issuecomment-153326100

vladikoff commented 8 years ago

We will try again to get the filter working locally, if it fails then we will proceed from there on...

vladikoff commented 8 years ago

Woohoo!

rfk commented 8 years ago

re-marking this as blocked on metrics team r?

vladikoff commented 8 years ago

ping needed if no activity next meeting

vladikoff commented 8 years ago

for next 2 weeks @philbooth ?

philbooth commented 8 years ago

for next 2 weeks @philbooth ?

@vladikoff, hopefully yep but I'm not promising anything. :grin:

vladikoff commented 8 years ago

Wooo @rfk!

rfk commented 8 years ago

We're giving up on the approach linked above and will revisit this using new metrics infra, leaving assigned to me to come up with a next step

rfk commented 8 years ago

We're going to explore the next iteration of these graphs as part of https://github.com/mozilla/fxa-features/issues/13 so I'm closing out the issue here.