Investigate decline in signup success rate

[davismtl]

Since December 20th, signup success rate on firstrun has been declining whereas it has been improving on other entry points.

To my knowledge, the funnelcakes that were released during this period were "Vanilla", meaning they were identical to release and used to measure attribution.

This issue is to investigate the cause of this decline.

[davismtl]

Here is a screenshot comparing to other signup touchpoints (which have improved during the same period)

(I cut the values of the Y axe, see values here: https://sql.telemetry.mozilla.org/queries/1975#3534)

[shane-tomlinson]

@davismtl - we are going to put this in next, but are unsure where to start.

[davismtl]

Prob a good first step is to see what the full funnel look like for first run Is there a drop at a specific step of the funnel or is it overall?

Dunno if we can segment by OS or Fx version?

[vbudhram]

what the full funnel look like for first run Is there a drop at a specific step of the funnel or is it overall?

Since redash supports using the sankey type diagrams, I will try to build a graph to help us visualize where these drop-offs might be occurring.

[davismtl]

@vbudhram The limitation to sankey is that you won't be able to see an evolution over time. You'll have to run queries separately for each day and then try to compare them. (unless I missed something)

We already have a few funnel charts plotted over time. None of them seem to be working right now (another prob to investigate) but they might be a good starting point. All we would need to do is filter out by entrypoint. When plotted over time, we'll be able to see if there is a particular stepped that changed all of a sudden. https://sql.telemetry.mozilla.org/queries/1702

[davismtl]

@vbudhram One other thing I thought of... Normally when there are firstrun experiments, utm_source in eventmeta will be changed to firstrun[something] . This may also help to drill down. Although we don't believe a test would have impacted our numbers since samples have been small up until now and dates don't align, it's worth keeping in mind for segmentation.

[davismtl]

For reference, this query could be helpful to breakdown firstrun by utm_source. https://sql.telemetry.mozilla.org/queries/2299/source?p_start_date=2017-01-25&p_end_date=2017-02-01#4217

[rfk]

FWIW the decline seems to have levelled off, although it hasn't returned to its previous height: https://sql.telemetry.mozilla.org/queries/1975/source#3534.

I also merged all the various "entrypoint=firstrun" flows into a single "entrypoint" category on that graph.

[rfk]

I'm working on a breakdown of each step in the funnel by entrypoint here: https://sql.telemetry.mozilla.org/queries/3123/source#5932

[rfk]

It looks like this might be a decline in % of form submissions o the first-run page:

https://sql.telemetry.mozilla.org/queries/3123/source#5931

There's a corresponding increase in the number of form engagement events:

https://sql.telemetry.mozilla.org/queries/3123/source#6315

And a smaller increase in form submission events. So...is this drop the result of more users seeing the sign-in page by default on first-run, increasing form engagement but not following through to submission?

[rfk]

more users seeing the sign-in page by default on first-run

Erm, no, this but is dealing with the sign-up form. Can we account for increase in engagement with sign-up form from the first-run page?

[davismtl]

From observations I've made:

• the number of firstrun users according to GA has remained steady.
• despite a lower success rate that we've observed, signups have been climbing for first run (and overall): https://sql.telemetry.mozilla.org/queries/1975#3535
• engagement rate with the form have climbed but nothing has really changed: https://sql.telemetry.mozilla.org/queries/3123/source#6315
  • All the A/B tests run have resulted in lower engagement rates with form.
• these three points are sorta contradictions since users x engagement rate x signup rate = new accounts

Could the engagement rate be the soul cause of this? Why would it be higher?

[rfk]

Why would it be higher?

One option here is to dig into the precise circumstances under which the "engage" event is emitted, and make sure nothing changed around that, e.g. if some change to the first-run page is resulting in more spurious focus events onto the FxA form.

[shane-tomlinson]

if some change to the first-run page is resulting in more spurious focus events onto the FxA form.

I believe the behavior on the firstrun page is to allow our 'autofocus' to take place on the email element. That shouldn't trigger an engagement event though.

[vladikoff]

from mtg: for the product meeting...

[vladikoff]

Closing for now