Investigation on larger than expected attrition across opening FTM and dragging first stone to monster

[burrage]

Reference: https://docs.google.com/presentation/d/1i7x_mh0oSN8vM1E5zk2A3wCewIDD33D3aKRL2SkYuJc/edit?usp=sharing

We now have tapped_start and level_selected events. Does it look like people aren't making it past tapped_start and the problem might be that they are having technical difficulties in downloading the app?

[amitsinghsutara]

Hey @burrage ,

Hope you are doing well ;) Regarding this task please give guidance from where we can start and investigate into the attrition between opening and dragging!

Best Regards Amit

[burrage]

Party time! Amit has entered the chat!!!

OK cool this is going to be a really good exercise in getting into BigQuery a little deeper and the type of data we collect to understand how far our players are getting within the app and the parts where there might be some friction that we need to fix! This is also so we have someone else on the team besides myself that would know how to navigate the data like this!

So first thing I wanted to share is that, in parallel to this investigation, Jeff is in the process of creating an internal-only dashboard which will display most of these metrics for us without the need for having to query BigQuery every time! However it's still in the works, we will need to add to it over time, and we are running into some hosting issues at the moment that we need to figure out. Also we will most likely want to continue with these series of BigQuery queries because it helps Jeff QA his dashboard to make sure we're in the same ballpark with the numbers because he's actually doing much more fancy things than straight querying like we're going to do (e.g. storing separate intermediate datasets and JOINing them together!, etc.)

Anyway... Here's what we have so far...

I've created this running list of BigQuery queries to start getting rough numbers of how many unique people are going from action to action that we want them to complete.

These queries help form a "funnel" of actions a user would take sequentially (think of these queries visualized like this dashboard view).

[burrage]

To describe the "funnel" dashboard view and how the queries link:

• "Learners Reached" is equivalent to how many unique people open the Curious Reader container app for the first time.
• Next we move into how many unique people launch a Feed The Monster app.
• Here is where we would need to make a query against task #1000 to show how many people successfully download the Feed The Monster web content to be able to play FTM JS (we'll need to write this query when task #1000 is finished).

This takes care of where we would be looking at the data with the Curious Reader container app...

• So we next hop on over to the FTM JS data and look at how many unique people tap to start the game.
• How many unique people select a level.
• How many unique people drag a stone to the monster (and get a puzzle_completed event).
• How many unique people end up finishing a level (either successfully or not successfully)-- it doesn't matter to us!
• How many unique people end up finishing a level successfully (this would be a proxy to the concept "Learners Acquired"-- that someone has successfully completed the first level of FTM).

These represent the string of actions we want someone to take to be considered a "learner". You can imagine that in the future, we want to start creating "milestones" that someone reaches which could be things like finishing 25, 50, 75, 100% of the app.

[burrage]

Now with that info dump out of the way you might be asking... So Ben... what in the heck am I even supposed to DO on this task card and didn't we already find out (from that dashboard screenshot) that there is actually a LARGER issue from the amount of people dragging the stone once to successfully completing a level?!?!

All true!!!

What I would need help with here is:

1.) Getting someone else up to speed on BigQuery querying for these types of events we want to be looking at. So taking a look at these queries in the query list, understanding some of the syntax here as it can be a little crazy, and asking questions.

So let's start there before moving on to the next bit :-)

[amitsinghsutara]

Thanks a lot @burrage , For the above information🙌

[burrage]

OK part 2 now that we've got those under the belt...

To compare and contrast to Jeff's fancier dashboard for QA, I was trying to build out a chart based on those queries by language with the new app events (tapped_start and selected_level)-- see "CR w/ new events" second tab in the following Google Sheet:

https://docs.google.com/spreadsheets/d/1Kc9xhG69BxjNklH17Rc9wvLqtUCEfoJlGusafE_06QM/edit?usp=sharing

[burrage]

Next step of what we'd like to do... is to make a copy of and modify some of the queries in the Google Doc with all the queries... and constrain those queries to be from March 5th onward to a set end date say... two weeks later to March 19th and make sure that for at least a few languages in the Google Sheet fill out the values for each event type constrained to March 5th - March 19th.

We'd also want to verify with the queries that the data we're getting back decreases for each event and the numbers never increase going from event to event. e.g. opened FTM -> tapped start -> chose level -> dragged stone-- 1000 -> 985 -> 914 -> 800).