Improve GLAM instrumentation to better understand the app usage

[Iinh]

One of our focuses in H2 is to understand GLAM usage better. Specifically, we want to have more clarity of what role GLAM should serve in the data analysis workflow: is it a starting point or an endpoint, or is it something in between? As part of this effort, we need to investigate a better way of instrumentation.

Currently, we use Google Analytics as our main tool to analyze user behaviors. Information we can get from GA includes:

• Most visited probes
• User count
• Where the users are from, etc.

While helpful, this doesn't necessarily give us a clear picture of how GLAM is used. For example, it doesn't give insight into:

• Number of sessions / lengths of sessions
• Number of links clicked out to other tools
• What custom toggles/menus are being used in each session

Another limitation of using Google Analytics is most Mozillians use some form of adblocker which blocks the service. We suspect that actual usage might be as high as 10x what GA is showing, according to the API log dashboard.

I don't have a clear answer to how we should approach this task yet. Probably we could start this work with:

1) Identify 2 - 4 questions we want to target initially. 2) Start executing on instrumentation and reporting for those questions. Note that if we decide to stick with Google Analytics as the main analytics tool, we need to make sure that the data we collect is in line with our privacy policy.

[rafrombrc]

Here is a list of possible questions we'd want to answer, extracted from some meeting notes:

• Which probes?
  • Most looked at
  • Least looked at (long tail?)
  • Specific collections of probes a’la OpMon or Nimbus “outcomes”?
  • Which probes are looked at in a single session?
  • What app's probes are most viewed?
• How many users of GLAM?
• Seasonality
  • Around release cycles?
  • Weekly?
  • Monthly?
• What channels?
• What filters, dimensions are used (e.g., toggle on 99%, switch aggregation to major version)?
• What links to other data tools are most used (e.g., view sql, open glean dictionary)

Of all of these, my hot take on which are the highest priority are probe visit counts, probe bundles (if we can get that signal), user counts (w some population analysis, if possible... what teams are heavy users would be great to know) and exit paths. There aren't that many places where we provide links out from GLAM to other tools or destinations, so we should be able to instrument all of them and get a sense of what workflows are actually proving useful.