App store analytics

[swwalker]

• As a system admin, I see an aggregated view of all custom URLS to analyze what apps we should investigate for integration and what integrations aren't working because users manually link instead. #40
• As a system admin, I see usage metrics for all apps in the app store because I want to know how to prioritize work and what is successful #46

How might we (the Kits project team) show system admins useful information about how the system is being used by users?

Questions Possible integrated app metrics

• number of installs
• number of clicks on installed apps
• number of unique clicks to installed apps
• comments below and on #77

Assumptions

Hypotheses

Related Problem Statements

203 #146

[JustinJohnsen]

Potential metrics to consider for apps:

Acquisition

• Lifetime usage: total amount of time users have spent in the app (either histogram or mean + standard deviation)
  • Average usage per user: mean value of lifetime usage
• User growth rate: tracking number of users over time
• Acquisition cost: cost of marketing campaigns divided by number of new users, usually classified by type of marketing approach to identify best channels for recruiting new users

Engagement

• Installations: number of times kit owners install an app in their kit
• Active users: number of users entering the app, measured by preferred period (day/week/month)
  • Stickiness: equals daily active users divided by monthly active users
• Session length: total duration of each user session between opening app and closing app (either histogram or mean + standard deviation)
• Retention: number of return users after first use, usually measured over a series of binned periods (return in first month vs. second month vs. third month, etc...)
• Time in app per period: total length of time a user spends in an app over a preferred period (day/week/month)

Quality/Performance

• App crashes: frequency that the app fails during a user session, preferably including cause for error
• Load speed: how quickly the app launches after the user clicks to open

Satisfaction

• Reviews: app store or in-app direct feedback from users
• Support response time: (mean/median) length of time between opening and closing a support ticket

[JustinJohnsen]

Decent web reference for these metrics and more: https://savvyapps.com/blog/mobile-app-analytics

[joliet]

This is related to feature: Create an Admin area for Kits where all admin features live - https://github.com/DukeLearningInnovation/kits/issues/440

[joliet]

Related to issue: Kits Analytics: build foundation for user and app event or activity tracking- https://github.com/DukeLearningInnovation/kits/issues/441

[joliet]

For reference: Here is what the stats page in Toolkits looks like now - https://duke.box.com/s/fc5g98ii6q84mikftlxql6nruqusmcyu

We still want all of this data and to be able to see it at a high level like this, but what would better to to be able to drill down on any of the numbers listed as a filter to see all the kits that fall into a specific data point. We'll want to look for trends on what disciplines/departments are using certain tools, and better understand how people are using user-created kits.

We might consider having app store analytics as just part of the overall admin analytics tool.

[joliet]

Information on dashboards curated by my colleague Kim Manturuk - https://duke.box.com/s/5uug16q3d2x3zwgzd52moxsfid8n1iau

[joliet]

Feature planning: Important questions we should answer before beginning to build this feature (see link below for more detail).

• Have you identified the main outcomes (changes you desire) and indicators (measures of these changes) for your project, program, team or organisation?
• Is it possible and meaningful to measure the indicators you have chosen on a regular basis? (anywhere from daily to quarterly).
• Define your primary purpose for the dashboard
• Identify the indicators that you will display on your dashboard
• Decide what you would like your dashboard to look like

https://www.betterevaluation.org/en/evaluation-options/data_dashboard

[joliet]

Idea for MVP for this: A page that prompts for a date range and has a 'run report' button that runs a query for data in that date range. The report would be a downloaded .csv file. The query would include:

Creator Duke ID | Creator name | Course ID | Course Name | Course Type | Create date | Resource | Resource Display Name | Resource ID (url for custom link, siteid for Sakai site, etc).

This row would repeat for every resource associated with the kit. This format would make it easy to use pivot tables and Tableau to analyze the data.

[joliet]

Update: Perhaps just make this a job that runs at specific intervals emailing this data to those who need it. I had a conversation recently with Matt Royal about this. He would like to have this report weekly with Box IDs included. Who else will need this report? Are there other identifiers needed in tools that we can include (Sakai site id, etc)?

Would we need a page that simply shows the Jobs and allows you to add new ones/modify exisitng?

[joliet]

Updates on requirements for this feature per conversation with Matt R and the Kits Team on 5/4:

• The NetID of the kit creator should be in its own column
• The tool id for each app in a kit should be included in the report in its own column with the appropriate descriptive heading

The goal is to provide a comprehensive report for all relevant Kits data and to get this data into Tableau. Service Owners for all integrated apps would have access to this and create the views they need for reporting.

Questions: Can we embed a Tableau dashboard into the Reporting feature in Kits? Should we still add an option to run a report for certain dates with an option to download the data? Or will those options be possible through Tableau?

Anything you'd like to add to this @ewendland23 ?

[joliet]

Notes from meeting with Susan Starnes today:

• If we have DUID, this can be tracked back to primary affiliation and department.
• Jolie will define requirements (how data should be viewed at high level and what more specific views are needed per Kits Team and Service Owners)
• Liz will work on getting webservice ready to send data to Susan
• Jolie will send Susan reports Liz have run and previous analysis/summary docs for Kits usage.

Next meeting will be a few weeks out. Jolie will organize.

[joliet]

@noahhh Here is the Figma mockup for the Kits Analytics page. Jen has an icon for this on the #440 Admin area frame in the same design file - https://www.figma.com/file/vbjsQkApG7rNknJUwv73xG/Admin-Console?node-id=0%3A2265 . Right next to the mockup is the embed code and button link text.

[noahhh]

@joliet this is out in test, but the icon is not showing up for some reason. Will have to fix that when I am back from vacation.

[joliet]

This looks good @noahhh . My only feedback is to edit the button text to read 'Open in Tableau'.

[joliet]

Looks great. Thanks @noahhh ! This feature is ready for production.

[noahhh]

@joliet this is in production