Report on completion rates and learner satisfaction by project and tutorial type

[terichadbourne]

Our Q3 OKRs include a goal to break down learner satisfaction and tutorial completion rates by segments (tutorial, content type, and project).

To accomplish this we need to break down the following key results:

• learner satisfaction scores (from the tutorial feedback tracking)
• lesson completion rates (from lesson and tutorial passed tracking)
• (nice-to-have) other success metrics such as submission of wrong answers, resetting of code or peeking at solutions)

by the following categories (in addition to by tutorial as we already do):

• tutorial type (multiple choice, code, text, etc.)
• project (IPFS, Filecoin, etc.)

Steps to completion for Q3:

• [x] Add relevant segments to Countly in order to automate this moving forward (see PR #532)
• [ ] Build the best reports Countly will allow us to :) @terichadbourne
  • [x] learner satisfaction by question and tutorial type (9 widgets)
  • [x] learner satisfaction by question and project (12 widgets)
  • [x] learner satisfaction by question and tutorial (21 widgets)
  • [ ] lesson completion rate by tutorial (7 widgets)
  • [ ] lesson completion rate by project (4 widgets)
  • [ ] lesson completion rate by content type (3 widgets)

NOTE: Will have to add each tutorial manually to OR logic in reports based on tutorial type or project until next quarter's retrofitting

Next quarter:

• [ ] Adjust existing Countly data to include new segments to fix historical data (with help from Countly) @zebateira
• [ ] Update formulas to use project and tutorialType fields rather than x tutorial OR y tutorial @terichadbourne

NOTE: Will only have to manually add completion rate and learner satisfaction reports per tutorial for new tutorials after this step.

[zebateira]

For tracking purposes, here are the data migration notes:

Steps:

• Add tutorial, tutorialType, and project segments to tutorialFeedbackSurveyAnswer and tutorialFeedbackSurveyCompleted tracking events
• Add tutorialType and project to tutorialPassed tracking events
• Add lessonType, tutorialType, and project to lessonPassed, submitWrongChoice, submitWrongCode, resetCode, and viewSolutionCode tracking events

Each value needs to be inferred from the paths, so here is the mapping to be used:

Tutorials Mapping

/data-structures

tutorial: Data Structures
tutorialType: text
project: DWeb Concepts
lessonType:
    all: text

/basics

tutorial: P2P Data Links with Content Addressing
tutorialType: code
project: IPFS
lessonType:
    all: code

/blog

tutorial: Blog
tutorialType: code
project: ipfs
lessonType:
    all: code

/mutable-file-system

tutorial: Mutable File System
tutorialType: file-upload
project: ipfs
lessonType:
    1: text
    2: code
    3: file-upload
    4-11: file-upload

/regular-files-api

tutorial: Regular Files API
tutorialType: file-upload
project: ipfs
lessonType:
    1: text
    2-3: file-upload
    4: code
    5-6: file-upload
    7-8: code

/anatomy-of-a-cid

tutorial: Anatomy of a CID
tutorialType: multiple-choice
project: multiformats
lessonType:
    all: multiple-choice

/verifying-storage-on-filecoin

tutorial: Verifying Storage on Filecoin
tutorialType: multiple-choice
project: filecoin
lessonType:
    all: multiple-choice

Specifically for the lessonType, it will vary according to the lesson ID, so for the Regular Files API lesson 1, lessonType is text, but for lesson 2 and 3, it's file-upload. The lesson ID is defined in the url. So in these two cases: /regular-files-api/01, /regular-files-api/02 and /regular-files-api/03.

[terichadbourne]

I've figured out how to create reports on the average answer for a specific survey question! On the left are the reports we already had, which show the count of people who gave a specific answer, and we were doing the math ourselves to get the average. On the right are the reports for average score by question, set to monthly.

I'll now try to do the same but break down the segmentation by tutorial type and project.

[terichadbourne]

In order to create segmented lists, we currently need to use OR statements to gather up all tutorials of a specific type, because our project and tutorialType segments only apply to more recent data. After we update the historical data, we'll be able to update these formulas to use those. Until then, every time we add a new tutorial we'll need to update these filters. For example, this is the one used to get the average score for the Excitement to Continue Learning question for all multiple choice tutorials:

Breakdowns by tutorial type are now available in the dashboard:

As are breakdowns of learner satisfaction by project or pseudo-project:

For reference, it took 3.5 hours to create these first 21 widgets because there are no templates available - you overwrite a formula if you try to edit it, and can't "save as" with a new name.

[terichadbourne]

Tutorial satisfaction by question and tutorial. These are sorted in order of most completed tutorials. Unfortunately only the top 3 appear to be real data. My suspicion is that the bottom 4 don't have enough survey results over the given time period to render useful data.

[zebateira]

Nicely done 👌