OATutor

OATutor is an Open-source Adaptive Tutoring System (OAT) based on Intelligent Tutoring System principles. It uses Bayesian Knowledge Tracing for skill mastery estimation and is implemented entirely in React JS with optional logging using Firebase. The system can be deployed to git-pages without the use of any backend. For LMS integration, a middleware backend is required by Learning Tools Interoperability (LTI). Our hosted backend server can be used or the middleware can be launched independently. OATutor is Section 508 accessibility compliant.

Quick clone and deploy notebook example

Quick content authoring notebook example

Sign-up for our mailinglist

Introduction to OATutor press article

Paper

To credit this system, please cite our CHI'23 paper:

Zachary A. Pardos, Matthew Tang, Ioannis Anastasopoulos, Shreya K. Sheel, and Ethan Zhang. 2023. OATutor: An Open-source Adaptive Tutoring System and Curated Content Library for Learning Sciences Research. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI '23). Association for Computing Machinery, New York, NY, USA, Article 416, 1–17. https://doi.org/10.1145/3544548.3581574

@inproceedings{pardos2023oat,
  title={OATutor: An Open-source Adaptive Tutoring System and Curated Content Library for Learning Sciences Research},
  author={Pardos, Z.A., Tang, M., Anastasopoulos, I., Sheel, S.K., Zhang, E},
  booktitle={Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems},
  pages={1--17},
  organization={Association for Computing Machinery},
  doi={https://doi.org/10.1145/3544548.3581574},
  year={2023}
}

Our new pre-print, reporting preliminary finding on learning gains and ChatGPT-based hint evaluation: https://arxiv.org/abs/2302.06871

Creative Commons Attribution

The content submodule repository includes three creative commons (CC BY) textbooks worth of algebra problems with tutoring supports in the form of hints and scaffolds, authored and edited by the OATutor project, also released under CC BY 4.0.

1. A subset of problems are derivatives of OpenStax: Elementary Algebra by OpenStax, used under CC BY 4.0
2. A subset of problems are derivatives of OpenStax: Intermediate Algebra by OpenStax, used under CC BY 4.0
3. A subset of problems are derivatives of Openstax: College Algebra by OpenStax, used under CC BY 4.0

Requirements

The installation assumes that you already have Git, Node.js, and npm installed.

Installation

git clone --recurse-submodules https://github.com/CAHLR/OATutor.git
cd OATutor

Dependencies

npm install

You may use an alternative package manager such as yarn or pnpm.

Local Development Server

npm run start

Building & Deployment

npm run build
npx serve -s build

The build folder now contains all of the static assets necessary to make a complete deployment on a static site hosting provider.

[Optional] Firebase Setup

OATutor can use Firebase to persistently store log data.

1. Navigate to the Firebase website
2. Add new project. Configure it as you wish (the options are not important for setup)
3. Click on Database and then create database. Start in test mode, leave the cloud location as is
4. Click on Project settings --> general. Copy SDK Setup & Configuration --> Config
5. Put configuration in src/config/firebaseConfig.js

Features:

1. Scaffolding/hint system - modularize the type of help
2. Adaptive item selection - Pick items to master weakest skills (isolate skills to master individually)
3. Centralized skill model - src/content-sources/*/skillModel.json
4. Data logging/collection - Based off of the Cognitive Tutor KDD dataset.
5. User login/registration - JSON Web Tokens

Technologies Used

• Frontend: ReactJS
  • Theme: Material UI
  • Database: localForage (localStorage, WebSQL, IndexedDB)
  • Deployment: Github Actions to Github Pages
  • [Optional] Logging: Firebase (Cloud Firestore)
• Middleware: ExpressJS
  • Database: Level-DB
• Offline Computation/Iteration:
  • Python (dAFM Machine Learning algorithm)

Project Structure:

Code for this project is located in the src directory.

. (src)

• App.css: Top level style sheet, contains colors for headers/logo
• App.js: Top level script, creates firebase object. Sets up the application context.
• index.js: Renders App.js

./models/BKT

• BKT-brain.js: Contains update function that implements the standard BKT update algorithm.
• problem-select-heuristics/*.js: These files contain a configurable heuristic for adaptive problem selection. The default heuristic iterates across the problems and chooses the one with the lowest average probability of mastery across all of its knowledge components, but this can be changed to any heuristic depending on the problem information and the previously completed problems.

./components

• Firebase.js: Class with methods to read/write to Firebase (Cloud Firestore).

./components/problem-layout

• HintSystem.js: Expandable panel component to display all the hints.

• HintTextbox.js: Textbox for scaffold types of hints with answers.

• Problem.js: The "problem" component created in App.js. The component is initailized with a "problem" object as one of its props. It then creates a series of "ProblemCards" in const parts (currently keys for ProblemCard components are random values, should be UUIDs in the future).

  The answerMade function is passed to each ProblemCard component and is called whenever an answer is submitted to a ProblemCard. This enables the Problem component to update the knowledge component variables after each answer and transition to the next problem after all answers are correct.

• ProblemCard.js: This component displays an individual card, updates the input text field to display the result of an answer attempt, and calls the Problem.js answerMade function when the submit button is pressed.

These two files heavily rely on Material-UI syntax (eg. all the useStyles and classes references). Check their website for more info on this syntax.

• problemCardStyles.js: This file contains all the styles for ProblemCard.js

./platform-logic

• checkAnswer.js: Function to check answers. 3 different types of answers are supported: Algebraic, String, Numeric. Algebraic will simplify numeric expressions, numeric checks numeric equivalence, string requires answers to exactly match.

• Platform.js: Creates top "AppBar" and presents the first "problem" (everything under the app bar is part of the problem component). Also imports all of the problem files and stores them in const problemIndex. The function nextProblem is used to determine the next problem to be displayed.

• Firebase.js: Class with methods to read/write to Firebase (Cloud Firestore).

• renderText.js: Method called to render text. Fills in dynamic text generation.

./content-sources [Configurable]

• Each sub-folder can be considered its own isolated content source.
  • oatutor contains OATutor-curated content but can be removed if the content is not being used.
• See the Content Source section for more details.

Markdown Support

• All \ must be escaped as \\ because values are strings
• Wrap Latex in $ for inline LaTeX
• Newlines can be created with \n, escaped as \\n

./config [Configurable]

• config.js: Central place where options can be configured. Also includes function to get the treatment id given a userID, imports all appropriate treatments (Ex. BKTParam, HintPathway, Adaptive Problem selection heuristic)

• firebaseConfig.js: File containing firebase set up configuration.

./tools [Optional]

Data parsing and spreadsheet populating tools are stored in src/tools. If you would like to use any of the tools in this directory, the following steps must be taken to ensure Firebase access.

1. Navigate to the Firebase website
2. Click on the project made in Firebase Setup
3. Click on Project settings --> service accounts. Generate a new Node.js private key.
4. Put that private key in src/tools/service-account-credentials.json
5. cd src/tools
6. npm install

populateGoogleSheets.js

This tool allows you to sync the feedback received from your website to a Google Spreadsheet of your choosing.

1. Navigate to the Firebase website
2. Click on the project made in Firebase Setup
3. Click on Project settings --> service accounts
4. Create another service account and save its private key to src/tools/sheets-service-account.json
5. Copy the email address for the service account
6. Share the target spreadsheet with that email address and give them editor access
7. Create .env.local in src/tools and add this line SPREADSHEET_ID=YOUR_SPREADSHEET_ID_HERE
8. If your school runs on a quarter system, you may change the first line in common/global-config.js to QUARTER

From the src/tools directory, node populateGoogleSheets.js

firebaseExportCSV.mjs

This tool requires no additional set up, and allows you to download a CSV of the Firebase collections.

./util

Contains common helper methods for the frontend React app.

Listeners

Mouse Logging

• Size of screen is the size of the scrollable browser canvas
• Wrap <Platform /> with the following in App.js

<ReactCursorPosition onPositionChanged={(data) => {
    if (DO_LOG_MOUSE_DATA) {
        this.firebase.mouseLog(data);
    }
}}>
    <Platform props_here/>
    <ReactCursorPosition/>
    }}>

Focus Logging

• Records the times at which the user enters/leaves the tab
• Uses efficient Firestore storage by partitioning data into collections
• Turn off by editing config.js setting

  DO_FOCUS_TRACKING = false;

Adding Listeners

1. Install the React component for the listener.
2. Wrap the Platform component with the listener. The listener must take a prop that is a function to log data that it receives.
3. In /ProblemLogic/Firebase.js add a new function to log the new type of data. Create a new collection for this listener logs.
4. Configure buffer size and granularity of logging

Content Sources

• OATutor can support multiple content sources simultaneously, compartmentalizing courses, lessons, and problems of different topics
• Currently, the oatutor content source is included in this repository as a git submodule to enable separate versioning
• However, content sources can be copied in as entire folders as well and committed to this repository

Content Source Directory Structure

./content-pool

• Each problem is contained in its own folder.
• Problems can contain steps which are contained in their own sub-folder.
• Steps can contain hints which are stored as pathways in the tutoring sub-folder.
• All problems are pre-processed before being ingested by the frontend platform. All problems are accumulated in the generated/processed-content-pool/[source_name].json file prior to each run or build.

./bkt-params

• bktParams.js: Contains the mastery, transit, slip, and guess probabilities for each skill. Used by the BKT model.

Meta / Tagging files

• skillModel.json: This file contains all the problem to skill mappings. The format is as follows:

  {
  problemID1a: ['skill1', 'skill2'],
  problemID1b: ['skill2', 'skill3'],
  // ...
  }

• coursePlans.json: This file contains all of the courses relating to this content source. Each course specified in this file is associated with multiple lessons. See the creating lesson plans section for more details.

Adding a problem to the Content Pool

1. Create a folder in ./content-pool for that problem (Ex. circle1)
2. Create a metadata json file for that problem id (Ex. circle1.json)
3. Create a folder called figures if the problem has image figures
4. Create a sub-folder for each problem step (Ex. circle1a, circle1b)
5. In each sub-folder, create a json file for that problem step (Ex. circle1a.json)
   1. Ensure that the step name matches the folder name
6. Create a sub-folder within the step's sub-folder called tutoring
7. Place each hint pathway within the folder (Ex. circle1aDefaultPathway.json)
8. In ./skillModel.json, tag each problem with the appropriate skills
9. If the skill does not already exist in bktParams and you are using the BKT model, add its BKT parameters in the appropriate bkt-params/bktParams.json files

Types of problems

• TextBox : Box for student to enter answer. 3 different types of answers are supported: Algebraic, String, Numeric. Algebraic will simplify numeric expressions, numeric checks numeric equivalence, string requires answers to exactly match.
• MultipleChoice: List choices as choices: ["Choice A", "Choice B"], must have answerType: "string"

Example Directory Structure

content-sources/
└── oatutor [submodule]/
    ├── bkt-params/
    │   ├── bktParams1.json
    │   └── bktParams2.json
    ├── content-pool/
    │   ├── circle1/
    │   │   ├── circle1.json
    │   │   └── steps/
    │   │       ├── circle1a/
    │   │       │   ├── circle1a.json
    │   │       │   └── tutoring/
    │   │       │       └── circle1aDefaultPathway.json
    │   │       └── circle1b/
    │   │           ├── circble1b.json
    │   │           └── tutoring/
    │   │               └── circle1bDefaultPathway.json
    │   └── slope1/
    │       ├── slope1.json
    │       └── ...
    ├── coursePlans.json
    └── skillModel.json

Example Problem File

{
    "id": "circle1",
    "title": "Buying a Big Rug",
    "body": "Bob wants to surprise Alice by buying a new rug for their living room. Their living room is 28 feet wide and 20 feet long. To further surprise Alice, Bob wants to buy the biggest circular rug that will fit.",
    "variabilization": {},
    "oer": "https://example.com",
    "lesson": "1.1 Circle Radius",
    "courseName": "Geometry"
}

Example Step File

{
    "id": "circle1a",
    "stepAnswer": [
        "10"
    ],
    "problemType": "TextBox",
    "stepTitle": "1. Maximum Radius",
    "stepBody": "What is the maximum radius of a circular rug that will fit in the room?",
    "answerType": "numeric",
    "variabilization": {}
}

Example Hint Pathway File

[
    {
        "id": "circle1a-h1",
        "title": "Size of the room",
        "text": "Consider the shape of the room and the limitations this has on the radius of the rug.",
        "type": "hint",
        "dependencies": [],
        "variabilization": {}
    },
    {
        "id": "circle1a-h2",
        "title": "Constricting dimension",
        "text": "The length (20ft) creates limitations on the size of the circle. What is the maximum diameter that the circle can be?",
        "hintAnswer": ["20"],
        "problemType": "TextBox",
        "answerType": "numeric",
        "type": "scaffold",
        "dependencies": [0],
        "variabilization": {}
    },
    {
        "id": "circle1a-h3",
        "title": "Solution",
        "text": "Recall that the radius is half the diameter, so $r = \\frac{d}{2} = 10$",
        "type": "solution",
        "dependencies": [1],
        "variabilization": {}
    }
]

Using OATutor custom markdown parser for images and LaTeX

{
    "id": "pythag1", //Substeps will be in the form problem.id + 'a' and so on
    "title": "Car Forces",
    "body": "A %CAR% experiences three horizontal forces of -3.10N, 1.70N and -4.00N. It also experiences three vertical forces of -4.30N, 0.20N and 4.20N. \\n Round all answers to the hundredths place. \\n##triangle.png## ",
    "variabilization": {}
}

Creating Lesson Plans

{
    id: "lesson1",
    name: "Lesson 1",
    topics: "Pythagorean Theorem",
    allowRecycle: true,
    learningObjectives:
    {
        pythagorean: 0.95
    }
}

// src/App.js
this.userID = generateRandomInt().toString();
getTreatment = () => {
    return this.userID % 2;
}

getTreatmentObject = (targetObject) => {
    return targetObject[this.getTreatment()]
}

// src/App.js
import { heuristic as lowestHeuristic } from './models/BKT/problem-select-heuristics/problemSelectHeuristic1.js';
import { heuristic as highestHeuristic } from './models/BKT/problem-select-heuristics/problemSelectHeuristic2.js';

...
const treatmentMapping = {
    heuristic: {
        0: lowestHeuristic,
        1: highestHeuristic
    },
}

// src/App.js
import bktParams1 from './content-sources/oatutor/bkt-params/bktParams1.json';
import bktParams2 from './content-sources/oatutor/bkt-params/bktParams2.json';

...
const treatmentMapping = {
    bktParams: {
        0: cleanObjectKeys(bktParams1),
        1: cleanObjectKeys(bktParams2)
    },
}

// src/App.js
const treatmentMapping = {
    hintPathway: {
        0: "DefaultPathway",
        1: "YourNewPathwayHere"
    }
}

content-sources/
└── oatutor [submodule]/
    ├── content-pool/
    │   ├── circle1/
    │   │   ├── circle1.json
    │   │   └── steps/
    │   │       ├── circle1a/
    │   │       │   ├── circle1a.json
    │   │       │   └── tutoring/
    │   │       │       ├── circle1aDefaultPathway.json
    │   │       │       └── circle1aYourNewPathwayHere.json  <--- Add your new pathway here
    │   │       └── circle1b/
    │   │           ├── circble1b.json
    │   │           └── tutoring/
    │   │               ├── circle1bDefaultPathway.json
    │   │               └── circle1bYourNewPathwayHere.json  <--- Add your new pathway here
    │   └── slope1/
    │       ├── slope1.json
    │       └── ...
    ├── ...