douglasrizzo / catsim

Computerized Adaptive Testing Simulator
GNU Lesser General Public License v3.0
124 stars 35 forks source link
assessment cat computerized-adaptive-testing irt item-response-theory simulations

Logo


Unit tests Test Coverage Latest Version Digital Object Identifier

catsim is a Python package for computerized adaptive testing (CAT) simulations. It provides multiple methods for:

These methods can either be used in a standalone fashion [1] to power other software or be used with catsim to simulate the application of computerized adaptive tests [2], given a sample of examinees, represented by their ability levels, and an item bank, represented by their parameters according to some logistic Item Response Theory model.

What's a CAT

Computerized adaptive tests are educational evaluations, usually taken by examinees in a computer or some other digital means, in which the examinee\'s ability is evaluated after the response of each item. The new ability is then used to select a new item, closer to the examinee\'s real ability. This method of test application has several advantages compared to the traditional paper-and-pencil method or even linear tests applied electronically, since high-ability examinees are not required to answer all the easy items in a test, answering only the items that actually give some information regarding his or hers true knowledge of the subject at matter. A similar, but inverse effect happens for those examinees of low ability level.

More information is available in the docs and over at Wikipedia.

Installation

Install it using pip install catsim.

Basic Usage

NEW: there is now a Colab Notebook teaching the basics of catsim!

  1. Have an item matrix;
  2. Have a sample of examinee proficiencies, or a number of examinees to be generated;
  3. Create an initializer, an item selector, a ability estimator and a stopping criterion;
  4. Pass them to a simulator and start the simulation.
  5. Access the simulator\'s properties to get specifics of the results;
  6. Plot your results.
from catsim.initialization import RandomInitializer
from catsim.selection import MaxInfoSelector
from catsim.estimation import NumericalSearchEstimator
from catsim.stopping import MaxItemStopper
from catsim.simulation import Simulator
from catsim.cat import generate_item_bank
initializer = RandomInitializer()
selector = MaxInfoSelector()
estimator = NumericalSearchEstimator()
stopper = MaxItemStopper(20)
Simulator(generate_item_bank(100), 10).simulate(initializer, selector, estimator, stopper)

Dependencies

All dependencies are listed on setup.py and should be installed automatically.

To run the tests, you\'ll need to install the testing requirements pip install catsim[testing].

To generate the documentation, install the necessary dependencies with pip install catsim[docs].

To ensure code is valid and formatted before submission, install the necessary development dependencies with pip install catsim[dev].

Compatibility

catsim is compatible and tested against Python 3.5, 3.6, 3.7, 3.8 and 3.9.

Important links

Citing catsim

You can cite the package using the following bibtex entry:

@article{catsim,
  author = {Meneghetti, Douglas De Rizzo and Aquino Junior, Plinio Thomaz},
  title = {Application and simulation of computerized adaptive tests through the package catsim},
  year = 2018,
  month = jul,
  archiveprefix = {arXiv},
  eprint = {1707.03012},
  eprinttype = {arxiv},
  journal = {arXiv:1707.03012 [stat]},
  primaryclass = {stat}
}

If you are looking for IRT item parameter estimation...

catsim does not implement item parameter estimation. I have had great joy outsourcing that functionality to the mirt R package along the years. However, since many users request packages with item parameter estimation capabilities in the Python ecosystem, here are a few links. While I have not used them personally, specialized packages like these are hard to come by, so I hope these are helpful.