Swedish primary school curricula (Läroplaner för grundskolan) in digital format.
We use optical character recognition (OCR) to transform curricula in image format into text. For each curriculum we construct datasets at the paragraph, sentence, and word levels.
In ./analysis/output/
you will find the following for the Läroplan for year YYYY
:
lgrYYYY_counts.csv
: counts of individual wordslgrYYYY_paragraphs.csv
: individual paragraphs with page of appearancelgrYYYY_sentences.csv
: individual sentences with page and paragraph of appearanceIn the folder ./example/
we provide an illustration on how to use the data to search for, and plot, counts of a desired set of words.
You may want to compile some or all of the code yourself. To do so, you need the following prerequisites.
A python
compiler version 3.8 or above.
requirements.txt
. See quick start belowgit
for version control.
git-lfs
for versioning large filesThe software Tesseract-OCR
:
For Mac, you also need to install poppler for Mac
in order to use the python package pdf2image
, see here.
Users may notice small differences in output across machines due to the way images are processed.
./run.py
is a python script used to build the entire repository.
./raw/
contains pdf files of Swedish curricula (Läroplaner).
/orig/
contains curricula in pdf format/docs/
contains documentation./derived/
contains code that runs the OCR and cleans its output.
/code/make_images.py
transforms curricula in pdf format to separate jpg files, the jpg files are not included in the repo/code/ocr.py
reads the jpg files into text/code/clean.py
cleans the text files produced by the OCR./analysis/
constructs datasets from the digitized text of each curricula.
/code/make_data.py
transforms the cleaned text files into paragraph- and sentence-level datasets/code/count.py
counts all appearances of words in each curriculaEach folder hosts an /output/
subfolder where output from each script is saved.
Clone the repository to your local machine.
# Using SSH
git lfs clone git@github.com:JMSLab/LaroplanOCR.git
# Using HTTPS
git lfs clone https://github.com/JMSLab/LaroplanOCR.git
Install dependencies. From the root of the repo run:
pip install -r requirements.txt
To compile the entire project, open the command-line and run
python run.py
You may also compile specific steps of the pipeline.
For example, python derived/code/make_images.py
will transform the pdf files of the curricula into jpg files.
We thank our dedicated research assistants for contributions to this project.