Finalize OCR Accuracy Metrics

[ZoeLeBlanc]

Context

1. What's the issue that needs to be solved?
   • Currently the quality of OCR varies depending on if I take the time to identify and annotate paragraph separations.
   • I also may need to quantify the difference between the google ocr engine and transcribing the data
   • I need to know what's the error margin between ordered vs unordered OCR for various publications, and transcription vs ordered vs unordered
2. How do you plan to solve/revise?
   • develop ocr accuracy methodology and test it before deploying it across my data

     Associated Issues

To Do List

• [x] Test methods for document comparisons/similarity analysis
• [x] develop set of metrics for assessing ocr accuracy
• [x] finalize jupyter notebook or script to run these methods
• [x] collect one full magazine/newspaper/document to run eda on the methods

[ZoeLeBlanc]

Tested the following methods:

• ngrams / CountVectorizer with cosine similarity
• tfidf with cosine similarity
• doc2vec with cosine similarity
• leveshtein distance /edit distance
• sorensen distance
• smith waterman
• difflib sequence matcher
• gensim similarity martix
• spacy glove similarity
• jaccard distance

Keeping notes here

[ZoeLeBlanc]

I have everything now running in a jupyter notebook titled ordered and unordered text. It seems like spacy and smw alignment are a bit useless but I want to look at Arabic before I remove them.

[ZoeLeBlanc]

• [ ] test metrics with Arabic materials