Shreeshrii
commented
3 years ago Please share your groundtruth files so that we can try to reproduce the issue.
Closed jwijffels closed 3 years ago
Shreeshrii
commented
3 years ago Please share your groundtruth files so that we can try to reproduce the issue.
jwijffels
commented
3 years ago Thanks for the help. Here they are. Looking forward to your comments! nld-ground-truth.zip
Shreeshrii
commented
3 years ago I tested with the latest source of tesseract from github along with latest version of tesstrain repo. And, it seems to be working fine. See attached log file for the command given as:
nohup make training MODEL_NAME=nld START_MODEL=nld TESSDATA=$HOME/tessdata_best MAX_ITERATIONS=10000 > plot/nld.log &I noticed that you have very few training samples and they cover only a small subset of the nld unicharset.
Wrote unicharset file data/nld/my.unicharset
merge_unicharsets data/nld/nld.lstm-unicharset data/nld/my.unicharset "data/nld/unicharset"
Loaded unicharset of size 151 from file data/nld/nld.lstm-unicharset
Loaded unicharset of size 61 from file data/nld/my.unicharset
Wrote unicharset file data/nld/unicharset.
find data/nld-ground-truth -name '*.lstmf' | python3 shuffle.py 0 > "data/nld/all-lstmf"
+ head -n 178 data/nld/all-lstmf
+ tail -n 20 data/nld/all-lstmfI suggest that you use the latest tesseract and tesstrain and add more training samples.
jwijffels
commented
3 years ago Thanks for taking the time, I indeed used the default tesseract on Ubuntu 20.04, let me try the github version. Indeed the training data does not cover all letters in the tessdata_best nld model.
Shreeshrii
commented
3 years ago See the CER plots with your current training data. Between 1000-1600 iterations seem to be good enough. After that it seems to be overfitting.
You can test with the finetuned traineddata in the attached tgz file.
jwijffels
commented
3 years ago Thanks for the graphs, I'm compiling now tesseract from source, in order to see if that solved the core dump.
And the model as well - thanks! Wasn't expecting this help. Indeed seems to work a bit better on the Latin numbers compared to the original nld model but I sure need to see that it doesn't overtrain. I made a model using only that ground-truth data and it sure did overtrain. The graphs are really helpfull, I'll also try to make these.
I'll get back to this is tesseract is compiled from source on the Ubuntu 20.04 machine which I'm running the training on. Many thanks already!
FYI. This is typically the reason why I wanted to retrain the model, Latin numbers, which we need in the analysis to see how many taxes people payed in the 18-19th century.
Shreeshrii
commented
3 years ago Have you tried using multiple language recognition, for example -l nld+eng, to see if the Latin numbers are better recognized?
jwijffels
commented
3 years ago Wasnt aware of this. Will try this out further iin 2 weeks.
stale[bot]
commented
3 years ago This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
I'm trying to improve the nld model such that it better predicts Latin numbers. The type of images I have are these, containing some Dutch and some Latin numbers.
I've taken the nld.traineddata model from https://github.com/tesseract-ocr/tessdata_best and put the ground truth in a folder data/nld-ground-truth When making the training, I get a core dump, shown below. Can someone help in fixing this. This is my environment: