search

tesseract-ocr / langdata

Source training data for Tesseract for lots of languages
Apache License 2.0
826 stars 886 forks source link

Language request: Kurdish-Kurmanji #124

Closed brandones closed 5 years ago

brandones commented 6 years ago

It's the most popular dialect of Kurdish, written in a Latin alphabet with a few diacritics.

Shreeshrii commented 6 years ago

Please try the current kur_ara traineddata from tessdata_fast and tessdata_best, it seems to be incorrectly labelled and might be for Kurdish written in Latin script.

Provide the feedback after testing here, so that required changes can be made.

On Wed 11 Apr, 2018, 11:23 PM Brandon Istenes, notifications@github.com wrote:

It's the most popular dialect of Kurdish, written in a Latin alphabet with a few diacritics.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/tesseract-ocr/langdata/issues/124, or mute the thread https://github.com/notifications/unsubscribe-auth/AE2_o0Yhlvi9gBZkJvhwX-ByPPDaPJSTks5tnkL8gaJpZM4TQeaH .

brandones commented 6 years ago

The data in both kur and kur_ara is in Arabic script, neither is in a Latin script.

Shreeshrii commented 6 years ago

I am not referring to the langdata repo which has not been updated for 4.0.0.

There are traineddata files for kur_ara in tessdata_best and tessdata_fast.

You can use combine_tessdata to unpack the file and look at the unicharset in it.

Use dawg2wordlist to convert the words dawg to the wordlist.

Of course, you could try to recognise any image with those traineddata files and check output.

On Wed 11 Apr, 2018, 11:42 PM Brandon Istenes, notifications@github.com wrote:

The data in both kur and kur_ara is in Arabic script, neither is in a Latin script.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/tesseract-ocr/langdata/issues/124#issuecomment-380545936, or mute the thread https://github.com/notifications/unsubscribe-auth/AE2_o_TbxSvm63_vCN0BYWRRUOb7FV0wks5tnkdwgaJpZM4TQeaH .

Shreeshrii commented 6 years ago

See https://github.com/tesseract-ocr/tessdata_best/issues/23

On Wed 11 Apr, 2018, 11:46 PM ShreeDevi Kumar, shreeshrii@gmail.com wrote:

I am not referring to the langdata repo which has not been updated for 4.0.0.

There are traineddata files for kur_ara in tessdata_best and tessdata_fast.

You can use combine_tessdata to unpack the file and look at the unicharset in it.

Use dawg2wordlist to convert the words dawg to the wordlist.

Of course, you could try to recognise any image with those traineddata files and check output.

On Wed 11 Apr, 2018, 11:42 PM Brandon Istenes, notifications@github.com wrote:

The data in both kur and kur_ara is in Arabic script, neither is in a Latin script.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/tesseract-ocr/langdata/issues/124#issuecomment-380545936, or mute the thread https://github.com/notifications/unsubscribe-auth/AE2_o_TbxSvm63_vCN0BYWRRUOb7FV0wks5tnkdwgaJpZM4TQeaH .

brandones commented 6 years ago

Oh interesting! I will give that a try, thank you.

brandones commented 6 years ago

Yes, it's definitely trained on Kurmanji. It's doing a great job OCRing Kurmanji text. This is wonderful, thank you!

Shreeshrii commented 6 years ago

Thank you for your feedback.

We need to change the traineddata name so that it correctly reflects the language.

What would you suggest?

kur kur_lat

Any other official language code for kurmanji.

brandones commented 6 years ago

Sure! My recommendations:

Don’t use kur for anything.

Depends on if you’re more interested in the script or the dialect. There are a few Kurdish languages using each of the scripts, the Latin-based one and the Persian-based one. Historically Cyrillic and Armenian scripts have been used also. I’m guessing the language is itself important though.

The ISO 639-3 code for Kurmanji (aka Northern Kurdish) is kmr.

It wouldn’t be at all unreasonable to prefix it as kur_kmr, and refer to it as Kurdish Kurmanji. Or just list kmr as Kurdish Kurmanji.

Similarly, I’d recommend using ckb or kur_ckb for the one with the Arabic-based script, Sorani.

Shreeshrii commented 6 years ago

Thanks.

@jbreiden @theraysmith what is your recommendation?

At a minimum kur_ara should be changed as it doesn't have Arabic in it.

Shreeshrii commented 6 years ago

https://github.com/tesseract-ocr/tessdata_fast/issues/14#issuecomment-375881384

OK. I think we should follow the suggestion by @amitdo, since it is in line with the way tesseract names other languages.

amitdo commented 6 years ago

Shree, please send PRs.

brandones commented 6 years ago

In case it’s not clear, this choice impedes the development of Tesseract support for Zaza, Gorani/Horami, and Southern Kurdish, each of which will require quite different dictionaries.

Shreeshrii commented 6 years ago

@brandones Yes, that will require different dictionaries. As of now I am not sure exactly which language dictionary is being used.

I am attaching a zip file with two traineddatas for Kurdish in Arabic script - Sorani.

If you are familiar with the script, please review and provide feedback as to accuracy and also whether the word frequency list is appropriate. (or refer to someone who can provide that feedback). Thanks!

kur_ara.traineddata.zip

Shreeshrii commented 6 years ago

https://en.wikipedia.org/wiki/Kurdish_languages

ISO 639-3 kur – inclusive code Individual codes: ckb – Central Kurdish kmr – Northern Kurdish sdh – Southern Kurdish

ckb - Sorani (Arabic/Persian script). kmr - Kurmanji (Latin script)

Shreeshrii commented 5 years ago

@zdenop This issue can be closed as Kurdish-Kurmanji is available at

https://github.com/tesseract-ocr/tessdata_best/blob/master/kmr.traineddata https://github.com/tesseract-ocr/tessdata_fast/blob/master/kmr.traineddata

https://github.com/tesseract-ocr/langdata_lstm/tree/master/kmr

keyochali commented 4 years ago

I want to improve and work on Kurdish Kurmanji and Sorani for both Latin an Arabic scripts. how can I do that? what is needed to be done? how can I submit it?

Shreeshrii commented 4 years ago

Kurdish-Kurmanji is available at

https://github.com/tesseract-ocr/tessdata_best/blob/master/kmr.traineddata https://github.com/tesseract-ocr/tessdata_fast/blob/master/kmr.traineddata

https://github.com/tesseract-ocr/langdata_lstm/tree/master/kmr

You can test it to identify the improvements needed.

Shreeshrii commented 4 years ago

Sorani in Arabic script is not available in tessdata_best and tessdata_fast.

https://github.com/tesseract-ocr/langdata_lstm/tree/master/kur has minimal language data available for it.

You will need to identify unicode fonts and training text for the same before running any training.

keyochali commented 4 years ago

thank you so much I just tested the kmr.traineddata

but for the sorani in Arabic script where can I train it? what data is needed except for the text?

I mean do I need to do boxing some images? can you show me an example of training in tessercat? is there any documentation for training it?

Shreeshrii commented 4 years ago

https://github.com/tesseract-ocr/tesseract/wiki/TrainingTesseract-4.00

https://github.com/Shreeshrii/tess4training

https://github.com/tesseract-ocr/tesstrain

keyochali commented 4 years ago

wonderful thanks a lot

keyochali commented 4 years ago

how can I do it in windows when tesseract is installed it has the training tools with it how can I use them is there any python script to do it? what is a textilne(a word? or the line?)