search

kermitt2 / entity-fishing

A machine learning tool for fishing entities
http://nerd.readthedocs.io/
Apache License 2.0
244 stars 24 forks source link

Training entity-fishing for other languages #93

Open rautant opened 5 years ago

rautant commented 5 years ago

Can the training instructions (https://nerd.readthedocs.io/en/latest/train.html) be used to train models in languages that are currently not supported? Or are they just for retraining with supported languages? I would like a have entity-fishing for Finnish language, how could I do that? I noticed that there is a Japanese branch (https://github.com/kermitt2/entity-fishing/tree/japanese).

lfoppiano commented 5 years ago

Hi @rautant, in princiiple to add a new language there is a preprocessed step non yet documented (it requires some more review before).

However regarding Finnish (and in general the condition to have or not a new supported language) the coverage in Wikipedia (number of articles) is too low. As you can see here: https://meta.wikimedia.org/wiki/List_of_Wikipedias#1_000_000+_articles fininsh has less than 500000 articles, which is likely to output poor disambiguation performances.

rautant commented 5 years ago

That's too bad. It is mentioned in the 'Train and evaluate' section that only a sample of documents is used for training. How big is this sample?

kermitt2 commented 5 years ago

Hello! Another issue with Finnish is the super rich morphology of the language, we would probably need a dedicated morphological analyzer to get enough mentions and candidates prior to disambiguation.

Usually 5000 wikipedia articles are used for training the ranker model, and 500 for the selection model.

rautant commented 5 years ago

It is certainly true that Finnish is very morphologically rich, but there are existing solutions for this already. I have tried FinnPos (https://github.com/mpsilfve/FinnPos) for lemmatization and the results seem great. So you could change all words to a base form before using NERD, maybe even treat compound words. If you publish documentation for training in other languages one day I would still consider trying it.

kermitt2 commented 5 years ago

There are solutions for sure, my point was rather that for supporting Finnish it won't be just importing the fiwiki data and retraining. It has an impact on the current pipeline/architecture both on the side of parsing and loading all wikipedia data (label statistics are on word-form level, not for lemma, so we need to lemmatize all the fiwiki articles), and on the side of processing incoming text (as we need to lemmatize text).