Tokenization: Tackle german language

B3J4y / UniDisk

A Crawler to search for keywords and compare the score

MIT License

0 stars 1 forks source link

Open B3J4y opened 7 years ago

B3J4y commented 7 years ago

To search correctly in the german language a main challenge is to find tokens in compounded words like Pfandrücknahmeautomat

two main fields:

in solr indexed textes, there has to be a way to set it
for Stichwort there has to be an own implementation or maybe a work around with solr

B3J4y commented 7 years ago

A workaround could be:

Use DictionaryCompoundWordTokenFilterFactory
Create decompounded word from inserted Stichworte (maybe the user has to help via GUI)
Update a file like "germanwords.txt" in Solr dynamically if it hasn't already saved the word, with http://localhost:8983/solr/admin/cores?action=RELOAD&core=coreName&wt=json
Update the working solr-core
Searched Stichworte were also found in compounded words