Suggestion: better handle combining diacritics - Githubissues

duydo / elasticsearch-analysis-vietnamese

Vietnamese Analysis Plugin for Elasticsearch

Apache License 2.0

509 stars 212 forks source link

Suggestion: better handle combining diacritics #34

Closed Trey314159 closed 7 years ago

Trey314159 commented 7 years ago

Combining characters (incluing diacritics and other characters in non-Latin scripts) cause tokens to split. Some examples from various scripts:

ضَمَّة — Arabic
বাংলা — Bengali
Михаи́л — Cyrillic + combining acute
दिल्ली — Devanagari
ಅಕ್ಷರಮಾಲೆ — Kannada
áa - Latin + combining acute (vs áa with precomposed character)
ଓଡ଼ିଆ — Oriya
தமிழ் — Tamil
తెలుగు — Telugu
อักษรไทย —Thai
g͡b — International Phonetic Alphabet

duydo commented 7 years ago

I close this issue, it will be fixed in new tokenizer #37