Invisible characters and MORFOLOGIK

languagetool-org / languagetool

Style and Grammar Checker for 25+ Languages

https://languagetool.org

GNU Lesser General Public License v2.1

12.32k stars 1.39k forks source link

Invisible characters and MORFOLOGIK #3335

Open ghost opened 4 years ago

ghost commented 4 years ago

Many common words appear to be in the list of words so far unknown to MORFOLOGIK. But these are in fact known. After analysis, it appears there are invisible characters in those words. Could MORFOLOGIK ignore these? Or could a different rule report 'INVISIBLE CHARACTERS IN CORRECT WORD' e.g. ?

danielnaber commented 4 years ago

Could you post some examples?

ghost commented 4 years ago

Only in a few weeks time. I guess I will have to hex-dump them to see what is wrong. What I noticed today was hex 00 and hex 03 at the start of the word, but I can not check more now, since the computer is completely off now (holiday). When I return, I will be able to produce a (huge) list.