[en] spelling suggestions for words with hyphen

[jaumeortola]

We don't provide good spelling suggestions for words united by a hyphen (ex. Paris-Lonton).

This is related to word tokenization. In other languages, a hyphened word that is not in the dictionary is split in different tokens, and each word gets its own suggestions.

I'm not sure if a change in tokenization like this will affect other issues.

[jaumeortola]

Another example.

Should we implement this change, @udomai? Obviously, with sufficient testing.

[udomai]

Sounds good! Let's do it! We'll need a prohibited.txt to be able to exclude certain pairings if they are most likely to be a typo.

We'll need to keep an eye on how this impacts rules concerning hyphenated verb forms like "faisons-le", right?

[jaumeortola]

verb forms like "faisons-le",

My proposal is for English, not French.

In French is already done this way: <token>Paris</token><token>-</token><token>London</token> and <token>faisons</token><token>-le</token>.

[jaumeortola]

There are two methods to resolve this problem:

• Change the tokenizer. It will take longer.
• Improve the spelling rule. If there is no suggestion and the word contains a hyphen, then try to find suggestions for the split words.