search

drahnr / cargo-spellcheck

Checks all your documentation for spelling and grammar mistakes with hunspell and a nlprule based checker for grammar
Apache License 2.0
323 stars 32 forks source link

hunspell stumbles over copyright symbol #281

Open drahnr opened 2 years ago

drahnr commented 2 years ago

Describe the bug

Encountered error Utf8Error { valid_up_to: 2, error_len: Some(1) } returned from Hunspell_suggest(handle, ["\"\\xc2\\xa9\""]): 0: "\xc2\x80\x93"
[2022-09-16T10:03:50Z DEBUG hunspell] © --{suggest}--> []

To Reproduce

Steps to reproduce the behaviour:

  1. A file containing ©
  2. Run cargo spellcheck file.rs
  3. ...

Expected behavior

Handle or ignore, currently hunspell-rs is hacked to print an error.

Screenshots

Please complete the following information:

drahnr commented 2 years ago

CC @lopopolo that was the issue at hand, the suggestion should accept 0xC2 0xA9 as valid since it is valid itself, but returns garbage suggestions instead. It's still present in 0.12.2 but will only be a verbose message that will be handled in the next release.

lopopolo commented 2 years ago

Oh awesome. That's in the generated headers of the Unicode files. Great catch @drahnr and thanks for debugging!

drahnr commented 2 years ago

@lopopolo I realized you have custom dict, with a - - removing the single char items from the list resolves the issue. It seems that trips the parser and makes it's way into the lut inside hunspell and then surfaces with some byte sequences.

lopopolo commented 2 years ago

I just tried this workaround and that seemed to work! I get no warning messages in cargo-spellcheck 0.12.1.

drahnr commented 2 years ago

The core issue is due to the fact that Hunspell uses the encoding used in the affix file, for the dictionaries as well. For en_us.aff (both builtin and Fedora 36) this was latin-1 encoding rather than utf-8. Hunspell then treats all inputs of encoding equiv to the affix file, implicitly by the used prefix tree.

Solution would be to use i.e. encoding_rs to re-encode the dictionaries to UTF-8 and only afterwards feed them to Hunspell or reject all encodings set besides utf-8.