Pyspellchecker corrects tokens that are not supposed to be corrected, and does not correct well some incorrect ones

[ljpetkovic]

Hello, I am using pyspellchecker (French dictionary) in order to correct this file.

However, after running the script below, I have noticed two types of problems (output file):

1. certain tokens (initially correct) were modified incorrectly (e.g. collection is corrected to colletion)
   
2. some tokens (initially incorrect) were not corrected properly (e.g. MÉMOÎRES should be corrected to MÉMOIRES):

   
   import re, glob
   from spellchecker import SpellChecker

entry = "6228000_r.txt" output = "6228000_r_corr.txt"

spell = SpellChecker(language='fr') text = open(entry).read()

do not correct the tokens containing the apostrophes (ex : l’empire, d’art, s’étend...)

r1 = re.findall(r"([lL]’\w+|[dD]’\w+|[sS]’\w+|[qQ]u’\w+|[cC]’\w+|[nN]’\w+|[jJ]’\w+|[Ll]orfqu’\w+|eft)",text)

tokenise the text with the pyspellchecker tokeniser

tokens = spell.split_words(text)

spell.word_frequency.load_words(r1) spell.known(r1) # the words l’empire, d’art, s’étend etc. are now in the dictionary of known words

print(tokens) misspelled = spell.unknown(tokens)

with open(output, "w") as f: for m in misspelled: corrected = spell.correction(m) text = text.replace(m, corrected)

f.write(c.replace('clafliques', 'classiques'))

f.write(text)

I cleaned up the original `.txt` file by replacing the single quote (`'`) with the apostrophe (`’`) in the words such as `l’empire`. 
I also tried to remove some other special characters (e.g. `^`, `&`, `<`, `>`), but the errors persist, and I cannot seem to locate exactly what causes them.

Do you have any idea how to resolve this issue?

[barrust]

It is likely an issue with the dictionary based on the data source used to build the dictionaries. I am not a French speaker and am not really able to validate the data in the dictionary. You can see the script used to build the dictionary here and there is a discussion on how it is done and how it could be improved in this discussion.

Any help on updating the dictionary would be helpful.