Closed dsvrsec closed 2 years ago
Hey @dsvrsec, I have mentioned this in my comments here: https://github.com/R1j1t/contextualSpellCheck/issues/59#issuecomment-811322409
The current logic for spelling correction is as follows:
provide spacy model: This will break the sentence into tokens. Now as this model is trained on a particular language (tweet specific models are also there) it knows the nuances
Check the token in the transformer model's vocab: If the token is not present consider it spelling error
Mask the OOV word and use the transformers model to predict words to replace mask
check the edit distance to see which is closest syntactically.
The default model (bert-base-cased)
Please let me know if you have any questions, also feel free to contribute!!
@R1j1t So it can work for any language if a spacy and a BERT LM is available right?
Hi here, actually I compared the output with pyspellchecker package, which is purely based on the edit distance, the results is almost same. Just wondering, if there are some grammar correct included is the bert model in use?
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
Can you please mention the methodology followed in detection and correction of mispelled words,may be at higher level,if possible, thank you