Closed justinbhopper closed 3 years ago
In the current logic, spell corrector pipeline can be divided as follows:
So, for the 3rd step i.e. Selection I am using Levenshtein distance. It is a metric to calculate difference between 2 strings (here misspell and candidate). So when you see score_spellCheck
it gives you the candidate word and their probability based on BERT prediction (Step 2 above). But step 3 only uses Levenshtein distance to pick a candidate. Hence the outcome_spellCheck
gives the output not based on probability from the model but from edit distance.
So, your doubt is correct, we could use probability as a metric for Step 3. The logic I finalized and implemented after reading from literature available. Please feel free to suggest some other implementation, or paper for reference. This package is still under development and before releasing v1.0
Feel free to open a PR if you would like to contribute!
@R1j1t I think using the Levenshtein distance is the right approach. I should probably just be using suggestions_spellCheck
to list out the closest candidate. I mistakenly thought the score represented the Levenshtein distance.
@R1j1t One last question - is it possible to get the Levenshtein distance value for the suggested candidate?
Great! Regarding your question to get the Levenshtein distance, it is not recorded in any spaCy extension.
But this package relies on https://github.com/roy-ht/editdistance to calculate the metric. So maybe you can use it in your code. https://github.com/R1j1t/contextualSpellCheck/blob/f8cbeb8a7d5dc085f9f8cc5d27d390848d2df274/contextualSpellCheck/contextualSpellCheck.py#L397
If you want to help in adding an extra spacy extension (suggestion_edit_distance
) please open a PR! You can check the code for reference here:
https://github.com/R1j1t/contextualSpellCheck/blob/f8cbeb8a7d5dc085f9f8cc5d27d390848d2df274/contextualSpellCheck/contextualSpellCheck.py#L139-L148
Describe the bug I am not sure if I am using this wrong, but it seems like the data in
score_spellCheck
does not match theoutcome_spellCheck
final output. I am assuming thatoutcome_spellCheck
is a result of performing the spelling corrections with the highest probability score?To Reproduce
Output:
Version (please complete the following information):
Additional information Note how the
outcome_spellCheck
has a good final output. However,score_spellCheck
has corrections that are wildly unexpected. They seem more like synonyms than actual spelling corrections (e.g. "relationships" is no where close to the spelling "habbits"). Note how "depressed" got a miserable 0.02555 score, listed way below other corrections that are much farther from the original word.