The similarity between example word pair is 0.85 and greater than 0.80 ,So this pair must considered a match and I expect that the final GeneralizedJaccard output for two near-duplicate sentences to be equal to 1 but it's 0.97:
import py_stringmatching as sm
str1='All tokenizers have a tokenize method'
str2='All tokenizers have a tokenize methods'
alphabet_tok_set = sm.AlphabeticTokenizer(return_set=True)
gj = sm.GeneralizedJaccard(sim_func=sm.Levenshtein().get_sim_score, threshold=0.8)
print(gj.get_raw_score(alphabet_tok_set.tokenize(str1),alphabet_tok_set.tokenize(str2)))
>>0.9761904761904763
I'm using GeneralizedJaccard from Py_stringmatching package to measure the similarity between two strings. According to this document:
For example for word pair 'method' and 'methods' we have:
The similarity between example word pair is 0.85 and greater than 0.80 ,So this pair must considered a match and I expect that the final GeneralizedJaccard output for two near-duplicate sentences to be equal to 1 but it's 0.97:
So what is the problem?!