understanding similitude differences between extract, cdist and extractOne

[abubelinha]

I am trying to understand by example how the different RapidFuzz methods work. My real use case is matching names column in a fuzzy dataframe, against another dataframe of name choices coming from a database, to extract the closest name in DB and its associated columns -index, parent, family-.

I looked into some examples in repository and tried to apply them to my sample data. See code below. I have some questions about the output:

1. Why does extract() return an empty list for some of the explicit scorer values I tried? (fuzz.ratio and fuzz.QRatio)
2. Which scorer should I use in first test, so similitudes are not the same? (can't understand why all matches get a 90.0 score)
3. Why 2nd and 3rd example produce a 71.42857 similitude of 'Soliva sessilis' against 'Soliva sessilis Ruiz & Pav.', whereas in 1st example the output is 90, as I said above?

Thanks a lot in advance @abubelinha

My code:

    from rapidfuzz import fuzz,process
    from rapidfuzz.process import cdist, extract

    if True: # single string tests:
        x = 'Soliva sessilis'
        print("\n1. RapidFuzz extract() using different scorers for matching '{}' against different strings:\n".format(x))
        choices = ['Soliva sessilis auct., non Ruiz & Pav.', 'Soliva sessilis Ruiz & Pav.', 'Soliva']
        print("String list (choices): ",choices)
        result = extract(query = x, choices = choices, score_cutoff=90)
        print("\n- Result with no scorer (default scorer):\n",result)
        for sco in ['fuzz.ratio','fuzz.WRatio','fuzz.QRatio']:
            try:
                result = extract(query = x, choices = choices, score_cutoff=90, scorer=eval(sco))
                print("\n+ Result when explicitly passing scorer={}:\n".format(sco),result)
            except Exception as e:
                print("\n* Error when explicitly passing scorer={}:\n".format(sco),str(e))
    if True: # dataframe matching tests:
        dfchoices = pd.DataFrame(data = [
                [1001,'Soliva sessilis auct., non Ruiz & Pav.',1006,'Asteraceae'],
                [1002,'Soliva sessilis Ruiz & Pav.',1006,'Asteraceae'],
                [1004,'Soliva pterosperma (Juss.) Samp.',1006,'Asteraceae'],
                [1005,'Soliva sp.',1006,'Asteraceae'],
                [1006,'Soliva',1006,'Asteraceae'],
                [1007,'Solanum L.',1007,'Solanaceae'],
                [1009,'Solanum tuberosum L.',1007,'Solanaceae'],
            ],
            columns=['id','name','parent','family']
        )
        dfuzdata = pd.DataFrame(data = [
                ['Soliva sessilis','USA'],
                ['Saliva sesilis','Brazil'],
                ['Solanun tuberosun','Perú-Bolivia'],
            ],
            columns = ['fuzname','origin']
        )
        print("\n","="*50)

        # TESTING cdist() as in https://github.com/rapidfuzz/RapidFuzz/issues/283#issuecomment-1291704481
        name_score = cdist(dfuzdata['fuzname'], dfchoices['name'])
        name_score = cdist(dfuzdata.iloc[0,:][['fuzname']], dfchoices['name'])
        print("\n2. cdist() as in https://github.com/rapidfuzz/RapidFuzz/issues/283#issuecomment-1291704481 outputs ",type(name_score),":")
        print(name_score)
        print("\n","="*50)

        print("\n3. process.extractOne as in https://github.com/rapidfuzz/RapidFuzz/issues/91#issuecomment-799548760 :\n")
        for fuzname in dfuzdata['fuzname']:
            choice,score,index = process.extractOne(fuzname, dfchoices["name"], scorer=fuzz.ratio, processor=None)
            print(" | ".join([str(x) for x in [fuzname,choice,score,index]]))

Output:

1. RapidFuzz extract() using different scorers for matching 'Soliva sessilis' against different strings:

String list (choices):  ['Soliva sessilis auct., non Ruiz & Pav.', 'Soliva sessilis Ruiz & Pav.', 'Soliva']

  - Result with no scorer (default scorer):
 [('Soliva sessilis auct., non Ruiz & Pav.', 90.0, 0), ('Soliva sessilis Ruiz & Pav.', 90.0, 1), ('Soliva', 90.0, 2)]

  + Result when explicitly passing scorer=fuzz.ratio:
 []

  + Result when explicitly passing scorer=fuzz.WRatio:
 [('Soliva sessilis auct., non Ruiz & Pav.', 90.0, 0), ('Soliva sessilis Ruiz & Pav.', 90.0, 1), ('Soliva', 90.0, 2)]

  + Result when explicitly passing scorer=fuzz.QRatio:
 []

 ==================================================

2. cdist() as in https://github.com/rapidfuzz/RapidFuzz/issues/283#issuecomment-1291704481 outputs  <class 'numpy.ndarray'> :
[[56.603775 71.42857  46.80851  64.       57.142857 40.       40.      ]]

 ==================================================

3. process.extractOne as in https://github.com/rapidfuzz/RapidFuzz/issues/91#issuecomment-799548760 :

Soliva sessilis | Soliva sessilis Ruiz & Pav. | 71.42857142857143 | 1
Saliva sesilis | Soliva sessilis Ruiz & Pav. | 63.41463414634146 | 1
Solanun tuberosun | Solanum tuberosum L. | 81.08108108108108 | 6

[maxbachmann]

for sco in ['fuzz.ratio','fuzz.WRatio','fuzz.QRatio']:
    result = extract(query = x, choices = choices, score_cutoff=90, scorer=eval(sco))

why would you use eval for this :fearful: You can simply iterate over the functions:

for scorer in [fuzz.ratio, fuzz.WRatio, fuzz.QRatio]:
    result = extract(x, choices, score_cutoff=90, scorer=scorer)

Why does extract() return an empty list from some of the explicit scorer values I tried? (fuzz.ratio and fuzz.QRatio)

process.extract filters out results if you pass a score_cutoff. In your case fuzz.ratio/fuzz.QRatio return a similarity below 90 for each element.

Which scorer should I use in first test, so similitudes are not the same? (can't understand why all matches get a 90.0 score)

A lot of the scorers use a partial match and since:

• 'Soliva sessilis' is a substring of 'Soliva sessilis auct., non Ruiz & Pav.'
• 'Soliva sessilis' is a substring of 'Soliva sessilis Ruiz & Pav.'
• 'Soliva' is a substring of 'Soliva sessilis'

This partial match always has a similarity of 100. fuzz.WRatio puts a weight of 0.9 on these partial matches, so you end up with a similarity of 90. E.g. fuzz.ratio or the distances in rapidfuzz.distance.* do not show this behaviour.

Why 2nd and 3rd example produce a 71.42857 similitude of 'Soliva sessilis' against 'Soliva sessilis Ruiz & Pav.', whereas in 1st example the output is 90, as I said above?

fuzz.ratio and fuzz.QRatio are based on the normalized Indel similarity. Not sure whether you have any specific questions into how they work.

[abubelinha]

Thanks a lot for your prompt answer and explanations @maxbachmann !

why would you use eval for this 😨 You can simply iterate over the functions

I knew, but I didn't know how to output functions' names in the next line of code. I was rushy to post the question so I used strings for the names, and eval() for this simple test.
Now I know I should use function.__name__.

In your case fuzz.ratio/fuzz.QRatio return a similarity below 90 for each element.

Thanks, So I need to lower my cutoff and also read more carefully the defaults. I misread and thought fuzz.ratio was the default scorer value for extract() so the 1st two lines of my test 1 output looked inconsistent to me (but its default is actually fuzz.WRatio, which explains their different output). Changed cutoff to 70 and this is the output of 1st test:

String list (choices):  ['Soliva sessilis auct., non Ruiz & Pav.', 'Soliva sessilis Ruiz & Pav.', 'Soliva']

- Result with no scorer (default scorer):
 [('Soliva sessilis auct., non Ruiz & Pav.', 90.0, 0), ('Soliva sessilis Ruiz & Pav.', 90.0, 1), ('Soliva', 90.0, 2)]

+ Result when explicitly passing scorer=fuzz.ratio:
 [('Soliva sessilis Ruiz & Pav.', 71.42857142857143, 1)]

+ Result when explicitly passing scorer=fuzz.WRatio:
 [('Soliva sessilis auct., non Ruiz & Pav.', 90.0, 0), ('Soliva sessilis Ruiz & Pav.', 90.0, 1), ('Soliva', 90.0, 2)]

+ Result when explicitly passing scorer=fuzz.QRatio:
 [('Soliva sessilis Ruiz & Pav.', 71.42857142857143, 1)]

That makes clear that I should use either choose fuzz.ratio or fuzz.QRatio for my use case.

fuzz.ratio and fuzz.QRatio are based on the normalized Indel similarity. Not sure whether you have any specific questions into how they work.

If it's not too advanced stuff yes, I'd like to know their difference (when using one or the other). But I guess I'll better get how it works by example.

With my sample data above they both produce the same similarity values. Any particular modification of my query that would produce differences between fuzz.ratio and fuzz.QRatio outputs? ... so I can figure out which of them suits better to what I want to do.

EDIT: I had also pasted my example data but I guess this is not related to the issue title. I better move it to discussions #347

[maxbachmann]

I misread and thought fuzz.ratio was the default scorer value for extract() so the 1st two lines of my test 1 output looked inconsistent to me (but its default is actually fuzz.WRatio, which explains their different output).

Yes this is a bit inconsistent, since cdist defaults to fuzz.ratio, while extract and extractOne default to fuzz.WRatio.

Any particular modification of my query that would produce differences between fuzz.ratio and fuzz.QRatio outputs?

They are pretty much exactly the same. The only difference between the two is the handling of empty strings.

>>> from rapidfuzz import fuzz
>>> fuzz.ratio("", "")
100.0
>>> fuzz.QRatio("", "")
0.0

In the past they had different defaults for the preprocessing function, but this was changed in v3.0.0.

[abubelinha]

Thanks a lot for explaining. Issue solved!