stschubert / VPR_Tutorial

GNU General Public License v3.0
160 stars 19 forks source link

What are the assumptions on the `evaluation.createPR()` computation? #1

Closed doraemon96 closed 1 year ago

doraemon96 commented 1 year ago

When reading the createPR method, there's no information on the assumption of the inputs:

def createPR(S_in, GThard, GTsoft):
...

It would be nice to have something like:

def createPR(S_in, GThard, GTsoft):
    """
    Calculates the Precision and Recall at 100 equally spaced threshold values
    for a given similarity matrix S and ground truth matrices GThard and GTsoft.

    The matrices S_in, GThard and GTsoft should all have the same shape.
    The matrices GThard and GTsoft should be binary matrices, where the
    entries are only zeros or ones.
    The matrix S_in should have positive values only.
    """

Unsure if S_in needs to have only positive values, I believe that's expected of a similarity matrix, right?

Also something should be said about what those values represent. Is higher better or worst? One would say "the greater similarity of two objects, the greater the value of the measure" right?


edit: removed extra question (wrong)

stschubert commented 1 year ago

Thanks for your great comment.

As proposed by you, I added a description to the createPR-function.

Unsure if S_in needs to have only positive values, I believe that's expected of a similarity matrix, right?

S_in does not need to contain only positive values (in createPR and in general), because the similarities vary depending on the used descriptor comparison function. For example, when using cosine similarity, the values can vary between [-1,1].

Also something should be said about what those values represent. Is higher better or worst? One would say "the greater similarity of two objects, the greater the value of the measure" right?

Yes, higher similarities mean that two images are considered more similar.