search

cvlab-epfl / tf-lift

Tensorflow port of LIFT (ECCV 2016), with training code.
197 stars 55 forks source link

How to choose descriptor thresholds for NNmAP and matching score across the methods evaluated in your bench ? #26

Closed abenbihi closed 5 years ago

abenbihi commented 5 years ago

Hello,

First thank you for releasing the code. I read the paper where you choose the metrics to be repeatability, NNmAP and matching score. You run extensive experiments on several methods which descriptor dimension, range and even semantic meanings (ORB for example uses binary descriptor) changes. I am looking for a way to choose the descriptor thresholds used in NNmAP and matching score definition that would be consistent for all the methods. Could you recommend me some resources or explain me how to do it please ?

Thank you in advance.

kmyi commented 5 years ago

mAP is computed by considering all thresholds. Please see other descriptor papers, and study precision recall curves and how area under curve is used.

Thanks, Kwang

abenbihi commented 5 years ago

Hello thank you for your answer.

I already studied the metric definition and computation. The only thing I am lacking are the thresholds hard values. For NNmAP you suggest using all the thresholds. But if I have a set of matching features with distance from 0.2 to 1.6, i still need to set a step and evaluate NNmAP for thresholds in linspace(0.2, 1.6, step) no ?

Also for the matching score, I have to choose a unique descriptor distance threshold, how can choose it ?

I found some code with hard-coded thresholds values such as the matlab code from oxford vgg affine however these values are not necessarily relevant for other methods such as orb.

Thanks in advance

kmyi commented 5 years ago

I already studied the metric definition and computation. The only thing I am lacking are the thresholds hard values. For NNmAP you suggest using all the thresholds. But if I have a set of matching features with distance from 0.2 to 1.6, i still need to set a step and evaluate NNmAP for thresholds in linspace(0.2, 1.6, step) no ?

You can simply go through the matches in the order of distances, which effectively does infinite intervals.

Also for the matching score, I have to choose a unique descriptor distance threshold, how can choose it ?

We use nearest neighbour matches, i.e. descriptor distance threshold is infinite.

abenbihi commented 5 years ago

Thank you for these recommendations.