Precision-Recall Question

[kmactavish]

Question from @wengxiuling36:

Sorry to bother you.I see you paper "Towards Hierarchical Place Recognition for Long-Term Autonomy". Could I ask you how to get Precison-Recall curves using the New College datasets.I downloaded NewCollegeGroundTruth.mat. FabMap1.0 outputs a matrix psame.But this psame matrix is not a 0-1 matrix,and the high probability entries on the main diagonal indicate the detection of new places. So I set main diagonal element in psame to 0.The matlab code:

psame=psame+diag(-diag(psame)); psame=(psame>=0.99); npsame=length(find(psame==1)); tp=length(intersect(find(psame1==1),find(truth==1))); ntruth=length(find(truth==1)); r=tp/ntruth; p=tp/npsame;

Is it right?I find the ntruth is too big and tp is too small compared to npsame.

[kmactavish]

Hi, there are different ways to build the precision-recall curve.

One option, that you have discovered, is mask out the diagonal (new place detection), and only consider the below-diagonal place-recognition probabilities. You could even mask out a larger diagonal band (so you don't just always match to the previous image)---just explain that that is what you're doing. You will also need to mask those entries out of the truth matrix, since those do not contribute to ntruth---you aren't trying to match to them any more, so they don't contribute to your precision-recall.

By picking a single threshold, 0.99, you will only get a single precision-recall value, not a curve. The curve is generated by sweeping the threshold from 0 to 1, and recording every precision/recall value along the way.

[lynnsky]

Thank you for your advice.But I still don't know how to deal with truth matrix.There are continuous 1 in same row but not near the diagonal band.So how to get the right ntruth.The new matlab code: for k=0:10 psame=psame+diag(-diag(psame,-k),-k); end R=[];P=[]; for k=0:10 truth=truth+diag(-diag(truth,-k),-k); end ntruth=length(find(truth==1)); for ploop=0:0.001:1 psame1=(psame>=ploop); npsame=length(find(psame1==1)); tp=length(intersect(find(psame1==1),find(truth==1))); r=tp/ntruth; p=tp/npsame; R=[R r];P=[P p]; end plot(R,P)

[kmactavish]

The ground truth means that the two frames are within some distance of each other (fixed parameter). So there could be multiple correct loop closures for a query (multiple ones in a row). With high confidence, fabmap only gives you one (since the likelihood is normalized and sums to one). So you will have a whole bunch of false negatives, this is normal. You could filter out these false negatives by 'not counting', in ntruth, those that are 'close enough' to the identified loop closure.

[lynnsky]

With low confidence,fabmap will give multiple ones in a row.So I deal with thentruthlike this: ntruth=length(find(sum(truth,2)>0));%how many lines have 1 in truth ntruth=ntruth+npsame-length(find(sum(psame1,2)>0));% The PR curve is shown follows: It seems to be right.

[haoanw]

the name of the x and y axis in your figure should be swapped