Bug for AP Evaluation for IoU threshold 0.9

[kemaloksuz]

Hi,

I wanted to check other thresholds than 0.5 and 0.75 for mAP. To do that, I simply added several lines for each IoU such as:

stats[9] = _summarize(1, iouThr=.9, maxDets=self.params.maxDets[2])

However, there is a bug only for 0.9, which causes to return -1 as AP@90. Here is an example output of 3 thresholds:

Average Precision (AP) @[ IoU=0.85 | area= all | maxDets=100 ] = 0.230 Average Precision (AP) @[ IoU=0.90 | area= all | maxDets=100 ] = -1.000 Average Precision (AP) @[ IoU=0.95 | area= all | maxDets=100 ] = 0.018

This surprised me since AP@0.95 > AP@0.90. When I debug, I noticed that p.iouThrs[8] is not exactly 0.9 but 0.89999. Here is the output from debugger in the _summarize function:

(Pdb) p p.iouThrs[8] 0.8999999999999999

For me, I solved this bug by a small workaround, but I think it should be systematically handled in this repo.

Regards,

Kemal

[youngwanLEE]

@kemaloksuz I came across the same problem.

How to solve it ?

[kemaloksuz]

Hello,

As I mentioned, my solution is not an elegant one, just a workaround. However this is how I do it. These are the relevant lines from summarize() function (I want to see all the thresholds):

def summarize(self):
    '''
    Compute and display summary metrics for evaluation results.
    Note this functin can *only* be applied on the default parameter setting
    '''
    def _summarize( ap=1, iouThr=None, areaRng='all', maxDets=100 ):
        p = self.params
        iStr = ' {:<18} {} @[ IoU={:<9} | area={:>6s} | maxDets={:>3d} ] = {:0.3f}'
        titleStr = 'Average Precision' if ap == 1 else 'Average Recall'
        typeStr = '(AP)' if ap==1 else '(AR)'
        iouStr = '{:0.2f}:{:0.2f}'.format(p.iouThrs[0], p.iouThrs[-1]) \
            if iouThr is None else '{:0.2f}'.format(iouThr)

        aind = [i for i, aRng in enumerate(p.areaRngLbl) if aRng == areaRng]
        mind = [i for i, mDet in enumerate(p.maxDets) if mDet == maxDets]
        if ap == 1:
            # dimension of precision: [TxRxKxAxM]
            s = self.eval['precision']
            # IoU
            if iouThr is not None:
                if iouThr == 0.9:
                    t = [8]
                else:
                    t = np.where(iouThr == p.iouThrs)[0]
                s = s[t]
            s = s[:,:,:,aind,mind]
        else:
            # dimension of recall: [TxKxAxM]
            s = self.eval['recall']
            if iouThr is not None:
                if iouThr == 0.9:
                    t = [8]
                else:
                    t = np.where(iouThr == p.iouThrs)[0]
                s = s[t]
            s = s[:,:,aind,mind]
        if len(s[s>-1])==0:
            mean_s = -1
        else:
            mean_s = np.mean(s[s>-1])
        print(iStr.format(titleStr, typeStr, iouStr, areaRng, maxDets, mean_s))
        return mean_s
    def _summarizeDets():
        stats = np.zeros((20,))
        stats[0] = _summarize(1)
        stats[1] = _summarize(1, iouThr=.5, maxDets=self.params.maxDets[2])
        stats[2] = _summarize(1, iouThr=.55, maxDets=self.params.maxDets[2])                        
        stats[3] = _summarize(1, iouThr=.6, maxDets=self.params.maxDets[2])
        stats[4] = _summarize(1, iouThr=.65, maxDets=self.params.maxDets[2])
        stats[5] = _summarize(1, iouThr=.7, maxDets=self.params.maxDets[2])                        
        stats[6] = _summarize(1, iouThr=.75, maxDets=self.params.maxDets[2])
        stats[7] = _summarize(1, iouThr=.8, maxDets=self.params.maxDets[2])     
        stats[8] = _summarize(1, iouThr=.85, maxDets=self.params.maxDets[2])                                                       
        stats[9] = _summarize(1, iouThr=.9, maxDets=self.params.maxDets[2])   
        stats[10] = _summarize(1, iouThr=.95, maxDets=self.params.maxDets[2])                                 
        stats[11] = _summarize(1, areaRng='small', maxDets=self.params.maxDets[2])
        stats[12] = _summarize(1, areaRng='medium', maxDets=self.params.maxDets[2])
        stats[13] = _summarize(1, areaRng='large', maxDets=self.params.maxDets[2])
        stats[14] = _summarize(0, maxDets=self.params.maxDets[0])
        stats[15] = _summarize(0, maxDets=self.params.maxDets[1])
        stats[16] = _summarize(0, maxDets=self.params.maxDets[2])
        stats[17] = _summarize(0, areaRng='small', maxDets=self.params.maxDets[2])
        stats[18] = _summarize(0, areaRng='medium', maxDets=self.params.maxDets[2])
        stats[19] = _summarize(0, areaRng='large', maxDets=self.params.maxDets[2])
        return stats

The rest of the function is the same.

[youngwanLEE]

@kemaloksuz thanks for the reply :)

[Spritea]

I met the same issue, and I found that the reason is the generated ndarray p.iouThrs by np.linspace is not exact.

It is supposed to be [0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8, 0.85, 0.9, 0.95], but in fact it is [0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8, 0.85, 0.8999999999999999, 0.95] in my machine with numpy version=1.20.2, so IoU with 0.9 can't work. This is an inherent problem of IEEE float point, and we should always avoid doing float comparisons with ==.

One simple way is to change the line: t = np.where(iouThr == p.iouThrs)[0] to: t = np.where(np.isclose(iouThr, p.iouThrs))[0].

Note that t = np.where(iouThr == p.iouThrs)[0] appeares twice in the code block of https://github.com/cocodataset/cocoapi/issues/409#issuecomment-632591018, so change both places.

[ThreeStones1029]

I met the same issue, and I found that the reason is the generated ndarray p.iouThrs by np.linspace is not exact.

It is supposed to be [0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8, 0.85, 0.9, 0.95], but in fact it is [0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8, 0.85, 0.8999999999999999, 0.95] in my machine with numpy version=1.20.2, so IoU with 0.9 can't work. This is an inherent problem of IEEE float point, and we should always avoid doing float comparisons with ==.

One simple way is to change the line: t = np.where(iouThr == p.iouThrs)[0] to: t = np.where(np.isclose(iouThr, p.iouThrs))[0].

Note that t = np.where(iouThr == p.iouThrs)[0] appeares twice in the code block of #409 (comment), so change both places.

Thanks