search

jin-s13 / xtcocoapi

Extended COCO-API
Other
63 stars 24 forks source link

oks set to 1 when x,y are at 0 #23

Closed niccle27 closed 2 years ago

niccle27 commented 2 years ago

Hi !

I am having an issue with the current api you provide. I compared it to the official api and the related code is the same.

Concerning the computeOks function, the following lines are setting x0, y0 as a twice as large bbox coordinates: https://github.com/jin-s13/xtcocoapi/blob/master/xtcocotools/cocoeval.py#L347-L348 In my experience, in most cases, x0, y0 will be set to a negative number, while x1, y1 will be set to a very high number.

This create an issue at line https://github.com/jin-s13/xtcocoapi/blob/master/xtcocotools/cocoeval.py#L376-L378 because the dx, dy vector returned is a vector full of zeros. Under OKS computation, this suppose a perfect match and therefore an OKS of 1.

Do you mind explaining the goal of these lines ?

As a fix for me, i replaced the lines 376-378 with : 

ious[i, j] = 0
continue
# measure minimum distance to keypoints in (x0,y0) & (x1,y1)
z = np.zeros((k))
dx = np.max((z, x0-xd),axis=0)+np.max((z, xd-x1),axis=0)
dy = np.max((z, y0-yd),axis=0)+np.max((z, yd-y1),axis=0)

Which set the oks to false if visibility is 0 for all keypoints. But as a results, the mAP computed is much lower and therefore doesn't match the SOTA once on mmpose repo.

Thank you for your time !

jin-s13 commented 2 years ago

According to the metric of COCO, the invisible keypoints are considered correct even if the predictions are far away from the ground-truths.

niccle27 commented 2 years ago

Hello ! Thank you for your reply. To the best of my knowledge, from the coco website, here is the formula used: OKS = Σi[exp(-di2/2s2κi2)δ(vi>0)] / Σi[δ(vi>0)]

As far as i understand, it only consider values with visibility over 0 (AKA visible)

jin-s13 commented 2 years ago

I understand your points, and I think this is an issue of official cocoapi. Note that this xtcocotools will try to keep the same output as cocoapi. I tend to think that its documentation is not clear enough, rather than the code implementation is wrong.

I think what you said is also reasonable. Invisible points shall be ignored instead of considered correct. BUT this is not what coco evaluation does...