search

cocodataset / cocoapi

COCO API - Dataset @ http://cocodataset.org/
Other
6.11k stars 3.76k forks source link

Calculating keypoint sigma values for custom dataset #399

Open amruthkarun opened 4 years ago

amruthkarun commented 4 years ago

I have a custom dataset with 20 keypoints and I want to find mAP for this dataset. How can I calculate keypoint sigma values for a custom dataset?

YinRui1991 commented 4 years ago

@amruthkarun Do you solve it? I have the same problem. My dataset has 27 key-points and I don't know how to modify "sigmas" to match my 27 key-points.

Michael-J98 commented 4 years ago

This might help you. I'm not sure, but I think, the sigma value represents the var of deviation caused by the different difficulty when labeling different keypoints. For example, you have much more area，which means greater var, to label when you label a wrist than an eye. So if the two detection keypoints have the same distance and area, the one with greater var is more precise. Principally, you'll need group of people labeling the same dataset, then you can calculate the labeling deviation var of different keypoints.

YinRui1991 commented 4 years ago

@Michael-J98 Thanks. Do you know how to calculate the labeling deviation var of different keypoints?

Michael-J98 commented 4 years ago

Still, I'm not very confident about it. But, sure, I can make a guess. For a specific class of keypoint in a certain image, you'll have lots of point-labels labeled by different people. You might consider the centroid of these points as the 'perfect GT', then you can calculate the the var which is the mean of the sum of the distances between every point and the 'GT'. Finally, take an average of var over all images then you can get the val of this class of keypoint. BTW, you may need to convert var to standard deviation to get the exact segma value. Some details may not true. Hope it's helpful to you. If possible, let me know when you get a more solid recognition about it.

YinRui1991 commented 4 years ago

@Michael-J98 Thanks again. It seems I have to ask a group of people to label my dataset and then I can calculate the deviations. I think it will takes some time. I will share my dataset sigmas when I finish.

amruthkarun commented 4 years ago

@Michael-J98 Thanks. What if the dataset is synthetic and the labeling process is automated with the image generation process. We only get a single label for a given keypoint. How can we set the sigma values in such a scenario?

Michael-J98 commented 4 years ago

@Michael-J98 Thanks. What if the dataset is synthetic and the labeling process is automated with the image generation process. We only get a single label for a given keypoint. How can we set the sigma values in such a scenario?

Sorry, I don't know what to do in this scenario. From my point of view, dividing by segma is some kind of normalization which makes the var of different distributions all equal to 1. But if there is no deviation, as you said, it seems having some trouble converting '0' to '1'. Maybe you can modify you own metric by removing the sigma.

YinRui1991 commented 4 years ago

@amruthkarun My understanding is that you can ask a group of people to annotate your synthetic dataset. Then you can calculate sigma with manual annotation and your GTs. You can refer to http://cocodataset.org/#keypoints-eval or the link(http://presentations.cocodataset.org/COCO17-Keypoints-Overview.pdf) Michael-J98 offered for the sigma calculation.

Jang-Jian commented 3 years ago

We have same question about this issue

tucachmo2202 commented 3 years ago

Still, I'm not very confident about it. But, sure, I can make a guess. For a specific class of keypoint in a certain image, you'll have lots of point-labels labeled by different people. You might consider the centroid of these points as the 'perfect GT', then you can calculate the the var which is the mean of the sum of the distances between every point and the 'GT'. Finally, take an average of var over all images then you can get the val of this class of keypoint. BTW, you may need to convert var to standard deviation to get the exact segma value. Some details may not true. Hope it's helpful to you. If possible, let me know when you get a more solid recognition about it.

I think it may be not true because the var of different labeling people is less than var of predict keypoint?

cucdengjunli commented 1 year ago

same question

aayushsingla commented 1 year ago

same question. Any leads anyone? Is there some standard value that we can use such that, it doesn't cause the results to differ by alot?