conflict with loadRes and CoCoEvaluate

[lucasjinreal]

this code:

coco_dt = coco_gt.loadRes(coco_results)
    print('result loaded.')
    coco_eval = COCOeval(coco_gt, coco_dt, iou_type)
    # Use the COCO default keypoint OKS sigmas unless overrides are specified
    if kpt_oks_sigmas:
        coco_eval.params.kpt_oks_sigmas = np.array(kpt_oks_sigmas)
    coco_eval.evaluate()

loadRes load segmentation mask as uncompressed RLEs, which indicates in cocoapi code, but evaluate() call, reading mask as polygons.

So how to make them same? Why doesn't using the same API?

[niazahamd89]

Hi @jinfagang how you generate coco_results file which is usually in json format for your model let me please explain that.

[lucasjinreal]

@niazahamd89 Hi, I have found that, loadRes is loading from uncompressed RLEs is reasonable, since when evaluation all results contains in memory, if using mask image array that would be very big.

And when evaluate, actually, it loads Gt and Dt all as polygons, it will convert to mask internal.

So seems reasonable here, am I understand right?

[niazahamd89]

you are right @jinfagang but what i want to clear is i train my model on coco train and validation set for pose estimation now i want to evaluate my model to find the AP and AR. i am going to use coco evaluation metric for this purpose, in demo.ipynb the first file i need is the Gt file which i downloaded from coco site the second file i need the Dt file now i am confuse how i can generate this Dt file? my training is complete the model generate weights file in .ckpt format and i test the model is able to plot pose in pictures.

there are some result.json files in cocoapi result folder https://github.com/philferriere/cocoapi/tree/master/results basically the same file person_keypoints_val2014_fakekeypoints100_results.json i want to generate for my model to evaluate.

did you went through this process? did you generate these files for your model evaluation?

[lucasjinreal]

@niazahamd89 loadRes is loading result same format as annotations. But somwhat, I am breaks at somewhere.

[niazahamd89]

@jinfagang did you tray to evaluate your model result on coco evaluation metric.

[xuluxulu]

@jinfagang Hi, Can you explain what the score is in the json file

[hebangwen]

@xuluxulu Hi, I also have a question about the variable score in the json file. I think score means model confidence score in object detection. But I don't know what it means in person keypoints detection. Have you solved this problem? Any reply or help would be appreciate.

[hebangwen]

@xuluxulu Hi, I also have a question about the variable score in the json file. I think score means model confidence score in object detection. But I don't know what it means in person keypoints detection. Have you solved this problem? Any reply or help would be appreciate.

I figured it out several days ago by debug mode. For keypoints problem, It means the average confidence score for all visible keypoints. Like the code here, it means the average confidence score of all visible keypoints(t_s > in_vis_thre), in which t_s is the confidence of a single keypoint. t_s comes from this file, so it actually represents the confidence score.