Incorrect COCO validation annotation misses / errors. All merged into single mask.

[miquel-espinosa]

Hi. Thanks for the work.

First I convert to coconut_instances_val.json following the instructions from here: https://github.com/bytedance/coconut_cvpr2024/tree/main/tutorials/kmaxdeeplab_instance

Then, I try to use the generated coconut_instances_val.json file with the usual COCO validation, I am encountering a few annotation misses. This is, images that in the original COCO Val were annotated, but in COCONut Val are not! This is problematic if we want to use COCONut for evalutation and comparison.

For example:

• Image id: 360325

• Image id: 329323

Let me know if you can reproduce the error. I hope this can be solved soon. Thanks!

[xdeng7]

Hi. Thanks for the work.

First I convert to coconut_instances_val.json following the instructions from here: https://github.com/bytedance/coconut_cvpr2024/tree/main/tutorials/kmaxdeeplab_instance

Then, I try to use the generated coconut_instances_val.json file with the usual COCO validation, I am encountering a few annotation misses. This is, images that in the original COCO Val were annotated, but in COCONut Val are not! This is problematic if we want to use COCONut for evalutation and comparison.

For example:

• Image id: 360325
• Image id: 329323

Let me know if you can reproduce the error. I hope this can be solved soon. Thanks!

which tool are you using to visualize COCO and COCONut instance?

[miquel-espinosa]

@xdeng7 It's just the usual pycocotools, with the new generated json file.

coco2=COCO(annFile2)
# put specific imgids you want to visualize
img_id_list2=[360325]
imgIds2 = coco2.getImgIds(imgIds = img_id_list2)
img2 = coco2.loadImgs(imgIds2[0])[0]
I2 = io.imread(img2['coco_url'])
# load and display instance annotations
plt.imshow(I2); plt.axis('off')
annIds2 = coco2.getAnnIds(imgIds=img2['id'],  iscrowd=1)
anns2 = coco2.loadAnns(annIds2)
coco2.showAnns(anns2)

[miquel-espinosa]

In particular, here are some image ids, so that you can too check:

• 329323, 360325, 562197, 128658, 185157

[xdeng7]

In particular, here are some image ids, so that you can too check:

• 329323, 360325, 562197, 128658

thank you for the question. I think that's the problem of some of masks were identified as "crowd" regardless of the object size. We will fix these image ids to separate the masks from "crowd" masks. Check the following visualization. From left to right are: image, COCO-val annotation and Relabeled COCO-val annotation.

[miquel-espinosa]

Yes, that is exactly what it's happening. Let me know when it is solved, so I can rerun instance segmentation evaluation with the fixed COCONut. Thanks a lot!

[xdeng7]

Yes, that is exactly what it's happening. Let me know when it is solved, so I can rerun instance segmentation evaluation with the fixed COCONut. Thanks a lot!

yes, will make announcement when upload new version of the dataset.

[JohannesTheo]

Hey @miquel-espinosa this ist not a bug. These crowd annotations come from sama-coco. If I understand correctly, sama-coco is used as a starting point and its masks are refined to create the instances (things) in relabeled_coco_val_panoptic.json. Is this assumption correct @xdeng7? I'm a little surprised by your previous answer.

• 329323, 360325, 562197, 128658, 185157