Segmentations are not correct when decoded with pycocotools

[PhilippMarquardt]

Hi, I was having some problems when trying to use the mmdetection framework instance segmentation with the coco format. My dataset requires real precise segmentations for each object but the results were not good enough. To narrow down the problem I tried the pycocotools to decode the segmentations of the coco annotation json file. And to my suprise the resulting segmentation is not exactly the original segmentation but a segmentation that is missing a few pixels. Is this really an error in the cocoapi? To reproduce see the following code:

import json
import numpy as np
import cv2
import pycocotools
from pycocotools.coco import COCO

def create_coco_dict(file):
    '''
    Creates coco dataset
    '''
    files = {}
    files['info'] = {"year": 2222, "version": "1.0", "description": "Object detection", "date_created": "2222"}
    files['licenses'] = [{'id': 1,
      'name': 'GNU General Public License v3.0',
      'url': 'https://github.com/zhiqwang/yolov5-rt-stack/blob/master/LICENSE'}]
    files["type"] = "instances"
    files['categories'] = []
    files["annotations"] = []
    files['images'] = []
    files['categories'].append({'id': 0, 'name': "0", 'supercategory': "0"})
    all_annos = 0                    
    im = cv2.imread(file, 0)
    empty = np.zeros_like(im)
    files['images'].append({'date_captured': '2021',
                              'file_name': file,
                              'id': 0,
                              'height': im.shape[0],
                              'width': im.shape[1]})

    tmp = im.copy()
    #get contours of image
    contours,hierachy = cv2.findContours(tmp, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)
    #black images to draw the contours on
    blank = np.zeros_like(tmp)

    for cnt, cont in enumerate(contours):
        segmentation = []
        xmin,ymin,width,height = cv2.boundingRect(cont) #bounding box
        if width * height < 3:
            continue
        image_height = im.shape[0]
        image_width = im.shape[1]

        #draw contour for verification
        cv2.drawContours(empty, [cont], 0, 255, -1)

        cont = cont.flatten().tolist() #contour as 1d array has shape (x1,y1,x2,y2,...,x_n, y_n)
        #as in https://github.com/facebookresearch/Detectron/issues/100#issuecomment-362882830
        if len(cont) > 4: #only of at least 2 points are there
            segmentation.append(cont)
        else:
            continue
        if len(segmentation) == 0: #check again if segmentations are in list
            continue
        files["annotations"].append({'segmentation': segmentation,
                                      'area': width * height,
                                      'image_id': 0,
                                      'iscrowd':0,
                                      'bbox': [xmin,ymin,width,height],
                                      "category_id": 0,
                                      "id": all_annos})
        all_annos += 1

        cv2.imwrite("drawn_contours.png", empty)

    return files

filename = "test_file.png" #TODO: insert your filename here!
di = create_coco_dict(filename)
with open("test.json", "w") as handle:
    json.dump(di, handle)

coco_annotation = COCO("test.json")
ann_ids = coco_annotation.getAnnIds(imgIds=[0], iscrowd=None)
anns = coco_annotation.loadAnns(ann_ids)

mask = np.zeros(cv2.imread(filename,0).shape)
for i in range(len(anns)):
    mask += coco_annotation.annToMask(anns[i])
cv2.imwrite("coco_out.png", mask * 255.)

assert(np.array_equal(cv2.imread("coco_out.png",0),cv2.imread("drawn_contours.png", 0))) #images are not equal even though they should be(?)

The test_file.png I have used is just a white circle drawn over a black background in paint. Feel free to download it and use it as input to the function. See here:

I am quite sure that I did everything right when creating the annotation file. To my understand the drawn_contours.png which are just the contours directly drawn onto a black image and the coco_out.png should be exactly the same. But as you can see when running the code this is not the case. The coco_out.png is missing a few pixels which makes the overall white circle smaller than it should be. Thanks :)

[ninafiona]

I'm having the same problem. I think this is a bug..

I used this instead:

w, h = img.shape[:2]
mask = np.zeros((w, h))
segmentation = coco.loadAnns(annId)[0]['segmentation'][0]
mask[segmentation[i][1::2], segmentation[i][::2]] = 1