writer.CocoAnnotationsWriter creating annotations with empty segmentations and bboxes with 0 area

travis575757 commented 4 years ago

Error Details

The creation of empty segmentation map lists or bboxes which have a area of zero are generated in certain circumstances with the current writer.CocoAnnotationsWriter. This did not appear in an older version of BlenderProc which we were using a few months ago. Inspection of generated renders show that these cases have objects which are likely sub-pixel in size (appearing in the render because of effects such as anti-aliasing). We mainly have a problem with this because these annotations are incompatible with Facebook's Detectron.

This problem is subjective since these segmentations might have a use-case. If this behavior and its consequences are intentional I will probably put in a feature request to enable the old functionality.

Examples in generated data

Annotation

{'id': 35267, 'image_id': 9142, 'category_id': 3, 'iscrowd': 0, 'area': [0], 'bbox': [423, 0, 1, 0], 'segmentation': [], 'width': 512, 'height': 512}

Image (likely source of error marked)

rgb_9142

Recreation

Copy the examples/bop_object_pose_sampling/config.yaml
Use BOP or provide your own modeling files. (make sure there will be cases where objects are outside the cameras point of view / have overlap with the cameras viewbox border)
Run for a large set of images (total_noof_cams > 2500)
Examine the output and it will likely contain annotations similar to the one above.

Errors Messages from Detectron

Crash Message 1

ERROR [06/12 12:20:49 d2.engine.train_loop]: Exception during training:
Traceback (most recent call last):
  File "/home/ryan/Documents/NIUVT/objDet/detectron/detectron2/detectron2/engine/train_loop.py", line 132, in train
    self.run_step()
  File "/home/ryan/Documents/NIUVT/objDet/detectron/detectron2/detectron2/engine/train_loop.py", line 209, in run_step
    data = next(self._data_loader_iter)
  File "/home/ryan/Documents/NIUVT/objDet/detectron/detectron2/detectron2/data/common.py", line 142, in __iter__
    for d in self.dataset:
  File "/home/ryan/.pyenv/versions/objDet3.7/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
    data = self._next_data()
  File "/home/ryan/.pyenv/versions/objDet3.7/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 838, in _next_data
    return self._process_data(data)
  File "/home/ryan/.pyenv/versions/objDet3.7/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 881, in _process_data
    data.reraise()
  File "/home/ryan/.pyenv/versions/objDet3.7/lib/python3.7/site-packages/torch/_utils.py", line 395, in reraise
    raise self.exc_type(msg)
KeyError: Caught KeyError in DataLoader worker process 1.
Original Traceback (most recent call last):
  File "/home/ryan/.pyenv/versions/objDet3.7/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 178, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/ryan/.pyenv/versions/objDet3.7/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/ryan/.pyenv/versions/objDet3.7/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/ryan/Documents/NIUVT/objDet/detectron/detectron2/detectron2/data/common.py", line 41, in __getitem__
    data = self._map_func(self._dataset[cur_idx])
  File "/home/ryan/Documents/NIUVT/objDet/detectron/detectron2/detectron2/utils/serialize.py", line 23, in __call__
    return self._obj(*args, **kwargs)
  File "/home/ryan/Documents/NIUVT/objDet/detectron/detectron2/detectron2/data/dataset_mapper.py", line 139, in __call__
    annos, image_shape, mask_format=self.mask_format
  File "/home/ryan/Documents/NIUVT/objDet/detectron/detectron2/detectron2/data/detection_utils.py", line 321, in annotations_to_instances
    segms = [obj["segmentation"] for obj in annos]
  File KeyError: Caught KeyError in DataLoader worker process 2.
Original Traceback (most recent call last):
  File "C:\Users\URI PC\Anaconda3\envs\Detectron22\lib\site-packages\torch\utils\data\_utils\worker.py", line 178, in _worker_loop
    data = fetcher.fetch(index)
  File "C:\Users\URI PC\Anaconda3\envs\Detectron22\lib\site-packages\torch\utils\data\_utils\fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "C:\Users\URI PC\Anaconda3\envs\Detectron22\lib\site-packages\torch\utils\data\_utils\fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "c:\users\uri pc\detectron2\detectron2\data\common.py", line 39, in __getitem__
    data = self._map_func(self._dataset[cur_idx])
  File "c:\users\uri pc\detectron2\detectron2\data\dataset_mapper.py", line 131, in __call__
    annos, image_shape, dataset_dict, mask_format=self.mask_format
  File "c:\users\uri pc\detectron2\detectron2\data\detection_utils.py", line 240, in annotations_to_instances
    polygons = [obj["segmentation"] for obj in annos]
  File "c:\users\uri pc\detectron2\detectron2\data\detection_utils.py", line 240, in <listcomp>
    polygons = [obj["segmentation"] for obj in annos]
KeyError: 'segmentation'"/home/ryan/Documents/NIUVT/objDet/detectron/detectron2/detectron2/data/detection_utils.py", line 321, in <listcomp>
    segms = [obj["segmentation"] for obj in annos]
KeyError: 'segmentation'

Crash Message 2

ERROR [06/12 12:17:15 d2.engine.train_loop]: Exception during training:
Traceback (most recent call last):
  File "/home/ryan/Documents/NIUVT/objDet/detectron/detectron2/detectron2/engine/train_loop.py", line 132, in train
    self.run_step()
  File "/home/ryan/Documents/NIUVT/objDet/detectron/detectron2/detectron2/engine/train_loop.py", line 215, in run_step
    loss_dict = self.model(data)
  File "/home/ryan/.pyenv/versions/objDet3.7/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/ryan/Documents/NIUVT/objDet/detectron/detectron2/detectron2/modeling/meta_arch/rcnn.py", line 123, in forward
    _, detector_losses = self.roi_heads(images, features, proposals, gt_instances)
  File "/home/ryan/.pyenv/versions/objDet3.7/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/ryan/Documents/NIUVT/objDet/detectron/detectron2/detectron2/modeling/roi_heads/roi_heads.py", line 669, in forward
    losses.update(self._forward_mask(features, proposals))
  File "/home/ryan/Documents/NIUVT/objDet/detectron/detectron2/detectron2/modeling/roi_heads/roi_heads.py", line 773, in _forward_mask
    return self.mask_head(mask_features, proposals)
  File "/home/ryan/.pyenv/versions/objDet3.7/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/ryan/Documents/NIUVT/objDet/detectron/detectron2/detectron2/modeling/roi_heads/mask_head.py", line 190, in forward
    return {"loss_mask": mask_rcnn_loss(x, instances, self.vis_period)}
  File "/home/ryan/Documents/NIUVT/objDet/detectron/detectron2/detectron2/modeling/roi_heads/mask_head.py", line 63, in mask_rcnn_loss
    gt_masks_per_image = instances_per_image.gt_masks.crop_and_resize(
  File "/home/ryan/Documents/NIUVT/objDet/detectron/detectron2/detectron2/structures/instances.py", line 60, in __getattr__
    raise AttributeError("Cannot find field '{}' in the given Instances!".format(name))
AttributeError: Cannot find field 'gt_masks' in the given Instances!

ideas-man commented 4 years ago

@travis575757 Hey, if I understood you correctly, the bbox area of 0 was/is an issue of the original coco code (due to they way it is calculated) thus since we strictly use the original code, this problem was always present in BlenderProc (object being visible by just one pixel in the image leads to the wrong segmentation and bounding box). For cases such as this one, we have a script that can format the generated annotations by dropping the faulty annotations. I hope this helps.

travis575757 commented 4 years ago

The scripts help. However I've generated 50000+ images before and never had this issue. Looking at src/utility/Coco/pycococreatortools.create_annotation_info I found this.

    @staticmethod
    def create_annotation_info(annotation_id, image_id, category_info, binary_mask, 
                            image_size=None, tolerance=2, bounding_box=None):
        """Creates info section of coco annotation
        Args:
            annotation_id: integer to uniquly identify the annotation
            image_id: integer to uniquly identify image
            category_info: dict which contains category info and a boolean attribute "crowd" which tells if the annotation is a crowd of a class 
        """
        if image_size is not None:
            binary_mask = PycocoCreatorTools.resize_binary_mask(binary_mask, image_size)

        if bounding_box is None:
            bounding_box = PycocoCreatorTools.bbox(binary_mask)
            area = bounding_box[2] * bounding_box[3]

        if category_info["is_crowd"]:
            is_crowd = 1
            segmentation = PycocoCreatorTools.binary_mask_to_rle(binary_mask)
        else :
            is_crowd = 0
            segmentation = PycocoCreatorTools.binary_mask_to_polygon(binary_mask, tolerance)
            if not segmentation:
                return None

        annotation_info = {
            "id": annotation_id,
            "image_id": image_id,
            "category_id": int(category_info["id"]),
            "iscrowd": is_crowd,
            "area": [area],
            "bbox": bounding_box,
            "segmentation": segmentation,
            "width": binary_mask.shape[1],
            "height": binary_mask.shape[0],
        }
        return annotation_info

So based on my problem above I would not experience an empty segmentation list based on this section of code. It seems like recently this file has been deleted and merged with src.utility.CocoUtililty.py which contains this.

    @staticmethod
    def create_annotation_info(annotation_id, image_id, object_id, binary_mask, tolerance=2):
        """Creates info section of coco annotation
        :param annotation_id: integer to uniquly identify the annotation
        :param image_id: integer to uniquly identify image
        :param object_id: The object id, should match with the object's category id
        :param binary_mask: A binary image mask of the object with the shape [H, W].
        :param tolerance: The tolerance for fitting polygons to the objects mask.
        """
        bounding_box = CocoUtility.bbox_from_binary_mask(binary_mask)
        area = bounding_box[2] * bounding_box[3]

        segmentation = CocoUtility.binary_mask_to_polygon(binary_mask, tolerance)

        annotation_info = {
            "id": annotation_id,
            "image_id": image_id,
            "category_id": object_id,
            "iscrowd": 0,
            "area": [area],
            "bbox": bounding_box,
            "segmentation": segmentation,
            "width": binary_mask.shape[1],
            "height": binary_mask.shape[0],
        }
        return annotation_info

Unlike the old code there is no empty list check so you can now receive empty lists. Am I getting something wrong here? Otherwise it seems like BlenderProc did once have this functionality. Link to the old code I'm referencing

MartinSmeyer commented 4 years ago

@travis575757 thank you, I believe you are correct. I don't know why this check present in the original pycococreatortools was taken out during refactoring.. Will be reintroduced.

Independent of the Coco Writer, the single pixel segmentations should be gone in newer BlenderProc versions (was related to a denoiser).

Tuebel commented 3 years ago

I have stumbled over the same issue today, when trying to load the generated data into Detectron2. Is it still planned to fix this?

MartinSmeyer commented 3 years ago

Hi,

Thanks. Your issue also relates to the polygon mask encoding format, right?

We thought, that we caught the bug by excluding empty binary_masks. https://github.com/DLR-RM/BlenderProc/blob/ec0eca2806eccac0b537f9dacada151a56ee07e3/src/utility/CocoUtility.py#L168-L170

But apparently for some special cases binary_mask_to_polygon can return empty segmentation lists for area>1.

So for the polygon format we will also introduce again

            if not segmentation:
                return None

That said, since this issue we have also integrated the RLE mask encoding format (as default) into the CocoAnnotationsWriter which should be preferred because it can represent masks with holes. The RLE format should not suffer from this issue in newer BlenderProc versions, please let us know if your experience is different :)

Tuebel commented 3 years ago

Hey, thanks for your quick reply. Indeed, the RLE encoding does not seem to suffer from this bug. At least none of my ~4000 generated annotations do not contain empty segmentation fields. Also, masks with holes are nice :)

The only issue I experienced is that the default iscrowd=1 for RLE leads to Detectron2 ignoring the masks.However, I guess that this should be another issue.

MartinSmeyer commented 3 years ago

Yes, Detectron2 really seems to ignore iscrowd=1 annotations. That is interesting because it is used for RLE annotations in the pycococreatortools:

https://github.com/waspinator/pycococreator/blob/207b4fa8bbaae22ebcdeb3bbf00b724498e026a7/pycococreatortools/pycococreatortools.py#L94-L101

But I also found an issue in their repo which supports the claim that is_crowd should be left out.

https://github.com/waspinator/pycococreator/issues/10

So we will probably change the "iscrowd" parameter to 0.

themasterlink commented 3 years ago

So we will probably change the "iscrowd" parameter to 0.

This change is now active in version 1.8.2 of BlenderProc.

I assume this issue can then be closed again.

DLR-RM / BlenderProc