Enable support for RLE encoded segmentations

[mjkvaak]

Hi there! Thanks of the amazing repository.

I tried training my model with custom data (following detectron2 guidelines) with annotation masks encoded in RLE. I also used cfg.INPUT.MASK_FORMAT='bitmask'. All the detectron2 magic works - loading the data, building the model etc. - but the code crashes when the model first tries to evaluate the losses. Attached is (the relevant part of) the error message, from which I would deduce that somewhere in the code a conversion of masks from Bitmasks to Polygons would be needed.

Traceback (most recent call last):
 ...

  File "../centermask/modeling/centermask/center_heads.py", line 401, in forward
    losses, mask_features, selected_mask, labels, maskiou_targets = self._forward_mask(features, proposals)
  File "../centermask/modeling/centermask/center_heads.py", line 476, in _forward_mask
    loss, selected_mask, labels, maskiou_targets = mask_rcnn_loss(mask_logits, proposals, self.maskiou_on)
  File "../centermask/modeling/centermask/mask_head.py", line 80, in mask_rcnn_loss
    cropped_mask = crop(instances_per_image.gt_masks.polygons, instances_per_image.proposal_boxes.tensor)
AttributeError: 'BitMasks' object has no attribute 'polygons'

[Schneey]

Hi there! Thanks of the amazing repository.

I tried training my model with custom data (following detectron2 guidelines) with annotation masks encoded in RLE. I also used cfg.INPUT.MASK_FORMAT='bitmask'. All the detectron2 magic works - loading the data, building the model etc. - but the code crashes when the model first tries to evaluate the losses. Attached is (the relevant part of) the error message, from which I would deduce that somewhere in the code a conversion of masks from Bitmasks to Polygons would be needed.

Traceback (most recent call last):
 ...

  File "../centermask/modeling/centermask/center_heads.py", line 401, in forward
    losses, mask_features, selected_mask, labels, maskiou_targets = self._forward_mask(features, proposals)
  File "../centermask/modeling/centermask/center_heads.py", line 476, in _forward_mask
    loss, selected_mask, labels, maskiou_targets = mask_rcnn_loss(mask_logits, proposals, self.maskiou_on)
  File "../centermask/modeling/centermask/mask_head.py", line 80, in mask_rcnn_loss
    cropped_mask = crop(instances_per_image.gt_masks.polygons, instances_per_image.proposal_boxes.tensor)
AttributeError: 'BitMasks' object has no attribute 'polygons'

I met the same problem as you. Have you solved it now？

[mjkvaak]

Unfortunately no: I had to switch from RLE to polygons. Here's a link, in case you need to make COCO style polygon annotations yourself out of segmentation masks: https://www.immersivelimit.com/tutorials/create-coco-annotations-from-scratch. However, I should mention that when simplifying the polygons' geometries with shapely, instead of using preserve_topology = False mentioned in the blog, one should prefer preserve_topology = True . The point is that in the former case a Polygon shape may get split into multiple polygons, a Multipolygon after the simplification. Then, a Multipolygon doesn't have the method for extracting the border points and the script will crash.

[fshamsafar]

Hi there! Thanks of the amazing repository.

I tried training my model with custom data (following detectron2 guidelines) with annotation masks encoded in RLE. I also used cfg.INPUT.MASK_FORMAT='bitmask'. All the detectron2 magic works - loading the data, building the model etc. - but the code crashes when the model first tries to evaluate the losses. Attached is (the relevant part of) the error message, from which I would deduce that somewhere in the code a conversion of masks from Bitmasks to Polygons would be needed.

Traceback (most recent call last):
 ...

  File "../centermask/modeling/centermask/center_heads.py", line 401, in forward
    losses, mask_features, selected_mask, labels, maskiou_targets = self._forward_mask(features, proposals)
  File "../centermask/modeling/centermask/center_heads.py", line 476, in _forward_mask
    loss, selected_mask, labels, maskiou_targets = mask_rcnn_loss(mask_logits, proposals, self.maskiou_on)
  File "../centermask/modeling/centermask/mask_head.py", line 80, in mask_rcnn_loss
    cropped_mask = crop(instances_per_image.gt_masks.polygons, instances_per_image.proposal_boxes.tensor)
AttributeError: 'BitMasks' object has no attribute 'polygons'

I also want to use bitmasks; the polygon annotations are tricky when simplifying using shapely, and also occupy larger space. In order to use bitmasks, one needs to modify mask_head.py. Somewhere the class BitMasks should be used as mentioned in https://github.com/facebookresearch/detectron2/issues/200#issuecomment-548163850 . However, I could not figure it out. Any help would be appreciated. Thanks.

[Paragjain10]

Hello @miasch @mjkvaak @Schneey ,

I am facing the same problem. how did yall solve the problem?

I am trying to implement CenterMask2 https://github.com/youngwanLEE/centermask2 which makes use of Detectron2. My annotations are in COCO RLE format, my data registration is successful. I visualized the data as well using the detectron2 visualizer, also I have set cfg.INPUT.MASk_FORMAT= bitmask, but somehow this error is being raised while I try to train the network.

Can somebody help me out here with what could be done?

/home/student2/anaconda3/envs/Parag_Centremask2/lib/python3.8/site-packages/detectron2/data/detection_utils.py:414: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at  /opt/conda/conda-bld/pytorch_1607370172916/work/torch/csrc/utils/tensor_numpy.cpp:141.)
  torch.stack([torch.from_numpy(np.ascontiguousarray(x)) for x in masks])
/home/student2/Desktop/Parag_Centremask2/centermask2/centermask/modeling/fcos/fcos_outputs.py:402: UserWarning: This overload of nonzero is deprecated:
    nonzero()
Consider using one of the following signatures instead:
    nonzero(*, bool as_tuple) (Triggered internally at  /opt/conda/conda-bld/pytorch_1607370172916/work/torch/csrc/utils/python_arg_parser.cpp:882.)
  per_candidate_nonzeros = per_candidate_inds.nonzero()
Traceback (most recent call last):
  File "/home/student2/Desktop/Parag_Centremask2/centermask2/train_net.py", line 291, in <module>
    launch(
  File "/home/student2/anaconda3/envs/Parag_Centremask2/lib/python3.8/site-packages/detectron2/engine/launch.py", line 62, in launch
    main_func(*args)
  File "/home/student2/Desktop/Parag_Centremask2/centermask2/train_net.py", line 285, in main
    return trainer.train()
  File "/home/student2/Desktop/Parag_Centremask2/centermask2/train_net.py", line 162, in train
    self.train_loop(self.start_iter, self.max_iter)
  File "/home/student2/Desktop/Parag_Centremask2/centermask2/train_net.py", line 151, in train_loop
    self.run_step()
  File "/home/student2/anaconda3/envs/Parag_Centremask2/lib/python3.8/site-packages/detectron2/engine/defaults.py", line 423, in run_step
    self._trainer.run_step()
  File "/home/student2/anaconda3/envs/Parag_Centremask2/lib/python3.8/site-packages/detectron2/engine/train_loop.py", line 228, in run_step
    loss_dict = self.model(data)
  File "/home/student2/anaconda3/envs/Parag_Centremask2/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/student2/anaconda3/envs/Parag_Centremask2/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 166, in forward
    _, detector_losses = self.roi_heads(images, features, proposals, gt_instances)
  File "/home/student2/anaconda3/envs/Parag_Centremask2/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/student2/Desktop/Parag_Centremask2/centermask2/centermask/modeling/centermask/center_heads.py", line 401, in forward
    losses, mask_features, selected_mask, labels, maskiou_targets = self._forward_mask(features, proposals)
  File "/home/student2/Desktop/Parag_Centremask2/centermask2/centermask/modeling/centermask/center_heads.py", line 476, in _forward_mask
    loss, selected_mask, labels, maskiou_targets = mask_rcnn_loss(mask_logits, proposals, self.maskiou_on)
  File "/home/student2/Desktop/Parag_Centremask2/centermask2/centermask/modeling/centermask/mask_head.py", line 80, in mask_rcnn_loss
    cropped_mask = crop(instances_per_image.gt_masks.polygons, instances_per_image.proposal_boxes.tensor)
AttributeError: 'BitMasks' object has no attribute 'polygons'

[mjkvaak]

If i remember correctly, I had the exact same error. I didn't ever put a lot of effort to try to solve problem, but switched to using conventional polygon annotations instead (see above).

[Paragjain10]

Hello @mjkvaak, Unfortunately, the application I am working on is such that I cannot use polygons as a mask. I have to use Rle segmentation mask only. Is there any other solution to this problem?

Is it possible to train Cenetrmask2 with Rle segmentation annotations?

[15024287710Jackson]

If MASKIOU_ON = True, the error will come up. If MASKIOU_ON = False, the error will not take place.

cropped_mask = crop(instances_per_image.gt_masks.polygons, instances_per_image.proposal_boxes.tensor)

The MASKIOU only can be trained with polygons instead of RLE format, but my dataset only has RLE format in segmentation.