[Is there any risk of labels missing during merging labels?] MMrotate function : merge_results()

[Shien11]

Branch

master branch https://github.com/open-mmlab/mmrotate/blob/main/demo/huge_image_demo.py

📚 The doc issue

I'm now running this code in MMrotate https://github.com/open-mmlab/mmrotate/blob/main/demo/huge_image_demo.py. It’s about using huge image to run model inference, it'll process huge image into several patches, and inference by patch, finally merge the labels in one.

The main function of merging labels is merge_results() As my understanding, it's the function merging labels from several patches. I'm curious about what's the mind set to merge all labels in one? Is there any risk of labels missing during merging? Any ideas would help, thank you.

Suggest a potential alternative/fix

I'm now running this code in MMrotate https://github.com/open-mmlab/mmrotate/blob/main/demo/huge_image_demo.py. It’s about using huge image to run model inference, it'll process huge image into several patches, and inference by patch, finally merge the labels in one.

The main function of merging labels is merge_results() As my understanding, it's the function merging labels from several patches. I'm curious about what's the mind set to merge all labels in one? Is there any risk of labels missing during merging? Any ideas would help, thank you.

---

def merge_results(results, offsets, img_shape, iou_thr=0.1, device='cpu'): """Merge patch results via nms. Args: results (list[np.ndarray] | list[tuple]): A list of patches results. offsets (np.ndarray): Positions of the left top points of patches. img_shape (tuple): A tuple of the huge image's width and height. iou_thr (float): The IoU threshold of NMS. device (str): The device to call nms.

Retunrns: list[np.ndarray]: Detection results after merging. """ assert len(results) == offsets.shape[0], 'The results should has the ' \ 'same length with offsets.' with_mask = isinstance(results[0], tuple) num_patches = len(results) num_classes = len(results[0][0]) if with_mask else len(results[0])

merged_bboxes = [] merged_masks = [] for cls in range(num_classes): if with_mask: dets_per_cls = [results[i][0][cls] for i in range(num_patches)] masks_per_cls = [results[i][1][cls] for i in range(num_patches)] else: dets_per_cls = [results[i][cls] for i in range(num_patches)] masks_per_cls = None

dets_per_cls = [
    translate_bboxes(dets_per_cls[i], offsets[i])
    for i in range(num_patches)
]
dets_per_cls = np.concatenate(dets_per_cls, axis=0)
if with_mask:
    masks_placeholder = []
    for i, masks in enumerate(masks_per_cls):
        translated = map_masks(masks, offsets[i], img_shape)
       masks_placeholder.extend(translated)
    masks_per_cls = masks_placeholder

if dets_per_cls.size == 0:
    merged_bboxes.append(dets_per_cls)
    if with_mask:
        merged_masks.append(masks_per_cls)
else:
    dets_per_cls = torch.from_numpy(dets_per_cls).to(device)
    nms_func = nms if dets_per_cls.size(1) == 5 else nms_rotated
    nms_dets, keeps = nms_func(dets_per_cls[:, :-1],
                               dets_per_cls[:, -1], iou_thr)
    merged_bboxes.append(nms_dets.cpu().numpy())
    if with_mask:
        keeps = keeps.cpu().numpy()
        merged_masks.append([masks_per_cls[i] for i in keeps])

if with_mask: return merged_bboxes, merged_masks else: return merged_bboxes

[zytx121]

@Shien11, the answer is YES. The merge function uses multi-class NMS to filter out some boxes caused by overlapping cropping.

[jsxyhelu]

对于最后的显示结果来说，multi-class NMS 肯定是有用的，因为不希望同一个物体被多种类型框起来。但是如果是打包比赛的话，不一定是这种选择。