[Bug] Rotated IoU get a wrong value (box_iou_rotated)

lzh420202 commented 1 year ago

Prerequisite

[X] I have searched Issues and Discussions but cannot get the expected help.
[X] The bug has not been fixed in the latest version(https://github.com/open-mmlab/mmcv).

Environment

device: 3090 24G cuda: 11.3 pytorch: 1.11.0 torchvision: 0.12.0 mmcv-full: 1.6.0

Reproduces the problem - code sample

In some cases, the IoU calculated for the same rotating target is 0.333 instead of 1.0

Reproduces the problem - command or script

We test a box [0.0, 0.0, 180.6422271729, 136.3633728027, 0.9559648633], Curiously, the function rbbox_overlaps results at 0.3333, which is obviously not what you'd expect. No other values have been found to cause this phenomenon. The result of many experiments is equal to 0.33.

Reproduces the problem - error message

None

Additional information

No response

hh123445 commented 1 year ago

We found the same problem: when the w deviation (x,y,w,h,theta) between the predicted box and the real box is 1, the IoU decreases to 0.33, but when the w deviation is 2, the IoU returns to 0.98.

HAOCHENYE commented 1 year ago

Hi, thanks for your feedback, can you provide an example to reproduce the problem?

hh123445 commented 1 year ago

Hi, thanks for your feedback, can you provide an example to reproduce the problem?

When the w deviation of the prediction box changes within a certain range, the IoU calculated using 'rbbox_overlaps' and 'diff_iou_rotated_2d' jumps instead of continuous, which is very different from the results obtained using the intersection and union ratio method of pixels.

Express gratitude in advance！

import copy

import cv2
import matplotlib.pyplot as plt
import numpy as np
import math
from mmrotate.core.bbox.iou_calculators.rotate_iou2d_calculator import rbbox_overlaps, RBboxOverlaps2D
import torch
from mmcv.ops import diff_iou_rotated_2d

'''
When the w deviation of the prediction box changes within a certain range, 
the IoU calculated using 'rbbox_overlaps' and 'diff_iou_rotated_2d' jumps instead of continuous, 
which is very different from the results obtained using the intersection and union ratio method of pixels.
'''

original_box = (339.15, 230.95, 308.218364151133, 33.12346705650146, 1.5331517813945545)
xc, yc, w, h, ag = original_box

RIoU_list = []
RIoU2_list = []
IoU_list = []
w_deviation_list = range(-20, 21)

for w_deviation in w_deviation_list:
    prediction_box = (xc, yc, w + w_deviation, h, ag)

    # Caculate IoU using rbbox_overlaps
    original_box_tensor = torch.Tensor(original_box).unsqueeze(0)
    prediction_box_tensor = torch.Tensor(prediction_box).unsqueeze(0)
    rbbox_overlaps_calculator = RBboxOverlaps2D()
    RIoU = rbbox_overlaps_calculator(original_box_tensor, prediction_box_tensor, mode='iou', is_aligned=False, version='le135')
    RIoU_list.append(RIoU.item())

    # Caculate IoU using diff_iou_rotated_2d
    device = torch.device('cuda:0')
    original_box_tensor_GPU = original_box_tensor.unsqueeze(0).to(device)
    prediction_box_tensor_GPU = prediction_box_tensor.unsqueeze(0).to(device)
    RIoU2 = diff_iou_rotated_2d(original_box_tensor_GPU, prediction_box_tensor_GPU)
    RIoU2_list.append(RIoU2.item())

    # Caculate IoU using the intersection and union ratio method of pixels
    # x, y, w, h, ag -> p1, p2, p3, p4
    xc2, yc2, w2, h2, ag2 = prediction_box
    wx2, wy2 = w2 / 2 * math.cos(ag2), w2 / 2 * math.sin(ag2)
    hx2, hy2 = -h2 / 2 * math.sin(ag2), h2 / 2 * math.cos(ag2)
    p1_new = (xc2 + wx2 - hx2, yc2 - wy2 + hy2)
    p2_new = (xc2 - wx2 - hx2, yc2 + wy2 + hy2)
    p3_new = (xc2 - wx2 + hx2, yc2 + wy2 - hy2)
    p4_new = (xc2 + wx2 + hx2, yc2 - wy2 - hy2)
    ps_new = [p1_new, p2_new, p3_new, p4_new]

    original_grasp_bboxes = np.array([[[328.4, 76.4], [361.5, 77.6], [349.9, 385.6], [316.8, 384.3]]], dtype=np.int32)      # 4-point representation of original box
    prediction_grasp_bboxes = np.array([ps_new], dtype=np.int32)        # 4-point representation of prediction box
    im = np.zeros((512, 640), dtype="uint8")
    im1 = np.zeros((512, 640), dtype="uint8")
    original_grasp_mask = cv2.fillPoly(im, original_grasp_bboxes, 255)          # original box
    prediction_grasp_mask = cv2.fillPoly(im1, prediction_grasp_bboxes, 255)     # prediction box
    masked_and = cv2.bitwise_and(original_grasp_mask, prediction_grasp_mask, mask=im)   # intersection
    masked_or = cv2.bitwise_or(original_grasp_mask, prediction_grasp_mask)      # union

    or_area = np.sum(np.float32(np.greater(masked_or, 0)))
    and_area = np.sum(np.float32(np.greater(masked_and, 0)))
    IOU = and_area / or_area
    IoU_list.append(IOU)

plt.figure()
plt.plot(w_deviation_list, RIoU_list, label='Caculate results in rbbox_overlaps')
plt.plot(w_deviation_list, RIoU2_list, label='Caculate results in diff_iou_rotated_2d')
plt.plot(w_deviation_list, IoU_list, label='Caculate results in intersection and union ratio of pixels')
plt.legend()
plt.show()

a=1

scnliew321 commented 1 month ago

Hi, thanks for your feedback, can you provide an example to reproduce the problem?

When the w deviation of the prediction box changes within a certain range, the IoU calculated using 'rbbox_overlaps' and 'diff_iou_rotated_2d' jumps instead of continuous, which is very different from the results obtained using the intersection and union ratio method of pixels.

Express gratitude in advance！

import copy

import cv2
import matplotlib.pyplot as plt
import numpy as np
import math
from mmrotate.core.bbox.iou_calculators.rotate_iou2d_calculator import rbbox_overlaps, RBboxOverlaps2D
import torch
from mmcv.ops import diff_iou_rotated_2d

'''
When the w deviation of the prediction box changes within a certain range, 
the IoU calculated using 'rbbox_overlaps' and 'diff_iou_rotated_2d' jumps instead of continuous, 
which is very different from the results obtained using the intersection and union ratio method of pixels.
'''

original_box = (339.15, 230.95, 308.218364151133, 33.12346705650146, 1.5331517813945545)
xc, yc, w, h, ag = original_box

RIoU_list = []
RIoU2_list = []
IoU_list = []
w_deviation_list = range(-20, 21)

for w_deviation in w_deviation_list:
    prediction_box = (xc, yc, w + w_deviation, h, ag)

    # Caculate IoU using rbbox_overlaps
    original_box_tensor = torch.Tensor(original_box).unsqueeze(0)
    prediction_box_tensor = torch.Tensor(prediction_box).unsqueeze(0)
    rbbox_overlaps_calculator = RBboxOverlaps2D()
    RIoU = rbbox_overlaps_calculator(original_box_tensor, prediction_box_tensor, mode='iou', is_aligned=False, version='le135')
    RIoU_list.append(RIoU.item())

    # Caculate IoU using diff_iou_rotated_2d
    device = torch.device('cuda:0')
    original_box_tensor_GPU = original_box_tensor.unsqueeze(0).to(device)
    prediction_box_tensor_GPU = prediction_box_tensor.unsqueeze(0).to(device)
    RIoU2 = diff_iou_rotated_2d(original_box_tensor_GPU, prediction_box_tensor_GPU)
    RIoU2_list.append(RIoU2.item())

    # Caculate IoU using the intersection and union ratio method of pixels
    # x, y, w, h, ag -> p1, p2, p3, p4
    xc2, yc2, w2, h2, ag2 = prediction_box
    wx2, wy2 = w2 / 2 * math.cos(ag2), w2 / 2 * math.sin(ag2)
    hx2, hy2 = -h2 / 2 * math.sin(ag2), h2 / 2 * math.cos(ag2)
    p1_new = (xc2 + wx2 - hx2, yc2 - wy2 + hy2)
    p2_new = (xc2 - wx2 - hx2, yc2 + wy2 + hy2)
    p3_new = (xc2 - wx2 + hx2, yc2 + wy2 - hy2)
    p4_new = (xc2 + wx2 + hx2, yc2 - wy2 - hy2)
    ps_new = [p1_new, p2_new, p3_new, p4_new]

    original_grasp_bboxes = np.array([[[328.4, 76.4], [361.5, 77.6], [349.9, 385.6], [316.8, 384.3]]], dtype=np.int32)      # 4-point representation of original box
    prediction_grasp_bboxes = np.array([ps_new], dtype=np.int32)        # 4-point representation of prediction box
    im = np.zeros((512, 640), dtype="uint8")
    im1 = np.zeros((512, 640), dtype="uint8")
    original_grasp_mask = cv2.fillPoly(im, original_grasp_bboxes, 255)          # original box
    prediction_grasp_mask = cv2.fillPoly(im1, prediction_grasp_bboxes, 255)     # prediction box
    masked_and = cv2.bitwise_and(original_grasp_mask, prediction_grasp_mask, mask=im)   # intersection
    masked_or = cv2.bitwise_or(original_grasp_mask, prediction_grasp_mask)      # union

    or_area = np.sum(np.float32(np.greater(masked_or, 0)))
    and_area = np.sum(np.float32(np.greater(masked_and, 0)))
    IOU = and_area / or_area
    IoU_list.append(IOU)

plt.figure()
plt.plot(w_deviation_list, RIoU_list, label='Caculate results in rbbox_overlaps')
plt.plot(w_deviation_list, RIoU2_list, label='Caculate results in diff_iou_rotated_2d')
plt.plot(w_deviation_list, IoU_list, label='Caculate results in intersection and union ratio of pixels')
plt.legend()
plt.show()

a=1

Hi, I am facing the same problem, is there any updates regarding this matter?

open-mmlab / mmcv