YOLOv8-obb map

[liu15509348793]

Search before asking

• [X] I have searched the Ultralytics YOLO issues and discussions and found no similar questions.

Question

Thank you for your work, I found that when using yolov8obb to calculate map, I use probiou to calculate, I am running the DOTA 1.0 dataset, but the map obtained in this way does not seem to be able to compare with the results of other models, because different iou is used, is there a common iou used to replace probiou when calculating map in DOTA?

Additional

No response

[UltralyticsAssistant]

👋 Hello @liu15509348793, thank you for reaching out to Ultralytics 🚀! We suggest checking the Docs where you might find information related to your query about mAP and IoU calculations. Specifically, you can explore more about model performance metrics in our Python and CLI guides.

If this is a 🐛 Bug Report, please provide a minimum reproducible example so we can assist in debugging it effectively.

Join our community! For real-time chat, head to Discord 🎧. Prefer detailed discussions? Visit Discourse or share insights on our Subreddit.

Upgrade

Ensure you have the latest ultralytics package with all requirements installed in a Python>=3.8 environment with PyTorch>=1.8:

pip install -U ultralytics

Environments

YOLOv8 can be run in several verified environments:

• Notebooks with GPUs:
• Google Cloud: See GCP Quickstart Guide
• Amazon: See AWS Quickstart Guide
• Docker: See Docker Quickstart Guide

Status

If this badge is green, all Ultralytics CI tests are currently passing. Our CI tests ensure correct operation across macOS, Windows, and Ubuntu for all YOLOv8 models.

This is an automated response; one of our Ultralytics engineers will assist you soon. 😊

[Y-T-G]

I don't understand the question.

Are you saying the calculation performed by ultralytics is not matching with your own calculation, or that the calculation performed by ultralytics doesn't match with other works?

[liu15509348793]

我不明白这个问题。

你是说 ultralytics 执行的计算与你自己的计算不匹配，还是 ultralytics 执行的计算与其他工作不匹配？

Hello, what I'm saying is that it is not possible to compare with other work, because in yolov8obb the calculation of map is using probiou, and the work of DOTA seems to be using ployiou

[Y-T-G]

I see. Is this polyiou?

https://github.com/PaddlePaddle/PaddleDetection/blob/55bad87fffee251ae19592a1ac3df42906299e28/configs/rotate/tools/onnx_infer.py#L157

[liu15509348793]

I see. Is this polyiou?

https://github.com/PaddlePaddle/PaddleDetection/blob/55bad87fffee251ae19592a1ac3df42906299e28/configs/rotate/tools/onnx_infer.py#L157

Yes, does yolov8obb provide a way to calculate the map using this iou so that it can be compared with other people's work

[mysort]

other object detection models use poly_iou to calculate the map. please provide this code

[Y-T-G]

I don't think so. You will have to do it manually.

I created a monkey patch based method:

from ultralytics.utils.ops import xywhr2xyxyxyxy      
from ultralytics import YOLO
import ultralytics.models.yolo.obb.val as val
import ultralytics.utils.metrics as metrics
import torch
from shapely.geometry import Polygon

def rbox_iou(g, p):
    g = Polygon(g.cpu().numpy())
    p = Polygon(p.cpu().numpy())        
    g = g.buffer(0)
    p = p.buffer(0)
    if not g.is_valid or not p.is_valid:
        return 0
    inter = Polygon(g).intersection(Polygon(p)).area
    union = g.area + p.area - inter
    if union == 0:
        return 0
    else:
        return inter / union

def batch_probiou_monkey_patch(obb1, obb2, eps=1e-7):
  obb1 = xywhr2xyxyxyxy(obb1)
  obb2 = xywhr2xyxyxyxy(obb2)
  N, M = obb1.shape[0], obb2.shape[0]
  similarities = torch.zeros((N, M))

  for i, obb1_i in enumerate(obb1):
        for j, obb2_j in enumerate(obb2):
          similarities[i, j] = rbox_iou(obb1_i, obb2_j)
  return similarities.to("cuda:0")
val.batch_probiou = batch_probiou_monkey_patch
metrics.batch_probiou = batch_probiou_monkey_patch

model = YOLO("yolov8n-obb.pt")
results = model.val(data="dota8.yaml")

[liu15509348793]

I don't think so. You will have to do it manually.

I created a monkey patch based method:

from ultralytics.utils.ops import xywhr2xyxyxyxy      
from ultralytics import YOLO
import ultralytics.models.yolo.obb.val as val
import ultralytics.utils.metrics as metrics
import torch
from shapely.geometry import Polygon

def rbox_iou(g, p):
    g = Polygon(g.cpu().numpy())
    p = Polygon(p.cpu().numpy())        
    g = g.buffer(0)
    p = p.buffer(0)
    if not g.is_valid or not p.is_valid:
        return 0
    inter = Polygon(g).intersection(Polygon(p)).area
    union = g.area + p.area - inter
    if union == 0:
        return 0
    else:
        return inter / union

def batch_probiou_monkey_patch(obb1, obb2, eps=1e-7):
  obb1 = xywhr2xyxyxyxy(obb1)
  obb2 = xywhr2xyxyxyxy(obb2)
  N, M = obb1.shape[0], obb2.shape[0]
  similarities = torch.zeros((N, M))

  for i, obb1_i in enumerate(obb1):
        for j, obb2_j in enumerate(obb2):
          similarities[i, j] = rbox_iou(obb1_i, obb2_j)
  return similarities.to("cuda:0")
val.batch_probiou = batch_probiou_monkey_patch
metrics.batch_probiou = batch_probiou_monkey_patch

model = YOLO("yolov8n-obb.pt")
results = model.val(data="dota8.yaml")

Thanks for the solution, after I debugged the way you did, I found that the batch_probiou used in nms is still the same as the original batch_probiou, but the batch_probiou used in the _process_batch function in OBBValidator has been changed to a new batch_probiou_monkey_patch, I'm not quite sure what the reason for this is, I would appreciate it if you could help me out

[mysort]

I don't think so. You will have to do it manually.

I created a monkey patch based method:

from ultralytics.utils.ops import xywhr2xyxyxyxy      
from ultralytics import YOLO
import ultralytics.models.yolo.obb.val as val
import ultralytics.utils.metrics as metrics
import torch
from shapely.geometry import Polygon

def rbox_iou(g, p):
    g = Polygon(g.cpu().numpy())
    p = Polygon(p.cpu().numpy())        
    g = g.buffer(0)
    p = p.buffer(0)
    if not g.is_valid or not p.is_valid:
        return 0
    inter = Polygon(g).intersection(Polygon(p)).area
    union = g.area + p.area - inter
    if union == 0:
        return 0
    else:
        return inter / union

def batch_probiou_monkey_patch(obb1, obb2, eps=1e-7):
  obb1 = xywhr2xyxyxyxy(obb1)
  obb2 = xywhr2xyxyxyxy(obb2)
  N, M = obb1.shape[0], obb2.shape[0]
  similarities = torch.zeros((N, M))

  for i, obb1_i in enumerate(obb1):
        for j, obb2_j in enumerate(obb2):
          similarities[i, j] = rbox_iou(obb1_i, obb2_j)
  return similarities.to("cuda:0")
val.batch_probiou = batch_probiou_monkey_patch
metrics.batch_probiou = batch_probiou_monkey_patch

model = YOLO("yolov8n-obb.pt")
results = model.val(data="dota8.yaml")

Thanks for the answer, theoretically the map with polyiou should be lower than with probiou. But after using your code, my map is higher. I'm very confused about this phenomenon.

[liu15509348793]

我不这么认为。您将不得不手动完成。 我创建了一个基于 monkey patch 的方法：

from ultralytics.utils.ops import xywhr2xyxyxyxy      
from ultralytics import YOLO
import ultralytics.models.yolo.obb.val as val
import ultralytics.utils.metrics as metrics
import torch
from shapely.geometry import Polygon

def rbox_iou(g, p):
    g = Polygon(g.cpu().numpy())
    p = Polygon(p.cpu().numpy())        
    g = g.buffer(0)
    p = p.buffer(0)
    if not g.is_valid or not p.is_valid:
        return 0
    inter = Polygon(g).intersection(Polygon(p)).area
    union = g.area + p.area - inter
    if union == 0:
        return 0
    else:
        return inter / union

def batch_probiou_monkey_patch(obb1, obb2, eps=1e-7):
  obb1 = xywhr2xyxyxyxy(obb1)
  obb2 = xywhr2xyxyxyxy(obb2)
  N, M = obb1.shape[0], obb2.shape[0]
  similarities = torch.zeros((N, M))

  for i, obb1_i in enumerate(obb1):
        for j, obb2_j in enumerate(obb2):
          similarities[i, j] = rbox_iou(obb1_i, obb2_j)
  return similarities.to("cuda:0")
val.batch_probiou = batch_probiou_monkey_patch
metrics.batch_probiou = batch_probiou_monkey_patch

model = YOLO("yolov8n-obb.pt")
results = model.val(data="dota8.yaml")

感谢您的回答，理论上带有 polyiou 的地图应该低于带有 probiou 的地图。但是在使用您的代码之后，我的 map 更高。我对这种现象感到非常困惑。

Hello, have you checked that the batch_probiou in nms is actually still called the original batch_probiou?

[mysort]

我不这么认为。您将不得不手动完成。 我创建了一个基于 monkey patch 的方法：

from ultralytics.utils.ops import xywhr2xyxyxyxy      
from ultralytics import YOLO
import ultralytics.models.yolo.obb.val as val
import ultralytics.utils.metrics as metrics
import torch
from shapely.geometry import Polygon

def rbox_iou(g, p):
    g = Polygon(g.cpu().numpy())
    p = Polygon(p.cpu().numpy())        
    g = g.buffer(0)
    p = p.buffer(0)
    if not g.is_valid or not p.is_valid:
        return 0
    inter = Polygon(g).intersection(Polygon(p)).area
    union = g.area + p.area - inter
    if union == 0:
        return 0
    else:
        return inter / union

def batch_probiou_monkey_patch(obb1, obb2, eps=1e-7):
  obb1 = xywhr2xyxyxyxy(obb1)
  obb2 = xywhr2xyxyxyxy(obb2)
  N, M = obb1.shape[0], obb2.shape[0]
  similarities = torch.zeros((N, M))

  for i, obb1_i in enumerate(obb1):
        for j, obb2_j in enumerate(obb2):
          similarities[i, j] = rbox_iou(obb1_i, obb2_j)
  return similarities.to("cuda:0")
val.batch_probiou = batch_probiou_monkey_patch
metrics.batch_probiou = batch_probiou_monkey_patch

model = YOLO("yolov8n-obb.pt")
results = model.val(data="dota8.yaml")

感谢您的回答，理论上带有 polyiou 的地图应该低于带有 probiou 的地图。但是在使用您的代码之后，我的 map 更高。我对这种现象感到非常困惑。

Hello, have you checked that the batch_probiou in nms is actually still called the original batch_probiou?

sorry I didn't do this. I just copy the code and created a test.py then run it.

[Y-T-G]

You can also monkey patch the NMS batch prob

import ultralytics.utils.ops as ops

ops.batch_probiou = batch_probiou_monkey_patch

But it's slow since it's not vectorized

[liu15509348793]

You can also monkey patch the NMS batch prob

import ultralytics.utils.ops as ops

ops.batch_probiou = batch_probiou_monkey_patch

But it's slow since it's not vectorized

I've found that ops.py is also used: from ultralytics.utils.metrics import batch_probiou However, after following your changes above, the original batch_probiou will still be called. Can I just edit the original batch_probiou?

[Y-T-G]

You can edit it. But it shouldn't be called. It would be using the monkey patched version if you added the above code along with the old code I posted.

[liu15509348793]

You can edit it. But it shouldn't be called. It would be using the monkey patched version if you added the above code along with the old code I posted.

Thanks for the reply, if I want to change the iou when the map is calculated, the batch_probiou in both val and nms should be replaced with batch_probiou_monkey_patch, right? I'm a little unsure if it should be like this? Can you help me with that? If yes, then I just modify the definition of the ultralytics.utils.metrics.batch_probiou so that I can compare the results of my yolov8obb model with other models, right? For example, the model in mmrotate

[Y-T-G]

All of them use the batch_probiou from metrics.py. If you're editing code, you just edit that.

[liu15509348793]

All of them use the batch_probiou from metrics.py. If you're editing code, you just edit that.

Okay, thank you very much

[Y-T-G]

If you install mmcv, you can use this. It's faster:

import torch
from ultralytics import YOLO
import ultralytics.models.yolo.obb.val as val
import ultralytics.utils.metrics as metrics
import ultralytics.utils.ops as ops
from mmcv.ops import box_iou_rotated

def rbbox_overlaps(bboxes1, bboxes2, eps=1e-7, mode='iou', is_aligned=False):
    """Calculate overlap between two set of bboxes.

    Args:
        bboxes1 (torch.Tensor): shape (B, m, 5) in <cx, cy, w, h, a> format
            or empty.
        bboxes2 (torch.Tensor): shape (B, n, 5) in <cx, cy, w, h, a> format
            or empty.
        mode (str): "iou" (intersection over union), "iof" (intersection over
            foreground) or "giou" (generalized intersection over union).
            Default "iou".
        is_aligned (bool, optional): If True, then m and n must be equal.
            Default False.

    Returns:
        Tensor: shape (m, n) if ``is_aligned`` is False else shape (m,)
    """
    assert mode in ['iou', 'iof']
    # Either the boxes are empty or the length of boxes's last dimension is 5
    assert (bboxes1.size(-1) == 5 or bboxes1.size(0) == 0)
    assert (bboxes2.size(-1) == 5 or bboxes2.size(0) == 0)

    rows = bboxes1.size(0)
    cols = bboxes2.size(0)
    if is_aligned:
        assert rows == cols

    if rows * cols == 0:
        return bboxes1.new(rows, 1) if is_aligned else bboxes1.new(rows, cols)

    # resolve `rbbox_overlaps` abnormal when input rbbox is too small.
    clamped_bboxes1 = bboxes1.detach().clone()
    clamped_bboxes2 = bboxes2.detach().clone()
    clamped_bboxes1[:, 2:4].clamp_(min=1e-3)
    clamped_bboxes2[:, 2:4].clamp_(min=1e-3)

    return box_iou_rotated(clamped_bboxes1, clamped_bboxes2, mode, is_aligned)

# Monkey patch the original function
val.batch_probiou = rbbox_overlaps
metrics.batch_probiou = rbbox_overlaps
ops.batch_probiou = rbbox_overlaps

# Usage example
model = YOLO("yolov8n-obb.pt")
results = model.val(data="dota8.yaml")

[liu15509348793]

If you install mmcv, you can use this. It's faster:

import torch
from ultralytics import YOLO
import ultralytics.models.yolo.obb.val as val
import ultralytics.utils.metrics as metrics
import ultralytics.utils.ops as ops
from mmcv.ops import box_iou_rotated

def rbbox_overlaps(bboxes1, bboxes2, eps=1e-7, mode='iou', is_aligned=False):
    """Calculate overlap between two set of bboxes.

    Args:
        bboxes1 (torch.Tensor): shape (B, m, 5) in <cx, cy, w, h, a> format
            or empty.
        bboxes2 (torch.Tensor): shape (B, n, 5) in <cx, cy, w, h, a> format
            or empty.
        mode (str): "iou" (intersection over union), "iof" (intersection over
            foreground) or "giou" (generalized intersection over union).
            Default "iou".
        is_aligned (bool, optional): If True, then m and n must be equal.
            Default False.

    Returns:
        Tensor: shape (m, n) if ``is_aligned`` is False else shape (m,)
    """
    assert mode in ['iou', 'iof']
    # Either the boxes are empty or the length of boxes's last dimension is 5
    assert (bboxes1.size(-1) == 5 or bboxes1.size(0) == 0)
    assert (bboxes2.size(-1) == 5 or bboxes2.size(0) == 0)

    rows = bboxes1.size(0)
    cols = bboxes2.size(0)
    if is_aligned:
        assert rows == cols

    if rows * cols == 0:
        return bboxes1.new(rows, 1) if is_aligned else bboxes1.new(rows, cols)

    # resolve `rbbox_overlaps` abnormal when input rbbox is too small.
    clamped_bboxes1 = bboxes1.detach().clone()
    clamped_bboxes2 = bboxes2.detach().clone()
    clamped_bboxes1[:, 2:4].clamp_(min=1e-3)
    clamped_bboxes2[:, 2:4].clamp_(min=1e-3)

    return box_iou_rotated(clamped_bboxes1, clamped_bboxes2, mode, is_aligned)

# Monkey patch the original function
val.batch_probiou = rbbox_overlaps
metrics.batch_probiou = rbbox_overlaps
ops.batch_probiou = rbbox_overlaps

# Usage example
model = YOLO("yolov8n-obb.pt")
results = model.val(data="dota8.yaml")

The speed has really improved a lot, I'm going to see how it works, thank you for your help

[glenn-jocher]

You're welcome! If you have any more questions or need further assistance, feel free to ask.

[oahnah]

If you install , you can use this. It's faster:mmcv

import torch
from ultralytics import YOLO
import ultralytics.models.yolo.obb.val as val
import ultralytics.utils.metrics as metrics
import ultralytics.utils.ops as ops
from mmcv.ops import box_iou_rotated

def rbbox_overlaps(bboxes1, bboxes2, eps=1e-7, mode='iou', is_aligned=False):
    """Calculate overlap between two set of bboxes.

    Args:
        bboxes1 (torch.Tensor): shape (B, m, 5) in <cx, cy, w, h, a> format
            or empty.
        bboxes2 (torch.Tensor): shape (B, n, 5) in <cx, cy, w, h, a> format
            or empty.
        mode (str): "iou" (intersection over union), "iof" (intersection over
            foreground) or "giou" (generalized intersection over union).
            Default "iou".
        is_aligned (bool, optional): If True, then m and n must be equal.
            Default False.

    Returns:
        Tensor: shape (m, n) if ``is_aligned`` is False else shape (m,)
    """
    assert mode in ['iou', 'iof']
    # Either the boxes are empty or the length of boxes's last dimension is 5
    assert (bboxes1.size(-1) == 5 or bboxes1.size(0) == 0)
    assert (bboxes2.size(-1) == 5 or bboxes2.size(0) == 0)

    rows = bboxes1.size(0)
    cols = bboxes2.size(0)
    if is_aligned:
        assert rows == cols

    if rows * cols == 0:
        return bboxes1.new(rows, 1) if is_aligned else bboxes1.new(rows, cols)

    # resolve `rbbox_overlaps` abnormal when input rbbox is too small.
    clamped_bboxes1 = bboxes1.detach().clone()
    clamped_bboxes2 = bboxes2.detach().clone()
    clamped_bboxes1[:, 2:4].clamp_(min=1e-3)
    clamped_bboxes2[:, 2:4].clamp_(min=1e-3)

    return box_iou_rotated(clamped_bboxes1, clamped_bboxes2, mode, is_aligned)

# Monkey patch the original function
val.batch_probiou = rbbox_overlaps
metrics.batch_probiou = rbbox_overlaps
ops.batch_probiou = rbbox_overlaps

# Usage example
model = YOLO("yolov8n-obb.pt")
results = model.val(data="dota8.yaml")

Hello, what is the purpose of this code? Is it replacing the original mAP calculation in order to compare it with other algorithms?

[glenn-jocher]

Yes, this code replaces the original mAP calculation with a different IoU method to allow for comparison with other algorithms that use a similar approach.