Make `InferenceSlicer` compatible with segmentation models

[mario-dg]

Search before asking

• [X] I have searched the Supervision issues and found no similar bug report.

Bug

The problem probably originates in the Detections class, where merging a list of Detections with masks predicted from a YOLOv8 segmentation model might fail to merge. The step of merging the predicted masks code-snippet can fail, when the final slices in the row and column dimensions are smaller than the specified slice dimension.

In addition, when calling move_detections only the bounding boxes are considered. It would be nice to maybe have a contours attribute in the Detections class, that additionally stores a list of contours for each mask, which then can be correctly moved according to the offset.

Environment

No response

Minimal Reproducible Example

No response

Additional

No response

Are you willing to submit a PR?

• [ ] Yes I'd like to help by submitting a PR!

[github-actions[bot]]

Hello there, thank you for opening an Issue ! 🙏🏻 The team was notified and they will get back to you asap.

[onuralpszr]

Hello, @mario-dg do you have small code or collab for trigger it so we can test as well.

[mario-dg]

Yes, sure!

import supervision as sv
import numpy as np
from ultralytics import YOLO
from PIL import Image

model = YOLO("yolov8s-seg.pt")

def callback(x):
    result = model(x, verbose=True, conf=0.15)[0]
    return sv.Detections.from_ultralytics(result)

slicer = sv.InferenceSlicer(
    callback=callback,
    slice_wh=(512, 512),
    iou_threshold=0.5,
)
sliced_detection = slicer(image=np.asarray(Image.open("image.jpg")))

The relevant part for this issue to occur, is that every slice needs to detect any object. Thus, all slice detections have mask != None, resulting in np.vstack() to fail

[mario-dg]

Is there any plan to further looking into correctly retrieving segmentation results from the InferenceSlicer?

If somebody can point me to an idea or an entry, I could work on a PR.

[SkalskiP]

Hi, @mario-dg 👋🏻! I'm sorry for such a late response. I've been on vacation for the past two weeks, and now I'm trying to catch up.

For now, our InferenceSlicer can only work with bounding boxes. @hardikdava, am I correct?

[hardikdava]

@SkalskiP yes, Slicer only works with bounding box for now.

[SkalskiP]

Okey. In that case, let me update the title and tag of this issue. I'll convert it from a bug report into a feature request.

[SkalskiP]

Hi everyone 👋🏻 We are already tracking this issue in: https://github.com/roboflow/supervision/issues/678. I'm marking this issue as a duplicate. For your information, we plan to tackle this feature in the first quarter of 2024.