Instance Segmentation Confusion Matrix

[mayankagarwals]

Search before asking

• [X] I have searched the Supervision issues and found no similar feature requests.

Description

The current evaluation api supports confusion matrix only for object detection. Giving a view on how the model is performing for instance segmentation per class is desirable.

Slight change in API expected:

@dataclass
class ConfusionMatrix:
    matrix: np.ndarray
    segmentationMatrix: np.ndarray
    classes: List[str]
    conf_threshold: float
    iou_threshold: float

    @classmethod
    def from_detections(
        cls,
        predictions: List[sv.Detections],
        target: List[sv.Detections],
        classes: List[str],
        conf_threshold: float = 0.3,
        iou_threshold: float = 0.5
    ) -> ConfusionMatrix:
        pass

   @classmethod
    def benchmark(
        cls,
        dataset: sv.DetectionDataset,
        callback: Callable[[np.ndarray], sv.Detections],
        conf_threshold: float = 0.3,
        iou_threshold: float = 0.5
    ) -> ConfusionMatrix:
        pass

    def plot(self, target_path: str) -> None:
        pass

Discussion points:

1. Note that I'm not changing the previous name matrix to detectionMatrix for backward compatibility. If we believe it's not necessary and brings readability for the long term, can make that change.
2. Should this be an opt-in feature. Not sure what the user behaviour is. If the 90 percentile user wants both let's not. If it's the other way around, we can instead make a separate confusion matrix class called SegmentationConfusionMatrix

Use case

No response

Additional

No response

Are you willing to submit a PR?

• [X] Yes I'd like to help by submitting a PR!

[mayankagarwals]

@SkalskiP @hardikdava @kirilllzaitsev Would like to have your input on the API change. I have implemented mask_iou_batch for an iou score on masks. Everything else should be fairly straight forward

[kirilllzaitsev]

@mayankagarwals , hi, thanks for bringing up this use case. I like the idea of having different ConfusionMatrix subclasses, each implementing in their own way these methods:

• detections_to_tensor
• _validate_input_tensors
• evaluate_detection_batch

And these look generic enough to be left as is, except for documentation:

• from_detections
• from_tensors
• _drop_extra_matches
• benchmark
• plot

@SkalskiP , what do you think?

[mayankagarwals]

Maybe from_tensors is better implemented in a subclass, else our prediction and target batch needs to be an np array. This is fine for now as both polygons and bounding boxes are np arrays.

    @classmethod
    def from_tensors(
        cls,
        predictions: List[np.ndarray],
        targets: List[np.ndarray],
        classes: List[str],
        conf_threshold: float = 0.3,
        iou_threshold: float = 0.5,
    ) -> ConfusionMatrix:

We can keep the following generic:

from_detections
_drop_extra_matches
benchmark
plot

[mayankagarwals]

@SkalskiP @hardikdava would like to have your views on this too

[hardikdava]

@mayankagarwals It would be great to have a single API sv.ConfusionMatrix which can support for object detection and segmentation confusion matrix. Something like check if masks exists then also find confusion matrix for masks, otherwise compute confusion matrix only for bounding box.

[mayankagarwals]

@hardikdava This is a design decision that we need to take based on what supervision ideology + user base is a. If we believe a more significant fraction of our users would want both object detection and segmentation confusion matrices computed every time they compute any confusion matrix, we should go ahead with the solution you have proposed b. If the use cases for these two matrices differ, it would make sense to have a separate function/class to compute it independently

Personally, I am with @hardikdava, the use case generally around these is to evaluate models, the more matrices available the better. The compute cost isn't the primary problem as these matrices are hardly used in production.

For now, I'll go ahead with this. @SkalskiP @kirilllzaitsev please let me know if you feel this is contradictory in anyway.

[mayankagarwals]

@SkalskiP Following up on this, let me know what you think. Would like to start dev soon

[mayankagarwals]

@SkalskiP Do let me know