Striveworks / valor

Valor is a centralized evaluation store which makes it easy to measure, explore, and rank model performance.
https://striveworks.github.io/valor/
Other
38 stars 4 forks source link

BUG: Object Detection IOU Computation #816

Closed czaloom closed 2 weeks ago

czaloom commented 2 weeks ago

valor version checks

Reproducible Example

def test_iou_computation():

    detection = Detection(
        uid="uid",
        groundtruths=[
            BoundingBox(xmin=0, xmax=10, ymin=0, ymax=10, labels=["0"]),
            BoundingBox(xmin=100, xmax=110, ymin=100, ymax=110, labels=["0"]),
            BoundingBox(
                xmin=1000, xmax=1100, ymin=1000, ymax=1100, labels=["0"]
            ),
        ],
        predictions=[
            BoundingBox(
                xmin=1,
                xmax=11,
                ymin=1,
                ymax=11,
                labels=["0", "1", "2"],
                scores=[0.5, 0.25, 0.25],
            ),
            BoundingBox(
                xmin=105,
                xmax=116,
                ymin=105,
                ymax=116,
                labels=["0", "1", "2"],
                scores=[0.5, 0.25, 0.25],
            ),
        ],
    )

    loader = DataLoader()
    loader.add_bounding_boxes([detection])

    assert len(loader.pairs) == 1

    # show that three unique IOUs exist
    unique_ious = np.unique(loader.pairs[0][:, 3])
    assert np.isclose(
        unique_ious, np.array([0.0, 0.12755102, 0.68067227])
    ).all()

Issue Description

There are two serious bugs.

  1. IOU is being computed after label permutation which means that a ton of work is being redone.
  2. (1) was introduced at some point without updating the IOU allocation. This means that all IOUs were identical for every pair within an image. This did not affect the results between images.

Expected Behavior

IOU should be computed the minimum number of times and should be properly reported per groundtruth-prediction pair.