Striveworks / valor

Valor is a centralized evaluation store which makes it easy to measure, explore, and rank model performance.

https://striveworks.github.io/valor/

Other

38 stars 4 forks source link

Lite ObjDet Accuracy Fix #801

Closed czaloom closed 1 month ago

czaloom commented 1 month ago

Changes

The Accuracy metric is now computed over all labels rather than being per-label.

Old Schema (per-label)

{
    "type": "Accuracy",
    "value": 2 / 3,
    "parameters": {
        "score_threshold": 0.25,
        "iou_threshold": 0.25,
        "label": "0",
    },
},
{
    "type": "Accuracy",
    "value":  1 / 3,
    "parameters": {
        "score_threshold": 0.25,
        "iou_threshold": 0.25,
        "label": "1",
    },
},

New Schema (aggregate)

{
    "type": "Accuracy",
    "value": 2 / 3,
    "parameters": {
        "score_threshold": 0.25,
        "iou_threshold": 0.25,
    },
},