Striveworks / valor

Valor is a centralized evaluation store which makes it easy to measure, explore, and rank model performance.

https://striveworks.github.io/valor/

Other

38 stars 4 forks source link

Lite Classification Accuracy Fix #798

Closed czaloom closed 1 month ago

czaloom commented 1 month ago

Changes

The Accuracy metric is now computed over all labels rather than being per-label.

Old Schema (per-label)

{
    "type": "Accuracy",
    "value": [2 / 3, 2 / 3],
    "parameters": {
        "score_thresholds": [0.25, 0.75],
        "hardmax": True,
        "label": "0",
    },
},
{
    "type": "Accuracy",
    "value": [1 / 3, 1 / 3],
    "parameters": {
        "score_thresholds": [0.25, 0.75],
        "hardmax": True,
        "label": "1",
    },
},

New Schema (aggregate)

{
    "type": "Accuracy",
    "value": [2 / 3, 1 / 3],
    "parameters": {
        "score_thresholds": [0.25, 0.75],
        "hardmax": True,
    },
},