Metrics for measuring Gabbar performance - Githubissues

mapbox / gabbar

Guarding OpenStreetMap from harmful edits using machine learning

MIT License

19 stars 7 forks source link

Metrics for measuring Gabbar performance #36

Closed bkowshik closed 7 years ago

bkowshik commented 7 years ago

NOTE: The numbers below are for the purpose of illustration only.

Everyday

Total number of changesets that day: 25,000
Number of changesets flagged as problematic by Gabbar: 500 (2%)
Number of changesets manually reviewed on osmcha: 250 (1%)
Confusion matrix between manually reviewed and flagged as problematic changesets:

	Predicted good	Predicted harmful
Labelled Good	200	5
Labelled Harmful	10	35

On the validation set (Changesets labelled on omscha in May, 2017)

Total number of changesets: 5,000
Confusion matrix between manually reviewed and flagged as problematic changesets:

	Predicted good	Predicted harmful
Labelled Good	4500	50
Labelled Harmful	100	350

cc: @batpad

bkowshik commented 7 years ago

No next actions here. Closing.