Open Udayraj123 opened 7 months ago
Crosslinking discord thread: https://discord.com/channels/590134763784896514/1270815064256938183/1270816309940326572
Types of accuracy metrics
- csv-based accuracy benchmark (using moderation portal)
- Needs to fit in the per-directory approach
- bubble detection/reading level accuracy
- confidence metrics
- preProcessor level accuracies
- confidence metrics for the matchers
- aggregate metric
- speed graphs to identify
- Since we have 4-5 confidence metrics and 2-3 accuracy metrics, it'd be great to find a color spectrum that caters to these (color strips of 3 at a time if needed). Where all-white means most accurate and all-black means least accurate, and the weighted magnitude of the color gives the numeric representation.
- Can also make a Radar chart of this for explanation purposes as it's across different dimensions.
Requirements
Software accuracy report
Ideas to start with
Further ideas
This implementation can be extended to create a test benchmark for any existing OMR checking software. We need to blackbox the logic and standardise the inputs and outputs in this case.