search

RI-SE / smirk

SMIRK is an experimental pedestrian emergency breaking ADAS facilitating research on quality assurance of critical components that rely on machine learning.
GNU General Public License v3.0
16 stars 4 forks source link

Accuracy is an invalid metric for SYS-PER-REQ1 #8

Closed mrksbrg closed 2 years ago

mrksbrg commented 2 years ago

"SYS-PER-REQ1: The pedestrian recognition component shall identify pedestrians with an accuracy of 93% when they are within 50 m."

True positive rate would be a better metric than accuracy, and also align with SYS-PER-REQ2 that specifies the false negative rate.

mrksbrg commented 2 years ago

To avoid redundant requirements, we shall make sure the TPR in SYS-PER-REQ1 and FPR in SYS-PER-REQ2 are calculated for different distance thresholds.