search

RI-SE / smirk

SMIRK is an experimental pedestrian emergency breaking ADAS facilitating research on quality assurance of critical components that rely on machine learning.
GNU General Public License v3.0
16 stars 4 forks source link

SYS-PER-REQ3 shall be relaxed #13

Closed mrksbrg closed 2 years ago

mrksbrg commented 2 years ago

Evaluation of object detection models is non-trivial and relies on the metric intersection over union. The fact that false positives (FP) appear due to low IoU scores despite parts of a pedestrian indeed is detected is often counter-intuitive, i.e., "how can a detected pedestrian ever be an FP?" An FP means that the intersection over union between a predicted bounding box and a ground truth bounding box is less that 0.5. This does not necessarily mean that the model predicted a non-pedestrian as a reason to break - our model testing shows that this often means that a fraction of a real pedestrian was detected. Commencing emergency braking in such cases would be valid.

The corresponding performance requirement shall be set to 1%. The hazardous non-pedestrian emergency braking will be mitigated by the safety cage architecture.

mrksbrg commented 2 years ago

Thanks to an improved autoencoder for out-of-distribution detection, we do not need to relax the requirement all the way to 1%. The quality target for FPPI in SYS-PER-REQ3 shall be updated from 0.01% to 0.1%. The new value has been validated through testing in ESI Pro-SiVIC.