Issue with mAP function

[pyaf]

Hey,

Currently, mAP function get_average_precisions fails when for a particular category there are no ground truths boxes but there are predicted ones. (possibly for a bad validation set)

We get divide-by-zero error here for such cases.

We make sure the class_name is present in predicted boxes here we should also make sure that the same class name is present in ground truth boxes too.

So, This can be fixed by replacing

        if class_name in pred_by_class_name:

by

        if class_name in pred_by_class_name and class_name in gt_by_class_name:

[ternaus]

This is an interesting corner case.

How did this happen that there are no gt labels but there are ones that are predicted?

I can modify the code for the metric so that it would throw an exception if there are classes that are not in ground truth.

[pyaf]

We may have such a situation if we have an unstratified validation set, like in this Kaggle kernel: https://www.kaggle.com/gzuidhof/reference-model there's no ground truth box for emergency_vehicle in the validation set, and because of which we get a divide-by-zero error in mAP evaluation code.

[ternaus]

I would say the this is the issue of the data preparation. We should have all the classes that we want to evaluate in the validation file.

The overall metric should be calculated over all classes. Number of classes for metric calculation is defined by the number of classes in the ground truth file.

=> if we have fewer classes there => mAP will be misleading.

If someone wants to have a metric for a subset of classes, they should probably call

            recalls, precisions, average_precision = recall_precision(
                gt_by_class_name[class_name], pred_by_class_name[class_name], iou_threshold
            )

for the classes of interest.

[pyaf]

Makes sense.