NaN precision/recall

[cinabars]

I'm running motmetrics on some non-MOTChallenge results and observed the following edge case behavior which I think should be changed:

• For a sequence where there are ground truths but no detections, precision and its related metrics are calculated as NaN.
  • These are essentially false negatives and should result in a precision of 0 rather than NaN. (None of the detected objects were relevant, because no objects were detected.)
• For a sequence where there are no ground truths but nonzero detections, recall and its related metrics are calculated as NaN.
  • These are essentially false positives and should result in a recall of 0. (None of the relevant objects were detected.)

I suppose a result of NaN is technically correct, but in my view a NaN does not properly punish the bad performance in these two cases. NaNs could also ruin calculation of compute_many when one sequence falls under this category.

If you agree with me that this should be changed, let me know and I'll be happy to make a pull request. Otherwise, I'd like to hear your thoughts.

[jvlmdr]

Hey,

Yep, as you mentioned, this is technically correct and the intended behaviour, since the precision and recall are effectively undefined. I'm open to arguments for returning 0 instead of nan. However, I think one could also make an argument for returning 1 (i.e. there were 0 FP for precision, or 0 FN for recall). One advantage of returning nan is that it is more meaningful: one can see immediately what happened.

For compute_many(), I think these nans generally don't pose a problem because we compute sum(numerators) / sum(denominators) instead of mean(numerators / denominators).

[cinabars]

Ah sorry, I got a little mixed up. I think the result should be 1 for these cases (not nan or 0). Given:

• precision = TP / (TP + FP)
• recall = TP / (TP + FN)

Then we could arguably say:

• if FP == 0, then precision = TP / TP = 1, even if TP is also 0.
• if FN == 0, then recall = TP / TP = 1, even if TP is also 0.

So I would argue returning 1 is "correct" behavior. But if you prefer to leave it as NaN to keep it pure, I can respect that too.

[cinabars]

Actually yes, I think I agree with you that it should stay NaN. Nevermind, and thanks for discussing with me!

[jvlmdr]

No worries! Happy to discuss :)

[cheind]

I close this problem as solved. If not, feel free to reopen.