search

Janelia-Trackathon-2023 / traccuracy

Utilities for computing common accuracy metrics on cell tracking challenge solutions with ground truth
https://traccuracy.readthedocs.io
Other
25 stars 6 forks source link

Edge case in Mitotic Branching Correctness #157

Open lmanan opened 2 weeks ago

lmanan commented 2 weeks ago

For certain datasets, where there are no ground truth divisions and an employed tracker model also predicts no divisions, in such a scenario, the number of true positive divisions would be equal to 0, false positive divisions would be equal to 0 and false negative divisions would be equal to 0.

In this scenario, reporting the mitotic branching correctness measure equal to zero (see here) might be misleading (?). Maybe a warning can be raised or np.nan can be returned (similar to how it is done for evaluation of F1 score here, see documentation for zero_division parameter), so that it is excluded from any further evaluation.

DragaDoncila commented 2 weeks ago

@lmanan thanks for raising this issue! I agree that reporting a 0 is misleading.

I would lean towards both a warning and a NaN, as the warning is only useful at runtime while the NaN could be written to file and persist even if the user wasn't looking at the terminal output.

I would probably vote against making it a settable argument like they have in the F1 score for scikit-learn, because I think it's additional complexity without much benefit. @cmalinmayor @msschwartz21 curious on your thoughts?

msschwartz21 commented 2 weeks ago

@DragaDoncila I agree with everything you proposed 👍