Ambiguity in counting TP cases (gt_tp != tp)

[Joeycho]

Dear @vpchung, @rachitsaluja,

First of all, thank you for sharing lesion-wise evaluation code. This helps that I understand that how SOTA lesion-wise evaluation is done currently. And it allows me to check each metrics, how they are calculated.

I have tried to count TP, FN, FP cases from all lesions to report detection accuracy (F1-score). But I recognized that counting TP is not trivial if there are many-one, one-many overlaps between the lesions. I briefly illustrated in the attached image.

Another point is in other lesion-wise approaches [1, 2] adopted >dice-score 0.1 for the overlap criteria to ensure the certain level of detection for big size lesions. Why is this not counted in the current Brats code?

Ultimately, can we move forward to the robust and reliable lesion-wise evaluation in medical image segmentation? The issue is now.. it depends on several hyperparameters (dialation factor.. voxel size limit) and as I presented it contains.. ambiguity.

Metrics reloaded [3] mentions this issue as an option, 'double assignment FP2.5.8'. Either treat additional matches as FPs, or ignore them.

[1] Lesion-wise evaluation for effective performance monitoring of small object segmentation, Irme Groothuis et al. 2021 [2] Automated detection and quantification of brain metastases on clinical MRI data using artificial neural networks, Irada Pflüger et al. 2022

[aymuos15]

This is my take, not sure if it is completely correct:

https://colab.research.google.com/drive/1yKAvh_zvKt8g4eCjetN0n6L1dfk0_yhh?usp=sharing

I feel this example explains well about why they use the prediction based TPs to calculate the actual FPs and use that in the final calculation.

for cc in intersecting_cc:
     tp.append(cc) #pred based tps calulated here.

     ...

fp = np.unique(
            pred_label_cc[np.isin(
                pred_label_cc,tp+[0],invert=True)]) #fps calculated based on pred based tps.

If you see my result on the colab, the gt tps are better to understand the overall "actual" tp count because the pred tps are based on the indexed values of the preds.

gt_tp -> [1, 2, 3, 4] (as you see all gt is accounted for) pred_tp-> [1, 1, 1, 2, 1, 2] (only the preds are accounted for).

Given that the metric/matching is based on iterating over the GT, this makes more sense to me atleast.

[Joeycho]

@aymuos15 Hi, good to see you here, and thank you for your explanation.

Hmm.. what if a user gets the exact opposite case? In your example, a user gets a prediction (4 lesions as same as Ground Truth (yours)), and have Ground Truth (2 lesions as Prediction (yours))? I got lesion-wise dice 0.4926.

0.29267 (gt-based), 0.4926 (prediction-based), which one looks better representative metric? Or should they be the same?

And maybe.. gt_tp should be [1,2,3,4,3,4], if pred_tp = [1,1,1,2,1,2]

[aymuos15]

Yeah, so this scheme works on the basis of many-to-one matching only for predictions to ground truth. Not the other way around. Something like panoptic quality. So 2 ground truths can not be matched to a single prediction so to say.