naga-karthik
commented
6 months ago The first results for pre-training and fine-tuning (both region-based) are in!
Fine-tuning performance on `dcm-zurich-lesions`
``` Test Phase Metrics [ANIMA] for sc: Jaccard --> Mean: 0.944, Std: 0.007 Dice --> Mean: 0.971, Std: 0.004 Sensitivity --> Mean: 0.971, Std: 0.003 Specificity --> Mean: 1.000, Std: 0.000 PPV --> Mean: 0.971, Std: 0.005 NPV --> Mean: 1.000, Std: 0.000 RelativeVolumeError --> Mean: 0.051, Std: 0.499 HausdorffDistance --> Mean: 1.414, Std: 0.000 ContourMeanDistance --> Mean: 0.204, Std: 0.039 SurfaceDistance --> Mean: 0.001, Std: 0.001 Test Phase Metrics [ANIMA] for lesion: Jaccard --> Mean: 0.432, Std: 0.328 Dice --> Mean: 0.517, Std: 0.377 Sensitivity --> Mean: 0.502, Std: 0.361 Specificity --> Mean: 1.000, Std: 0.000 PPV --> Mean: 0.535, Std: 0.395 NPV --> Mean: 1.000, Std: 0.000 RelativeVolumeError --> Mean: -28.241, Std: 32.595 HausdorffDistance --> Mean: 5.333, Std: 6.128 ContourMeanDistance --> Mean: 3.304, Std: 4.316 SurfaceDistance --> Mean: 1.895, Std: 2.646 PPVL --> Mean: 0.667, Std: 0.471 SensL --> Mean: 0.500, Std: 0.408 F1_score --> Mean: 0.556, Std: 0.416 ```Fine-tuning performance on `dcm-zurich-lesions-20231115`
``` Test Phase Metrics [ANIMA] for sc: Jaccard --> Mean: 0.933, Std: 0.023 Dice --> Mean: 0.965, Std: 0.012 Sensitivity --> Mean: 0.958, Std: 0.024 Specificity --> Mean: 1.000, Std: 0.000 PPV --> Mean: 0.973, Std: 0.010 NPV --> Mean: 1.000, Std: 0.000 RelativeVolumeError --> Mean: -1.500, Std: 2.833 HausdorffDistance --> Mean: 1.559, Std: 0.565 ContourMeanDistance --> Mean: 0.325, Std: 0.089 SurfaceDistance --> Mean: 0.001, Std: 0.001 Test Phase Metrics [ANIMA] for lesion: Jaccard --> Mean: 0.220, Std: 0.186 Dice --> Mean: 0.322, Std: 0.256 Sensitivity --> Mean: 0.246, Std: 0.227 Specificity --> Mean: 1.000, Std: 0.000 PPV --> Mean: 0.783, Std: 0.340 NPV --> Mean: 1.000, Std: 0.000 RelativeVolumeError --> Mean: -67.966, Std: 31.173 HausdorffDistance --> Mean: 6.302, Std: 3.874 ContourMeanDistance --> Mean: 2.602, Std: 2.090 SurfaceDistance --> Mean: 0.567, Std: 0.903 PPVL --> Mean: 0.375, Std: 0.484 SensL --> Mean: 0.562, Std: 0.464 F1_score --> Mean: 0.375, Std: 0.484 ```The performance on dcm-zurich-lesions-20231115 does not seem to be good. Maybe it could be due to region-based training? A multi-channel model might improve the results
With SCIseg, we noticed that the model was also decently segmenting DCM lesions, which strongly suggested that it simply learned to be sensitive to T2w hyperintensities in the images. We could use this to our advantage for DCM lesion segmenation.
Instead of training from scratch on a small dataset (~50 subjects), we could pretrain on SCI data from all the 3 sites and then fine-tune on DCM lesions. My hypothesis is that pretraining should result in some boost in the model performance for DCM lesion seg.