Inconsistencies between raters on lesion segmentation

ivadomed / model_seg_ms_mp2rage

Model repository for MS lesion segmentation on MP2RAGE data from University of Basel

MIT License

2 stars 0 forks source link

Inconsistencies between raters on lesion segmentation #13

Closed jcohenadad closed 1 year ago

jcohenadad commented 2 years ago

In this image (from P010):

One rater (red) segmented lesions above the medulla oblongata, while the other (blue) did not. This creates inconsistencies when computing inter-rater variability, and when evaluating the performance of the trained model.

A suggestion for now would be to ignore lesions that are not in the spinal cord.

uzaymacar commented 2 years ago

Check #12 for more information on the subjects.

jcohenadad commented 2 years ago

Further investigations by @uzaymacar shows an inter-rater average Dice score of 0.581 +/- 0.212 (+/- corresponds to the STD of the Dice across subjects). This calculation was done on this version of the dataset: 04b78bb3619b22e2e560b3211807a0b1e54a70cb

jcohenadad commented 2 years ago

It might be wise to consider manually revising the raters segmentation to minimize the discrepancy (which hurts the model capabilities).

uzaymacar commented 2 years ago

To illustrate and emphasize the low agreement / large variability among raters for lesion annotations in the spinal cord as reported in the previous comment, we can look at sub-P013:

low_agreement_for_sc_lesions

uzaymacar commented 2 years ago

As of today, we started working on generating manually corrected lesion segmentations (not to be confused with #14 in which we performed the same task on SC segmentations instead). The manually corrected lesion segmentations will be

placed in the git-annexed dataset,
will include the lesion-manual3 suffix, and
we started from the first rater (i.e., lesion-manual suffix) but ideally we will start from the majority vote.

jcohenadad commented 2 years ago

I'll re-do the seg from sub 1-17 and @uzaymacar will do the rest.

strategy:

getting the majority vote from this folder: ~/duke/temp/uzay/basel-mp2rage-preprocessed/data_processed/
use the NON CROPPED image, eg: sub-P001_UNIT1_lesion-manual-majvote.nii.gz
to aid the manual correction, we should apply slight denoising on the image, see https://github.com/ivadomed/model_seg_ms_mp2rage/issues/46#issuecomment-1171604401
create: sub-P001_UNIT1_lesion-manual3.nii.gz
copy to git-annex

jcohenadad commented 2 years ago

@mchen1110 is working on it

jcohenadad commented 1 year ago

Completed in #56