search

MIC-DKFZ / nnDetection

nnDetection is a self-configuring framework for 3D (volumetric) medical object detection which can be applied to new data sets without manual intervention. It includes guides for 12 data sets that were used to develop and evaluate the performance of the proposed method.
Apache License 2.0
543 stars 94 forks source link

[Question]Replicate result on paper #37

Closed vankhoa21991 closed 2 years ago

vankhoa21991 commented 2 years ago

Hello,

I tried to replicate the results of nndetection on LUNA16 as in your paper, but I can not achieve the same performance marked in the paper (CPM=0.92). In fact what I obtain is CPM=0.84. Did you use different set of parameter as presented in the paper? Or did you fine tuning the model?

Thanks,

mibaumgartner commented 2 years ago

Hi @vankhoa21991 ,

we didn't change the parameters and did not fine-tune the model in any way.

Did you run all 10 Folds? (the provided prepare script takes care of generating the split but the 10 folds need to be run)

For evaluation: Did you use the official LUNA script for evaluation? nndet_eval doesn't give you the correct CPM score here. LUNA defines additional "ignore" locations that are neither counted as True Positives nor False Positives but are neglected in nndet_eval. Furthermore nndet_eval evaluates based on the boxes and not point locations (FROC at IoU 0.1 computed by nndet was 0.829).

Best, Michael

vankhoa21991 commented 2 years ago

Yes indeed those "ignore" findings are important as lots of FP fall in there, in fact the non-nodule is 36x more than nodules.

Thanks for you reply, I was able to made a nice FROC curve :).

DongChen06 commented 2 years ago

@vankhoa21991 have you trained for 10-fold validation, in my understanding, it may take 2 days x 10 = 20 days to finish the training?

vankhoa21991 commented 2 years ago

Indeed, almost 2 days on 1 GPU, if you have several GPUs you can make it faster

BelieferQAQ commented 2 years ago

Hello,

I tried to replicate the results of nndetection on LUNA16 as in your paper, but I can not achieve the same performance marked in the paper (CPM=0.92). In fact what I obtain is CPM=0.84. Did you use different set of parameter as presented in the paper? Or did you fine tuning the model?

Thanks,

Hello, what is the confidence threshold (probability threshold) of the selected candidate frame when calculating FROC and drawing FROC curve?

mibaumgartner commented 2 years ago

nnDetection does not save the corresponding confidence threshold since the FROCis solely defined on the basis of FP per image

BelieferQAQ commented 2 years ago

You mean that all candidate frames predicted by the network are directly involved in the calculation of TP and FP (draw the FROC curve). Did not pass the first filtering of the confidence threshold