search

MIC-DKFZ / nnDetection

nnDetection is a self-configuring framework for 3D (volumetric) medical object detection which can be applied to new data sets without manual intervention. It includes guides for 12 data sets that were used to develop and evaluate the performance of the proposed method.
Apache License 2.0
543 stars 94 forks source link

[Question] Is tta needed for inference #50

Closed fgbxx closed 2 years ago

fgbxx commented 2 years ago

:question: Question

hi,
I have tried the module and train model, Was it necessary for tta transform? For the time consuming.

mibaumgartner commented 2 years ago

Hi,

TTA is not strictly necessary but usually gives a small performance boost. If inference time is a critical parameter that needs to be optimized in your use case, TTA can be disabled during nndet_predict by using the -ntta 0 flag.

Best, Michael

fgbxx commented 2 years ago

Hi, Michael, Thanks for your feedback! I have tried nnUnet and nnDetection for the lesion detection task. for nnunet default(argmax) has the ~0.8 sensitivity, IoU0.1,FP/case 0.14 . when I change confidence threshold(0.5 to 0.1), sensitivity to 0.88 @FP1 IoU0.1 But for nnDetection, FP1.6 get the 0.88 sensitivity IoU0.1. Were the results resonalble if the dataset is same? Another question is how IoU is selected as usual for medical detection task?

mibaumgartner commented 2 years ago

Hi,

performance numbers are highly dependent on the dataset and lesion detection is extremely broad. The general range of the numbers seems to be reasonable at first sight.

Reporting individual working points makes it quite hard to compare two methods since FP/case and Sensitivity can vary. IMO, curve based metrics (AP, FROC) are much more comprehensive and give a better overview of the performance across several working points.

The IoU value is empirically optimized on the validation set when nndet_consolidate --sweep_boxes is executed. nnUNetPlus (from our paper) uses a similar empirical optimization to find good postprocessing parameters.

Best, Michael