mibaumgartner
commented
3 years ago Hi @MengzhangLI ,
thanks for your interest in nnDetection and the nice words 👍
(1)
The mdt (medical detection toolkit) compared multiple detectors and proposed Retina U-Net which is an extension of RetinaNet. It was designed as a toolkit to be used by researchers and thus requires manual hyperparameter tuning and custom dataloaders for each new dataset.
nnDetection follows nnU-Net and contains a whole self-configuring pipeline that can be applied to arbitrary new data set without manual intervention (no hyperparameter tuning required) but focuses on one base architecture (Retina U-Net). More information can be found in our publication nnDetection: A Self-configuring Method for Medical Object Detection which can be found online very soon. Future versions will potentially add more architectures and make other models (such as Mask R-CNN) available. Since those approaches need to be tested thoroughly on all datasets to ensure generalizability and performance, there is no ETA yet.
(2) The provided Luna results were intended to reflect the literature landscape upon submission of nnDetection (March 2021). Unfortunately, it will be out of scope of this repository to reproduce the results of all previous methods. All of the included results should reflect a 10 fold cross-validation. We hope to include more benchmarks with public leaderboards and closed test sets in the future (which should be helpful to compare results).
We hope that nnDetection will foster more standardized evaluation in the medical object detection community by providing unified entrypoints to a wide number of data sets and a strong out-of-box baseline method for future research.
Hope this answer helps :)
Best, Michael
MengzhangLI
:question: Question
Hi, Michael: Thanks for sharing your nnDetection codebase. It seems very great and has potential for medical detection task. I am familiar with medicaldetectiontoolkit, so I have two questions and hoping for your reply.
(1) What's main difference between medicaldetectiontoolkit and nnDetection? As far as I know, when upgrading pytorch version from 0.4.1 to 1.0+, medicaldetectiontoolkit has gained great traning and inference performance. But its owner pfjaeger seems not make it as default branch.
(2) Detailed results on different dataset should be checked carefully. Instead of most citing paper directly, I think we would better checking its real performance. For example, whether it is 10-fold cross-validation, whether it used lung mask for pre/post processing, whether it is 5 fold model, etc.
Most so called medical image SOTA paper seldom open-sourced its official code. That makes us can not reproduce and testify its real numerical results. By the way, that's why many researchers include me have much respect on Fabian and you guys in DKFZ.
Just as you read the former issue, it claims its results on LUNA16 is 0.125（72.3）0.25（83.8） 0.5（88.7） 1（91.1）2（92.8）4（93.4）8（94.8）average 88.1.
Here is my re-implementation results on LUNA16 with medicaldetectiontoolkit. (1) Dataset Traning and validation set: subset 0-7. Test set: subset 8,9. (2) Results (1) 5 Model + 2Dc MaskRCNN (mAP10: 0.6894) 0.125（0.570）0.25（0.659） 0.5（0.776） 1（0.879）2（0.910）4（0.932）8（0.973）average 0.814. (2) 1 Model + NoTTA + 2Dc MaskRCNN (mAP10: 0.755) 0.125（0.404）0.25（0.605） 0.5（0.709） 1（0.834）2（0.883）4（0.0.915）8（0.937）average 0.755. (3) 1 Model + NoTTA + 2Dc FasterRCNN (mAP10: 0.5943) 0.125（0.501）0.25（0.632） 0.5（0.767） 1（0.830）2（0.883）4（0.0.915）8（0.937）average 0.782.
There are part of my results on LUNA16 and it is close with medicaldetectiontoolkit author results on LIDC-IDRI. Table 1 from "Retina U-Net: Embarrassingly Simple Exploitation ofSegmentation Supervision for Medical Object Detection" https://ml4health.github.io/2019/pdf/232_ml4h_preprint.pdf
Hoping to contact with you in the future.
Best,