FabianIsensee
commented
5 months ago Hey,
all models in the paper are just the 3d_fullres configuration, trained as 5-fold cross-validation. We should make that clearer in the next iteration of the paper!
The reported Dice scores are averaged over the file folds (custom script that reads the summary.json files produced during validation).
NSD is computed using the official deepmind repository on the validation set predictions from the cross-validation. Again, this is done with a custom script and is not part of the nnU-Net repo.
Computing the NSD using the repo linked above is quite simple. All you need to do is use compute_surface_dice_at_tolerance with the correct tolerance value (2mm in our paper). If you still want us to share the code we can do that - but it won't be valuable for you because it relies on our internal folder structure and adds no value to the deepmind codebase.
Best,
Fabian
Yanfeng-Zhou
Thanks for the excellent project nnUNet and the comparison of segmentation benchmark architectures in the new paper “nnU-Net Revisited: A Call for Rigorous Validation in 3D Medical Image Segmentation”!
But I still have some questions about the specific experimental setup in the new paper. Is the benchmark test on six data sets (BTCV, ACDC, LiTS, BraTS2021, KiTS2023 and AMOS22) conducted as follows: First train 2d, 3d_fullres, 3d_lowres, 3d_cascade_fullres models respectively, and then use nnUNetv2_find_best_configuration to print the DSC of 5-fold cross validation? But these four training methods will print a DSC.
By the way, can you open source the code for calculating Normalized Surface Distance (NSD) in nnUNet framework?
Looking forward to your reply! Thanks!