nnUNet_find_best_configuration

[maryamag85]

It seems that this command needs argumanet like -t 004 and when I put the argument I get the following message although I have finished all the prior steps and I have downloaded the trained model. in nnUNet_trained_models/nnUNet I have 3 folders

• -2d
• -3d_fullres
• -ensembles

and inside each of them a folder with the name of Task00x_Task and inside that foldet threre is a model it seems running missing postprocessing for Task004_Hippocampus and model 2d\

WARNING! [Errno 2] No such file or directory: '/home/sasha/nnUNet_trained_models/nnUNet/2d/Task004_Hippocampus/nnUNetTrainerV2nnUNetPlansv2.1/fold_0/validation_raw' WARNING! Output folder for model 3d_lowres is missing, expected: /home/sasha/nnUNet_trained_models/nnUNet/3d_lowres/Task004_Hippocampus/nnUNetTrainerV2nnUNetPlansv2.1 running missing postprocessing for Task004_Hippocampus and model 3d_fullres WARNING! [Errno 2] No such file or directory: '/home/sasha/nnUNet_trained_models/nnUNet/3d_fullres/Task004_Hippocampus/nnUNetTrainerV2nnUNetPlansv2.1/fold_0/validation_raw' WARNING! Output folder for model 3d_cascade_fullres is missing, expected: /home/sasha/nnUNet_trained_models/nnUNet/3d_cascade_fullres/Task004_Hippocampus/nnUNetTrainerV2CascadeFullResnnUNetPlansv2.1

Found the following valid models: [] Traceback (most recent call last): File "/home/sasha/.local/bin/nnUNet_find_best_configuration", line 33, in sys.exit(load_entry_point('nnunet', 'console_scripts', 'nnUNet_find_best_configuration')()) File "/home/sasha/nnUNet/nnunet/evaluation/model_selection/figure_out_what_to_submit.py", line 154, in main best = np.max(foreground_dices) File "<__array_function__ internals>", line 6, in amax File "/home/sasha/.local/lib/python3.6/site-packages/numpy/core/fromnumeric.py", line 2706, in amax keepdims=keepdims, initial=initial, where=where) File "/home/sasha/.local/lib/python3.6/site-packages/numpy/core/fromnumeric.py", line 87, in _wrapreduction return ufunc.reduce(obj, axis, dtype, out, **passkwargs) ValueError: zero-size array to reduction operation maximum which has no identity

[Joevaen]

Is this an issue when you are running an inference? Can you show me your whole command line?

[maryamag85]

I have downloaded the pretrained models witht the provided command and now using the following command and you see the result afterwards: it seems that validation_raw is not in the downloaded model folders and it could not go through.

============================= sasha@Sasha:~/nnUNet_trained_models/nnUNet/2d/Task004_Hippocampus/nnUNetTrainerV2__nnUNetPlansv2.1/fold_0$ nnUNet_find_best_configuration -m 2d 3d_fullres 3d_lowres -t 004 --strict

Please cite the following paper when using nnUNet: Fabian Isensee, Paul F. Jäger, Simon A. A. Kohl, Jens Petersen, Klaus H. Maier-Hein "Automated Design of Deep Learning Methods for Biomedical Image Segmentation" arXiv preprint arXiv:1904.08128 (2020). If you have questions or suggestions, feel free to open an issue at https://github.com/MIC-DKFZ/nnUNet

running missing postprocessing for Task004_Hippocampus and model 2d Traceback (most recent call last): File "/home/sasha/.local/bin/nnUNet_find_best_configuration", line 33, in sys.exit(load_entry_point('nnunet', 'console_scripts', 'nnUNet_find_best_configuration')()) File "/home/sasha/nnUNet/nnunet/evaluation/model_selection/figure_out_what_to_submit.py", line 119, in main raise e File "/home/sasha/nnUNet/nnunet/evaluation/model_selection/figure_out_what_to_submit.py", line 106, in main consolidate_folds(output_folder) File "/home/sasha/nnUNet/nnunet/postprocessing/consolidate_postprocessing.py", line 63, in consolidate_folds num_niftis += len(subfiles(v, suffix=".nii.gz")) File "/home/sasha/.local/lib/python3.6/site-packages/batchgenerators/utilities/file_and_folder_operations.py", line 24, in subfiles res = [l(folder, i) for i in os.listdir(folder) if os.path.isfile(os.path.join(folder, i)) FileNotFoundError: [Errno 2] No such file or directory: '/home/sasha/nnUNet_trained_models/nnUNet/2d/Task004_Hippocampus/nnUNetTrainerV2__nnUNetPlansv2.1/fold_0/validation_raw'

[Joevaen]

I am also suffering from this issue, no postprogress.pkl and validation_raw would be generated during my inference. Maybe we need other's help, really. Sorry about no help to you.

[Joevaen]

Hi, issue solved? I found these 2 folders we need is autometically generated when 1000 epoch run over. Maybe you can change this in code.

[FabianIsensee]

Hi, I have resolved an issue related to this today, please pull from the current master and try again. The issue was that a folder was not being created and then when nnU-Net tried to write to it you get the error mentioned above. Let me know if it works now. Best, Fabian

[FabianIsensee]

is it working now?

[annukkaa]

Hi, I stopped my training before epoch 1000 and these files were not generated. I'm also getting the same FileNotFoundError error when running:

nnUNet_find_best_configuration -m 3d_fullres -t 102 --strict

I have trained 5 folds of full_res. What can I do to obtain these files for my alredy trained folds?

[FabianIsensee]

Hi, you need to contonue the training (-c) and let it finish first. Otherwise this will not work. Best, Fabian

[annukkaa]

If I may I ask another question: Do I always need to run nnUNet_find_best_configuration?

My case is binary (tumour or no tumour) and so far I've trained 5 folds of 3d_fullres (each up until epoch +-500).

Because I didn't let it run til epoch 1000, I don't have post-processing files yet. May I still run nnUNet_predict without having finished all 1000 epochs and not having these files? From what I understand these are necessary to compare scores with and without connected component analysis, which I'm not sure is needed in my case.

In other words, is there any step I should do before being able to run nnUNet_predict, or can I do it straight after training (and without nnUNet_find_best_configuration)?

I'm asking this because my Average global foreground Dice always reaches a minimum of 0.6-0.65 in all 5 folds during training (see example of fold 0 in the image below). Yet, when I test the model on a raw independent test set using all 5 folds, average dice falls down to 0.23 (I'm using evaluate_folder to compute this dice). Because I didn't observe any obvious overfitting during training and I'm using the "best" model configuration of all folds, I would expect it to have similar performance as the observed in the validation set, but not significantly worse. So I'm wondering if I'm missing any step here before I can predict. For example, should I apply any pre-processing to my test set or should I let the model handle it?

Thank you in advance

[Joevaen]

Hi, an EXAMPLE: nnUNet_predict -i INPUT-DIRECTORY -o OUTPUT-DIRECTORY -t 8 -m 3d_fullres -f 4 -chk model_bestORnnUNet_predict -i INPUT-DIRECTORY -o OUTPUT-DIRECTORY -t 8 -m 3d_fullres -f 4 -chk model_lastest

[FabianIsensee]

Hi, please note this statement from the documentation:

Also shows an approximation of the evlauation metric (green). This approximation is the average Dice score of the foreground classes. It should, however, only to be taken with a grain of salt because it is computed on randomly drawn patches from the validation data at the end of each epoch, and the aggregation of TP, FP and FN for the Dice computation treats the patches as if they all originate from the same volume ('global Dice'; we do not compute a Dice for each validation case and then average over all cases but pretend that there is only one validation case from which we sample patches). The reason for this is that the 'global Dice' is easy to compute during training and is still quite useful to evaluate whether a model is training at all or not. A proper validation is run at the end of the training.

Average global foreground Dice is plotted as the green line. This is NOT representative of your actual Dice score at the end. For some datasets it is going to be close. For other it is not. If you have many images where no tumor is present then this drop can be expected.

You should always let it train all the way. After training there will be a final checkpoint and a best checkpoint. If you observe overfitting (if the green line starts falling again) you can choose to use the best checkpoint instead of final (final is the default).

You do not need to run nnUNet_find_best_configuration. You can do whatever you want. Unless you are using nnU-Net as a baseline in which case you should run it as intended.

@Joevaen has provided examples for how to use the different checkpoints (thanks!)

Best, Fabian

[annukkaa]

Thank you for pointing to the documentation. I overlooked that part. I will let folds run till epoch 1000 and hopefully get a better dice at the end.

Could you explain what you mean by

If you have many images where no tumor is present then this drop can be expected.

?

My images do suffer from class imbalance (in each image, tumour regions are significantly smaller than no tumour regions), but all of them contain tumour.

[FabianIsensee]

Hi, what I mean by this is that images that do not contain tumor in the reference annotation but have even a single false positive prediction will get 0 Dice score, which lowers the mean by a lot. But this does not seem to be the case for your dataset. Best, Fabian