MIC-DKFZ / nnUNet

Apache License 2.0
5.79k stars 1.74k forks source link

Problems during training #1253

Closed fyb99 closed 9 months ago

fyb99 commented 1 year ago

Please read the following resources before posting issues:

Common questions: https://github.com/MIC-DKFZ/nnUNet/blob/master/documentation/common_questions.md

Common Problems and their solutions: https://github.com/MIC-DKFZ/nnUNet/blob/master/documentation/common_problems_and_solutions.md

Expected epoch times and tips on how to identify bottlenecks: https://github.com/MIC-DKFZ/nnUNet/blob/master/documentation/expected_epoch_times.md

fyb99 commented 1 year ago

Dear Fabian Isensee, I have met two problems during the data preprocessing and training phases when using nnUNet framework.

  1. When prepocessing the data, I have met the problem "OverflowError: cannot convert float infinity to integer". After debugging, I found that in the "get_properties_for_stage'' function located in experiment_planner_baseline_3DUNet_v21.py, the first dimension of the output "new_median_shape" incorrectly changes to 0 of some my private cases, which caused subsequent batchsize define error. How can I solve the problem?
  2. As described above, constraints are added to my code to make 0 changed to 1 in the first dimension. When training, I met the problem "case does not contain any foreground classes" and found the shape of my .npz file in the folder "nnUNetData_plans_v2.1_stage0" is (4, 0, H, W). How can I solve the problem for correct training?
provbs commented 1 year ago

Hi, I was facing the same issue for days and found out what was going wrong.

for my case, the data was 2D (1, x, y) and the preprocessing made the "cannot convert float infinity to integer" error.

I read the article again, and found out that 2D data preprocessing should be little different

Link - https://github.com/MIC-DKFZ/nnUNet/blob/master/nnunet/dataset_conversion/Task120_Massachusetts_RoadSegm.py

in the below, you can see how to handle 2D (1, x, y) data with the preprocessing

once this is completed, you can use the dataset like any other nnU-Net dataset. Note that since this is a 2D
dataset there is no need to run preprocessing for 3D U-Nets. You should therefore run the 
`nnUNet_plan_and_preprocess` command like this:

> nnUNet_plan_and_preprocess -t 120 -pl3d None

not sure if this is your case, but hope it helps even a little

FabianIsensee commented 1 year ago

@fyb99 do you still face issues when using nnU-Net with 2D data?

FabianIsensee commented 1 year ago

thanks @provbs !