Open YUjh0729 opened 3 months ago
To add, when predicting multiple datasets simultaneously, smaller datasets (FLARE22_013, FLARE22_014) are able to output results normally. However, larger files (FLARE22_010) cause the program to fail to produce output and do not terminate properly.
Hi, that happens when you resample the output probabilities back to the image resolution (especially when you have many classes). Your system does not have enough RAM.
Is there any way to prevent nnunet from sending export to background?
Setting the following parameters fixed the problem for me:
-nps 1 -npp 1
Hi, that happens when you resample the output probabilities back to the image resolution (especially when you have many classes). Your system does not have enough RAM.
Hi, @constantinulrich , Yes, I guess the root cause is insufficient RAM. I've tried all the solutions from the issues, but none of them resolved the issue.Do I need to upgrade my current device to solve this problem?
I encountered a very strange issue when using the nnUNetv2_predict command. The program can't proceed and is unable to output the prediction results. These are the results I predicted on the cloud server, `Predicting FLARE22_010: perform_everything_on_device: True 0%| | 0/360 [00:00<?, ?it/s]resizing data, order is 3 data shape (1, 227, 512, 512) 11%|████████████████████▍ | 38/360 [00:05<00:48, 6.65it/s]resizing segmentation, order is 1 order z is 0 data shape (1, 227, 512, 512) 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 360/360 [00:54<00:00, 6.60it/s] sending off prediction to background worker for resampling and export done with FLARE22_010
Predicting FLARE22_011: perform_everything_on_device: True 38%|██████████████████████████████████████████████████████████████████████████▊ | 23/60 [00:03<00:05, 6.61it/s]resizing data, order is 1 data shape (14, 250, 628, 628) 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 60/60 [00:08<00:00, 6.74it/s] sending off prediction to background worker for resampling and export done with FLARE22_011 resizing data, order is 1 data shape (14, 109, 430, 430)and these are the results I tested locally,The output is similar, but there are these two additional lines of output. Both environments are identical:torch2.0.1,cudu11.8,python3.10perform_everything_on_device: True Prediction on device was unsuccessful, probably due to a lack of memory. Moving results arrays to CPU`
The CPU and GPU are no longer occupied, and the prediction results should already be in the CPU memory, but they cannot be exported to the output folder, causing the program to freeze and become stuck. This only happens in a few cases with larger data in Abdomen CT_3D. How do you predict larger data in Abdomen CT_3D?
Originally posted by @YUjh0729 in https://github.com/MIC-DKFZ/nnUNet/issues/2091#issuecomment-2273776717