search

MIC-DKFZ / nnUNet

Apache License 2.0
5.9k stars 1.76k forks source link

Questions about "This output is too large for python process-process communication" during the inference stage. #629

Closed Luoxd1996 closed 1 year ago

Luoxd1996 commented 3 years ago

Hi Fabian, I meet a problem when using nnUNet for inference, the log print this sentence "This output is too large for python process-process communication. Saving output temporarily to disk", but there are not any prediction output (some temporarily npy files were generated and deleted later). My task is about abdominal multi-organ segmentation, the average volume shape is 220 x 512 x 512. Could you give me some suggestions? Best, Xiangde.

FabianIsensee commented 3 years ago

This is just an information. Not a problem. No need to do anything. usually. Have you checked the RAM of your system? Is it getting full? Have you checked the entire text ourput for errors?

dorltcheng commented 2 years ago

Hi, I have the same issue as well. There is no prediction output saved in the output folder. How to solve the problem?

Zhangsir125 commented 1 year ago

INFO: Prediction is too large for python process-process communication. Saving to file... It's stuck here while you train the prediction part of the model

t21leluy commented 1 year ago

Hello, I seem to have the same issue. Has anyone managed to solve this issue? If so, how did you do it? Thank you in advance for your help!

Kobalt93 commented 1 year ago

Please update nnunet, the problem should no longer exist. Let us know if the issue still persists.

SuChenHong commented 7 months ago

INFO: Prediction is too large for python process-process communication. Saving to file... It's stuck here while you train the prediction part of the model

And now I also have this problem. Can someone help me? I always get stuck on the last predication, and then my validation file never has summary.json.

angleboy8 commented 6 months ago

INFO: Prediction is too large for python process-process communication. Saving to file... It's stuck here while you train the prediction part of the model

And now I also have this problem. Can someone help me? I always get stuck on the last predication, and then my validation file never has summary.json.

Did you resolve this problem? I discovered that two samples takes more than an hour to produce predicted results when the inputs are large. The GPU utilization is low. Hope who will give a better solution to it.

bghattas commented 6 months ago

I have actually the same problem; either none of the expected predictions is saved or only very few of them are. No specific error messages are given in the log files.

Luoxd1996 commented 6 months ago

Hi, I have fixed this issue by increasing the memory size or swapping memory (I tested the 24010241024, and it required more than 150G). Best.

Badih Ghattas @.***> 于2024年5月18日周六 16:39写道：

I have actually the same problem; either none of the expected predictions is saved or only very few of them are. No specific error messages are given in the log files.

— Reply to this email directly, view it on GitHub https://github.com/MIC-DKFZ/nnUNet/issues/629#issuecomment-2118703576, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEV7K52Q4MRNS373EDAZ3GTZC4HS7AVCNFSM44L7L572U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TEMJRHA3TAMZVG43A . You are receiving this because you authored the thread.Message ID: @.***>