MIC-DKFZ / nnUNet

Apache License 2.0
5.79k stars 1.74k forks source link

Background workers died #2392

Closed sami22ivr closed 2 months ago

sami22ivr commented 3 months ago

hi everyone, so im having an issue with doing prediction where it always shows this error : ####################################################################### Please cite the following paper when using nnU-Net: Isensee, F., Jaeger, P. F., Kohl, S. A., Petersen, J., & Maier-Hein, K. H. (2021). nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods, 18(2), 203-211. #######################################################################

There are 1 cases in the source folder I am process 0 out of 1 (max process ID is 0, we start counting with 0!) There are 1 cases that I would like to predict Process SpawnProcess-6: Traceback (most recent call last): File "/home/samir/miniconda3/envs/nnUNet/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap self.run() File "/home/samir/miniconda3/envs/nnUNet/lib/python3.11/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/home/samir/miniconda3/envs/nnUNet/lib/python3.11/site-packages/nnunetv2/inference/data_iterators.py", line 58, in preprocess_fromfiles_save_to_queue raise e File "/home/samir/miniconda3/envs/nnUNet/lib/python3.11/site-packages/nnunetv2/inference/data_iterators.py", line 31, in preprocess_fromfiles_save_to_queue data, seg, data_properties = preprocessor.run_case(list_of_lists[idx], ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/samir/miniconda3/envs/nnUNet/lib/python3.11/site-packages/nnunetv2/preprocessing/preprocessors/default_preprocessor.py", line 139, in run_case data, seg = self.run_case_npy(data, seg, data_properties, plans_manager, configuration_manager, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/samir/miniconda3/envs/nnUNet/lib/python3.11/site-packages/nnunetv2/preprocessing/preprocessors/default_preprocessor.py", line 78, in run_case_npy data = self._normalize(data, seg, configuration_manager, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/samir/miniconda3/envs/nnUNet/lib/python3.11/site-packages/nnunetv2/preprocessing/preprocessors/default_preprocessor.py", line 183, in _normalize scheme = configuration_manager.normalization_schemes[c]


IndexError: list index out of range
Traceback (most recent call last):
  File "/home/samir/miniconda3/envs/nnUNet/bin/nnUNetv2_predict", line 8, in <module>
    sys.exit(predict_entry_point())
             ^^^^^^^^^^^^^^^^^^^^^
  File "/home/samir/miniconda3/envs/nnUNet/lib/python3.11/site-packages/nnunetv2/inference/predict_from_raw_data.py", line 864, in predict_entry_point
    predictor.predict_from_files(args.i, args.o, save_probabilities=args.save_probabilities,
  File "/home/samir/miniconda3/envs/nnUNet/lib/python3.11/site-packages/nnunetv2/inference/predict_from_raw_data.py", line 256, in predict_from_files
    return self.predict_from_data_iterator(data_iterator, save_probabilities, num_processes_segmentation_export)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/samir/miniconda3/envs/nnUNet/lib/python3.11/site-packages/nnunetv2/inference/predict_from_raw_data.py", line 349, in predict_from_data_iterator
    for preprocessed in data_iterator:
  File "/home/samir/miniconda3/envs/nnUNet/lib/python3.11/site-packages/nnunetv2/inference/data_iterators.py", line 111, in preprocessing_iterator_fromfiles
    raise RuntimeError('Background workers died. Look for the error message further up! If there is '
RuntimeError: Background workers died. Look for the error message further up! If there is none then your RAM was full and the worker was killed by the OS. Use fewer workers or get more RAM in that case! 
at one point i tried to do prediction with --npp 1 and --nps 1 on one image it did work then tried with the default --nps & --npp it worked also but then i wanted to try on 20 images it didnt work then on 15 , 10, 5, and even 1 image this time it didnt work and its showing the same error so please help! 
im using  NVIDIA GeForce RTX 4080 with RAM 32 Go. 
i actually trained nnUnet before with different data and i didnt face any of this working on the same PC.!!
sten2lu commented 3 months ago

Hi @sami22ivr, it seems as if you are running out of RAM. A clear indicator of this is that the inference did run with a reduced number of processes. Sadly 32GB of RAM is not necessarily a lot of memory for many volumetric datasets.

So you have three options now:

  1. Try to increase the swap of your system.
  2. Keep on using a reduced number of processes for pre and postprocessing npp and nps
  3. Buy more RAM.

Sorry for not being able to help you more.

Best regards,

Carsten