GregorKoehler
commented
2 months ago Hi @ooooddddd, please provide some more context when opening issues, and don't just copy&paste error messages. In your case, I'd recommend checking your memory consumption, as it seems you are out of memory.
####################################################################### Please cite the following paper when using nnU-Net: Isensee, F., Jaeger, P. F., Kohl, S. A., Petersen, J., & Maier-Hein, K. H. (2021). nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods, 18(2), 203-211. #######################################################################
There are 9 cases in the source folder I am process 0 out of 1 (max process ID is 0, we start counting with 0!) There are 9 cases that I would like to predict
Predicting STS24_Validation_0011: perform_everything_on_gpu: True 100%|██████████| 320/320 [00:20<00:00, 15.46it/s] Prediction done, transferring to CPU if needed sending off prediction to background worker for resampling and export done with STS24_Validation_0011
Predicting STS24_Validation_0013: perform_everything_on_gpu: True 100%|██████████| 320/320 [00:19<00:00, 16.55it/s] Prediction done, transferring to CPU if needed sending off prediction to background worker for resampling and export done with STS24_Validation_0013
Predicting STS24_Validation_0014: perform_everything_on_gpu: True 100%|██████████| 384/384 [00:25<00:00, 15.26it/s] Prediction done, transferring to CPU if needed sending off prediction to background worker for resampling and export done with STS24_Validation_0014
Predicting STS24_Validation_0015: perform_everything_on_gpu: True 100%|██████████| 320/320 [00:08<00:00, 36.98it/s] Prediction done, transferring to CPU if needed sending off prediction to background worker for resampling and export done with STS24_Validation_0015
Predicting STS24_Validation_0016: perform_everything_on_gpu: True 100%|██████████| 320/320 [00:08<00:00, 37.09it/s] Prediction done, transferring to CPU if needed sending off prediction to background worker for resampling and export done with STS24_Validation_0016
Predicting STS24_Validation_0017: perform_everything_on_gpu: True 100%|██████████| 320/320 [00:08<00:00, 36.95it/s] resizing data, order is 3 data shape (1, 536, 440, 536) resizing segmentation, order is 1 order z is 0 data shape (1, 536, 440, 536) resizing data, order is 3 data shape (1, 640, 400, 640) resizing segmentation, order is 1 order z is 0 data shape (1, 640, 400, 640) resizing data, order is 3 data shape (1, 640, 400, 640) resizing segmentation, order is 1 order z is 0 data shape (1, 640, 400, 640) resizing data, order is 3 data shape (1, 640, 400, 640) resizing segmentation, order is 1 order z is 0 data shape (1, 640, 400, 640) resizing data, order is 3 data shape (1, 640, 400, 640) resizing segmentation, order is 1 order z is 0 data shape (1, 640, 400, 640) resizing data, order is 3 data shape (1, 640, 400, 640) resizing segmentation, order is 1 order z is 0 data shape (1, 640, 400, 640) Prediction done, transferring to CPU if needed sending off prediction to background worker for resampling and export done with STS24_Validation_0017
Predicting STS24_Validation_0018: perform_everything_on_gpu: True 100%|██████████| 320/320 [00:08<00:00, 37.09it/s] resizing data, order is 3 data shape (1, 640, 448, 640) resizing segmentation, order is 1 order z is 0 data shape (1, 640, 448, 640) resizing data, order is 3 data shape (1, 640, 400, 640) resizing segmentation, order is 1 order z is 0 data shape (1, 640, 400, 640) resizing data, order is 3 data shape (1, 640, 400, 640) resizing segmentation, order is 1 order z is 0 data shape (1, 640, 400, 640) Prediction done, transferring to CPU if needed sending off prediction to background worker for resampling and export done with STS24_Validation_0018
Predicting STS24_Validation_0019: perform_everything_on_gpu: True 100%|██████████| 320/320 [00:08<00:00, 37.10it/s] Prediction done, transferring to CPU if needed sending off prediction to background worker for resampling and export done with STS24_Validation_0019
Predicting STS24_Validation_0020: perform_everything_on_gpu: True 100%|██████████| 320/320 [00:08<00:00, 37.12it/s] Prediction done, transferring to CPU if needed sending off prediction to background worker for resampling and export done with STS24_Validation_0020 multiprocessing.pool.RemoteTraceback: """ Traceback (most recent call last): File "/home/z22zc/.conda/envs/umamba2/lib/python3.10/multiprocessing/pool.py", line 125, in worker result = (True, func(*args, kwds)) File "/home/z22zc/.conda/envs/umamba2/lib/python3.10/multiprocessing/pool.py", line 51, in starmapstar return list(itertools.starmap(args[0], args[1])) File "/home/z22zc/python/umamba/U-Mamba2/umamba/nnunetv2/inference/export_prediction.py", line 88, in export_prediction_from_logits ret = convert_predicted_logits_to_segmentation_with_correct_shape( File "/home/z22zc/python/umamba/U-Mamba2/umamba/nnunetv2/inference/export_prediction.py", line 30, in convert_predicted_logits_to_segmentation_with_correct_shape predicted_logits = configuration_manager.resampling_fn_probabilities(predicted_logits, File "/home/z22zc/python/umamba/U-Mamba2/umamba/nnunetv2/preprocessing/resampling/default_resampling.py", line 228, in resample_data_or_seg_to_shape data_reshaped = resample_data_or_seg(data, new_shape, is_seg, axis, order, do_separate_z, order_z=order_z) File "/home/z22zc/python/umamba/U-Mamba2/umamba/nnunetv2/preprocessing/resampling/default_resampling.py", line 327, in resample_data_or_seg reshaped.append(resize_fn(data[c], new_shape, order, kwargs)[None]) File "/home/z22zc/.conda/envs/umamba2/lib/python3.10/site-packages/skimage/transform/_warps.py", line 203, in resize out = ndi.zoom( File "/home/z22zc/.conda/envs/umamba2/lib/python3.10/site-packages/scipy/ndimage/_interpolation.py", line 823, in zoom output = _ni_support._get_output(output, input, shape=output_shape, File "/home/z22zc/.conda/envs/umamba2/lib/python3.10/site-packages/scipy/ndimage/_ni_support.py", line 79, in _get_output output = numpy.zeros(shape, dtype=input.dtype.name) numpy.core._exceptions._ArrayMemoryError: Unable to allocate 625. MiB for an array with shape (640, 400, 640) and data type float32 """
The above exception was the direct cause of the following exception:
Traceback (most recent call last): File "/home/z22zc/.conda/envs/umamba2/bin/nnUNetv2_predict", line 33, in
sys.exit(load_entry_point('nnunetv2', 'console_scripts', 'nnUNetv2_predict')())
File "/home/z22zc/python/umamba/U-Mamba2/umamba/nnunetv2/inference/predict_from_raw_data.py", line 838, in predict_entry_point
predictor.predict_from_files(args.i, args.o, save_probabilities=args.save_probabilities,
File "/home/z22zc/python/umamba/U-Mamba2/umamba/nnunetv2/inference/predict_from_raw_data.py", line 249, in predict_from_files
return self.predict_from_data_iterator(data_iterator, save_probabilities, num_processes_segmentation_export)
File "/home/z22zc/python/umamba/U-Mamba2/umamba/nnunetv2/inference/predict_from_raw_data.py", line 400, in predict_from_data_iterator
ret = [i.get()[0] for i in r]
File "/home/z22zc/python/umamba/U-Mamba2/umamba/nnunetv2/inference/predict_from_raw_data.py", line 400, in
ret = [i.get()[0] for i in r]
File "/home/z22zc/.conda/envs/umamba2/lib/python3.10/multiprocessing/pool.py", line 774, in get
raise self._value
numpy.core._exceptions._ArrayMemoryError: Unable to allocate 625. MiB for an array with shape (640, 400, 640) and data type float32
srun: error: dc1: task 0: Exited with exit code 1
export failed