avBuffer
commented
6 months ago Me 2. I have the same issue.
Closed CYYJL closed 4 months ago
avBuffer
commented
6 months ago Me 2. I have the same issue.
sten2lu
commented
5 months ago Hi @CYYJL and @avBuffer,
The error message suggests reducing the number of workers as you are running out of RAM.
You can do this by running with the flag np. For example, you could try:
nnUNetv2_plan_and_preprocess -d 6 --verify_dataset_integrity -np 2Best regards
CYYJL
commented
5 months ago Hi, thank you very much for your advice. I have solved this problem according to your method. Thank you very much.
Best!!! CYYJL
Hello, when I was preprocessing the MSD challenge task 006 dataset ( Target: Lung and cancer segmentation), I searched for this error on Google and tried to fix it, but the error still persists. Do you have any suggestion? Thank you CYYJL Below is the detailed error message： ` (nn_Unetv2) yjl@DESKTOP-257J6TD:~$ nnUNetv2_plan_and_preprocess -d 6 --verify_dataset_integrity Fingerprint extraction... Dataset006_Lung Using <class 'nnunetv2.imageio.simpleitk_reader_writer.SimpleITKIO'> as reader/writer
#################### verify_dataset_integrity Done. If you didn't see any error messages then your dataset is most likely OK! ####################
Using <class 'nnunetv2.imageio.simpleitk_reader_writer.SimpleITKIO'> as reader/writer 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 63/63 [00:30<00:00, 2.05it/s] Experiment planning... Attempting to find 3d_lowres config. Current spacing: [1.28232938 0.80871094 0.80871094]. Current patch size: (80, 192, 160). Current median shape: [244.66019417 497.08737864 497.08737864] Attempting to find 3d_lowres config. Current spacing: [1.32079926 0.83297227 0.83297227]. Current patch size: (80, 192, 160). Current median shape: [237.5341691 482.60910548 482.60910548] Attempting to find 3d_lowres config. Current spacing: [1.36042324 0.85796143 0.85796143]. Current patch size: (80, 192, 160). Current median shape: [230.61569816 468.55252959 468.55252959] Attempting to find 3d_lowres config. Current spacing: [1.40123593 0.88370028 0.88370028]. Current patch size: (80, 192, 160). Current median shape: [223.89873607 454.90536853 454.90536853] Attempting to find 3d_lowres config. Current spacing: [1.44327301 0.91021128 0.91021128]. Current patch size: (80, 192, 160). Current median shape: [217.37741366 441.6556976 441.6556976 ] Attempting to find 3d_lowres config. Current spacing: [1.4865712 0.93751762 0.93751762]. Current patch size: (80, 192, 160). Current median shape: [211.04603268 428.79193942 428.79193942] Attempting to find 3d_lowres config. Current spacing: [1.53116834 0.96564315 0.96564315]. Current patch size: (80, 192, 160). Current median shape: [204.89906086 416.30285381 416.30285381] Attempting to find 3d_lowres config. Current spacing: [1.57710339 0.99461245 0.99461245]. Current patch size: (80, 192, 160). Current median shape: [198.93112705 404.17752797 404.17752797] Attempting to find 3d_lowres config. Current spacing: [1.62441649 1.02445082 1.02445082]. Current patch size: (80, 192, 160). Current median shape: [193.13701655 392.40536696 392.40536696] Attempting to find 3d_lowres config. Current spacing: [1.67314898 1.05518434 1.05518434]. Current patch size: (80, 192, 160). Current median shape: [187.51166655 380.97608443 380.97608443] Attempting to find 3d_lowres config. Current spacing: [1.72334345 1.08683988 1.08683988]. Current patch size: (80, 192, 160). Current median shape: [182.0501617 369.87969362 369.87969362] Attempting to find 3d_lowres config. Current spacing: [1.77504376 1.11944507 1.11944507]. Current patch size: (80, 192, 160). Current median shape: [176.74772981 359.10649866 359.10649866] Attempting to find 3d_lowres config. Current spacing: [1.82829507 1.15302842 1.15302842]. Current patch size: (80, 192, 160). Current median shape: [171.59973768 348.64708608 348.64708608] Attempting to find 3d_lowres config. Current spacing: [1.88314392 1.18761928 1.18761928]. Current patch size: (80, 192, 160). Current median shape: [166.60168707 338.49231658 338.49231658] Attempting to find 3d_lowres config. Current spacing: [1.93963824 1.22324785 1.22324785]. Current patch size: (80, 192, 160). Current median shape: [161.74921074 328.63331707 328.63331707] Attempting to find 3d_lowres config. Current spacing: [1.99782739 1.25994529 1.25994529]. Current patch size: (80, 192, 160). Current median shape: [157.03806868 319.06147288 319.06147288] Attempting to find 3d_lowres config. Current spacing: [2.05776221 1.29774365 1.29774365]. Current patch size: (80, 192, 160). Current median shape: [152.46414435 309.76842027 309.76842027] Attempting to find 3d_lowres config. Current spacing: [2.11949507 1.33667596 1.33667596]. Current patch size: (80, 192, 160). Current median shape: [148.02344112 300.7460391 300.7460391 ] Attempting to find 3d_lowres config. Current spacing: [2.18307993 1.37677624 1.37677624]. Current patch size: (80, 192, 160). Current median shape: [143.71207876 291.98644573 291.98644573] Attempting to find 3d_lowres config. Current spacing: [2.24857232 1.41807952 1.41807952]. Current patch size: (80, 192, 160). Current median shape: [139.52629005 283.48198614 283.48198614] Attempting to find 3d_lowres config. Current spacing: [2.31602949 1.46062191 1.46062191]. Current patch size: (80, 192, 160). Current median shape: [135.46241753 275.22522927 275.22522927] Attempting to find 3d_lowres config. Current spacing: [2.38551038 1.50444057 1.50444057]. Current patch size: (80, 192, 160). Current median shape: [131.51691022 267.20896045 267.20896045] 2D U-Net configuration: {'data_identifier': 'nnUNetPlans_2d', 'preprocessor_name': 'DefaultPreprocessor', 'batch_size': 12, 'patch_size': (512, 512), 'median_image_size_in_voxels': array([512., 512.]), 'spacing': array([0.78515625, 0.78515625]), 'normalization_schemes': ['CTNormalization'], 'use_mask_for_norm': [False], 'resampling_fn_data': 'resample_data_or_seg_to_shape', 'resampling_fn_seg': 'resample_data_or_seg_to_shape', 'resampling_fn_data_kwargs': {'is_seg': False, 'order': 3, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_seg_kwargs': {'is_seg': True, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_probabilities': 'resample_data_or_seg_to_shape', 'resampling_fn_probabilities_kwargs': {'is_seg': False, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'architecture': {'network_class_name': 'dynamic_network_architectures.architectures.unet.PlainConvUNet', 'arch_kwargs': {'n_stages': 8, 'features_per_stage': (32, 64, 128, 256, 512, 512, 512, 512), 'conv_op': 'torch.nn.modules.conv.Conv2d', 'kernel_sizes': ((3, 3), (3, 3), (3, 3), (3, 3), (3, 3), (3, 3), (3, 3), (3, 3)), 'strides': ((1, 1), (2, 2), (2, 2), (2, 2), (2, 2), (2, 2), (2, 2), (2, 2)), 'n_conv_per_stage': (2, 2, 2, 2, 2, 2, 2, 2), 'n_conv_per_stage_decoder': (2, 2, 2, 2, 2, 2, 2), 'conv_bias': True, 'norm_op': 'torch.nn.modules.instancenorm.InstanceNorm2d', 'norm_op_kwargs': {'eps': 1e-05, 'affine': True}, 'dropout_op': None, 'dropout_op_kwargs': None, 'nonlin': 'torch.nn.LeakyReLU', 'nonlin_kwargs': {'inplace': True}}, '_kw_requires_import': ('conv_op', 'norm_op', 'dropout_op', 'nonlin')}, 'batch_dice': True}
Using <class 'nnunetv2.imageio.simpleitk_reader_writer.SimpleITKIO'> as reader/writer 3D lowres U-Net configuration: {'data_identifier': 'nnUNetPlans_3d_lowres', 'preprocessor_name': 'DefaultPreprocessor', 'batch_size': 2, 'patch_size': (80, 192, 160), 'median_image_size_in_voxels': (132, 267, 267), 'spacing': array([2.38551038, 1.50444057, 1.50444057]), 'normalization_schemes': ['CTNormalization'], 'use_mask_for_norm': [False], 'resampling_fn_data': 'resample_data_or_seg_to_shape', 'resampling_fn_seg': 'resample_data_or_seg_to_shape', 'resampling_fn_data_kwargs': {'is_seg': False, 'order': 3, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_seg_kwargs': {'is_seg': True, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_probabilities': 'resample_data_or_seg_to_shape', 'resampling_fn_probabilities_kwargs': {'is_seg': False, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'architecture': {'network_class_name': 'dynamic_network_architectures.architectures.unet.PlainConvUNet', 'arch_kwargs': {'n_stages': 6, 'features_per_stage': (32, 64, 128, 256, 320, 320), 'conv_op': 'torch.nn.modules.conv.Conv3d', 'kernel_sizes': ((3, 3, 3), (3, 3, 3), (3, 3, 3), (3, 3, 3), (3, 3, 3), (3, 3, 3)), 'strides': ((1, 1, 1), (2, 2, 2), (2, 2, 2), (2, 2, 2), (2, 2, 2), (1, 2, 2)), 'n_conv_per_stage': (2, 2, 2, 2, 2, 2), 'n_conv_per_stage_decoder': (2, 2, 2, 2, 2), 'conv_bias': True, 'norm_op': 'torch.nn.modules.instancenorm.InstanceNorm3d', 'norm_op_kwargs': {'eps': 1e-05, 'affine': True}, 'dropout_op': None, 'dropout_op_kwargs': None, 'nonlin': 'torch.nn.LeakyReLU', 'nonlin_kwargs': {'inplace': True}}, '_kw_requires_import': ('conv_op', 'norm_op', 'dropout_op', 'nonlin')}, 'batch_dice': False, 'next_stage': '3d_cascade_fullres'}
3D fullres U-Net configuration: {'data_identifier': 'nnUNetPlans_3d_fullres', 'preprocessor_name': 'DefaultPreprocessor', 'batch_size': 2, 'patch_size': (80, 192, 160), 'median_image_size_in_voxels': array([252., 512., 512.]), 'spacing': array([1.24497998, 0.78515625, 0.78515625]), 'normalization_schemes': ['CTNormalization'], 'use_mask_for_norm': [False], 'resampling_fn_data': 'resample_data_or_seg_to_shape', 'resampling_fn_seg': 'resample_data_or_seg_to_shape', 'resampling_fn_data_kwargs': {'is_seg': False, 'order': 3, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_seg_kwargs': {'is_seg': True, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_probabilities': 'resample_data_or_seg_to_shape', 'resampling_fn_probabilities_kwargs': {'is_seg': False, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'architecture': {'network_class_name': 'dynamic_network_architectures.architectures.unet.PlainConvUNet', 'arch_kwargs': {'n_stages': 6, 'features_per_stage': (32, 64, 128, 256, 320, 320), 'conv_op': 'torch.nn.modules.conv.Conv3d', 'kernel_sizes': ((3, 3, 3), (3, 3, 3), (3, 3, 3), (3, 3, 3), (3, 3, 3), (3, 3, 3)), 'strides': ((1, 1, 1), (2, 2, 2), (2, 2, 2), (2, 2, 2), (2, 2, 2), (1, 2, 2)), 'n_conv_per_stage': (2, 2, 2, 2, 2, 2), 'n_conv_per_stage_decoder': (2, 2, 2, 2, 2), 'conv_bias': True, 'norm_op': 'torch.nn.modules.instancenorm.InstanceNorm3d', 'norm_op_kwargs': {'eps': 1e-05, 'affine': True}, 'dropout_op': None, 'dropout_op_kwargs': None, 'nonlin': 'torch.nn.LeakyReLU', 'nonlin_kwargs': {'inplace': True}}, '_kw_requires_import': ('conv_op', 'norm_op', 'dropout_op', 'nonlin')}, 'batch_dice': True}
Plans were saved to /mnt/d/AIPRO/nnUNet/nnUNetFrame/DATASET/nnUNet_preprocessed/Dataset006_Lung/nnUNetPlans.json Preprocessing... Preprocessing dataset Dataset006_Lung Configuration: 2d... 0%| | 0/63 [00:09<?, ?it/s] Traceback (most recent call last): File "/home/yjl/anaconda3/envs/nn_Unetv2/bin/nnUNetv2_plan_and_preprocess", line 8, in
sys.exit(plan_and_preprocess_entry())
File "/mnt/d/AIPRO/nnUNet/nnunetv2/experiment_planning/plan_and_preprocess_entrypoints.py", line 197, in plan_and_preprocess_entry
preprocess(args.d, plans_identifier, args.c, np, args.verbose)
File "/mnt/d/AIPRO/nnUNet/nnunetv2/experiment_planning/plan_and_preprocess_api.py", line 142, in preprocess
preprocess_dataset(d, plans_identifier, configurations, num_processes, verbose)
File "/mnt/d/AIPRO/nnUNet/nnunetv2/experiment_planning/plan_and_preprocess_api.py", line 121, in preprocess_dataset
preprocessor.run(dataset_id, c, plans_identifier, num_processes=n)
File "/mnt/d/AIPRO/nnUNet/nnunetv2/preprocessing/preprocessors/default_preprocessor.py", line 246, in run
raise RuntimeError('Some background worker is 6 feet under. Yuck. \n'
RuntimeError: Some background worker is 6 feet under. Yuck.
OK jokes aside.
One of your background processes is missing. This could be because of an error (look for an error message) or because it was killed by your OS due to running out of RAM. If you don't see an error message, out of RAM is likely the problem. In that case reducing the number of workers might help
`