search

MIC-DKFZ / nnUNet

Apache License 2.0
5.45k stars 1.67k forks source link

Verification process: IndexError: index 189 is out of bounds for axis 0 with size 189 #2227

Open Yanfeng-Zhou opened 2 months ago

Yanfeng-Zhou commented 2 months ago

I run nnUNet on KiTS23 dataset and everything went fine during the training process. However, the following error occurred during the subsequent verification process: IndexError: index 189 is out of bounds for axis 0 with size 189

This is the specific error reporting process: ` 2024-05-27 15:21:46.901381: The split file contains 5 splits. 2024-05-27 15:21:46.902513: Desired fold for training: 4 2024-05-27 15:21:46.903332: This split has 392 training and 97 validation cases. 2024-05-27 15:21:46.906973: predicting case_00000 2024-05-27 15:21:46.909710: case_00000, shape torch.Size([1, 130, 256, 256]), rank 0 2024-05-27 15:22:23.336574: predicting case_00005 2024-05-27 15:22:23.344657: case_00005, shape torch.Size([1, 177, 272, 272]), rank 0 2024-05-27 15:22:45.196851: predicting case_00012 2024-05-27 15:22:45.207372: case_00012, shape torch.Size([1, 189, 209, 209]), rank 0 2024-05-27 15:23:01.032862: predicting case_00023 2024-05-27 15:23:01.065956: case_00023, shape torch.Size([1, 136, 217, 217]), rank 0 2024-05-27 15:23:17.362652: predicting case_00029 ... 2024-05-27 15:39:39.123084: predicting case_00409 2024-05-27 15:39:39.147258: case_00409, shape torch.Size([1, 231, 266, 266]), rank 0 2024-05-27 15:40:09.857457: predicting case_00415 2024-05-27 15:40:09.864996: case_00415, shape torch.Size([1, 127, 257, 257]), rank 0 2024-05-27 15:40:24.953745: predicting case_00431 2024-05-27 15:40:24.986037: case_00431, shape torch.Size([1, 269, 212, 212]), rank 0 [2024-05-27 15:40:25,642] torch._dynamo.convert_frame: [WARNING] torch._dynamo hit config.cache_size_limit (64) [2024-05-27 15:40:25,642] torch._dynamo.convert_frame: [WARNING] function: 'forward' (/ldap_shared/home/s_zyf/anaconda3/lib/python3.10/site-packages/dynamic_network_architectures/architectures/unet.py:60) [2024-05-27 15:40:25,642] torch._dynamo.convert_frame: [WARNING] to diagnose recompilation issues, set env variable TORCHDYNAMO_REPORT_GUARD_FAILURES=1 and also see https://pytorch.org/docs/master/compile/troubleshooting.html. 2024-05-27 15:41:02.554147: predicting case_00437 2024-05-27 15:41:02.589972: case_00437, shape torch.Size([1, 192, 242, 242]), rank 0 2024-05-27 15:41:14.987841: predicting case_00443 2024-05-27 15:41:15.009970: case_00443, shape torch.Size([1, 134, 238, 238]), rank 0 2024-05-27 15:41:27.288352: predicting case_00444 2024-05-27 15:41:27.304819: case_00444, shape torch.Size([1, 108, 207, 207]), rank 0 ... 2024-05-27 15:46:56.650295: predicting case_00582 2024-05-27 15:46:56.679888: case_00582, shape torch.Size([1, 277, 213, 213]), rank 0 2024-05-27 15:47:17.279826: predicting case_00585 2024-05-27 15:47:17.314358: case_00585, shape torch.Size([1, 188, 211, 211]), rank 0 Traceback (most recent call last): File "/ldap_shared/home/s_zyf/anaconda3/lib/python3.10/multiprocessing/resource_sharer.py", line 138, in _serve with self._listener.accept() as conn: File "/ldap_shared/home/s_zyf/anaconda3/lib/python3.10/multiprocessing/connection.py", line 466, in accept answer_challenge(c, self._authkey) File "/ldap_shared/home/s_zyf/anaconda3/lib/python3.10/multiprocessing/connection.py", line 757, in answer_challenge response = connection.recv_bytes(256) # reject large message File "/ldap_shared/home/s_zyf/anaconda3/lib/python3.10/multiprocessing/connection.py", line 216, in recv_bytes buf = self._recv_bytes(maxlength) File "/ldap_shared/home/s_zyf/anaconda3/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes buf = self._recv(4) File "/ldap_shared/home/s_zyf/anaconda3/lib/python3.10/multiprocessing/connection.py", line 379, in _recv chunk = read(handle, remaining) ConnectionResetError: [Errno 104] Connection reset by peer multiprocessing.pool.RemoteTraceback: """ Traceback (most recent call last): File "/ldap_shared/home/s_zyf/anaconda3/lib/python3.10/multiprocessing/pool.py", line 125, in worker result = (True, func(*args, kwds)) File "/ldap_shared/home/s_zyf/anaconda3/lib/python3.10/multiprocessing/pool.py", line 51, in starmapstar return list(itertools.starmap(args[0], args[1])) File "/ldap_shared/home/s_zyf/nnUNet/nnunetv2/inference/export_prediction.py", line 88, in export_prediction_from_logits ret = convert_predicted_logits_to_segmentation_with_correct_shape( File "/ldap_shared/home/s_zyf/nnUNet/nnunetv2/inference/export_prediction.py", line 30, in convert_predicted_logits_to_segmentation_with_correct_shape predicted_logits = configuration_manager.resampling_fn_probabilities(predicted_logits, File "/ldap_shared/home/s_zyf/nnUNet/nnunetv2/preprocessing/resampling/default_resampling.py", line 121, in resample_data_or_seg_to_shape data_reshaped = resample_data_or_seg(data, new_shape, is_seg, axis, order, do_separate_z, order_z=order_z) File "/ldap_shared/home/s_zyf/nnUNet/nnunetv2/preprocessing/resampling/default_resampling.py", line 174, in resample_data_or_seg reshaped_here[slice_id] = resize_fn(data[c, :, :, slice_id], new_shape_2d, order, kwargs) IndexError: index 189 is out of bounds for axis 0 with size 189 """

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/ldap_shared/home/s_zyf/anaconda3/bin/nnUNetv2_train", line 8, in sys.exit(run_training_entry()) File "/ldap_shared/home/s_zyf/nnUNet/nnunetv2/run/run_training.py", line 274, in run_training_entry run_training(args.dataset_name_or_id, args.configuration, args.fold, args.tr, args.p, args.pretrained_weights, File "/ldap_shared/home/s_zyf/nnUNet/nnunetv2/run/run_training.py", line 214, in run_training nnunet_trainer.perform_actual_validation(export_validation_probabilities) File "/ldap_shared/home/s_zyf/nnUNet/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py", line 1264, in perform_actualvalidation = [r.get() for r in results] File "/ldap_shared/home/szyf/nnUNet/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py", line 1264, in = [r.get() for r in results] File "/ldap_shared/home/s_zyf/anaconda3/lib/python3.10/multiprocessing/pool.py", line 774, in get raise self._value IndexError: index 189 is out of bounds for axis 0 with size 189 ` Thank you for your reply!

Yanfeng-Zhou commented 2 months ago

By the way, I didn't encounter this error on other datasets (AMOS, BTCV, LiTS, etc.)

seziegler commented 2 months ago

Hi @Yanfeng-Zhou , in the error message it says ConnectionResetError: [Errno 104] Connection reset by peer, could it be that the cluster you're running on has shut down the connection while you were running the prediction? Can you rerun the inference and see if the error persists?

Yanfeng-Zhou commented 2 months ago

Hi @Yanfeng-Zhou , in the error message it says ConnectionResetError: [Errno 104] Connection reset by peer, could it be that the cluster you're running on has shut down the connection while you were running the prediction? Can you rerun the inference and see if the error persists?

I've retried multiple times and still have this problem. I encountered this problem on both RTX 4090 and RTX A6000. Sad!

seziegler commented 2 months ago

Hi @Yanfeng-Zhou , please share your dataset.json and the commands you have run for preprocessing, training and inference so that I can look into your problem further

Yanfeng-Zhou commented 2 months ago

dataset.json { "channel_names": { "0": "CT" }, "labels": { "background": 0, "kidney": 1, "tumor": 2, "cyst": 3 }, "numTraining": 489, "file_ending": ".nii.gz", "name": "Dataset020_KiTS23" }

Preprocessing nnUNetv2_plan_and_preprocess -d 20 --verify_dataset_integrity

Training CUDA_VISIBLE_DEVICES=0 nnUNetv2_train 20 3d_lowres 0

inference I did not use inference, and this error occurred during the verification after each fold training was completed.

By the way, is it possible that it is a hardware problem, such as a bad CPU?

seziegler commented 2 months ago

I'm not sure if this can be caused by a bad CPU. Does it always happen at the same case? Can you try to exclude this case from your data and see if it occurs in other cases too? Also I forgot to ask earlier, can you please also share your plans file?

Yanfeng-Zhou commented 2 months ago

plans (/nnUNetFrame/nnUNet_preprocessed/Dataset020_KiTS23/nnUNetPlans.json) { "dataset_name": "Dataset020_KiTS23", "plans_name": "nnUNetPlans", "original_median_spacing_after_transp": [ 3.0, 0.78125, 0.78125 ], "original_median_shape_after_transp": [ 104, 512, 512 ], "image_reader_writer": "SimpleITKIO", "transpose_forward": [ 2, 0, 1 ], "transpose_backward": [ 1, 2, 0 ], "configurations": { "2d": { "data_identifier": "nnUNetPlans_2d", "preprocessor_name": "DefaultPreprocessor", "batch_size": 12, "patch_size": [ 512, 512 ], "median_image_size_in_voxels": [ 512.0, 512.0 ], "spacing": [ 0.78125, 0.78125 ], "normalization_schemes": [ "CTNormalization" ], "use_mask_for_norm": [ false ], "resampling_fn_data": "resample_data_or_seg_to_shape", "resampling_fn_seg": "resample_data_or_seg_to_shape", "resampling_fn_data_kwargs": { "is_seg": false, "order": 3, "order_z": 0, "force_separate_z": null }, "resampling_fn_seg_kwargs": { "is_seg": true, "order": 1, "order_z": 0, "force_separate_z": null }, "resampling_fn_probabilities": "resample_data_or_seg_to_shape", "resampling_fn_probabilities_kwargs": { "is_seg": false, "order": 1, "order_z": 0, "force_separate_z": null }, "architecture": { "network_class_name": "dynamic_network_architectures.architectures.unet.PlainConvUNet", "arch_kwargs": { "n_stages": 8, "features_per_stage": [ 32, 64, 128, 256, 512, 512, 512, 512 ], "conv_op": "torch.nn.modules.conv.Conv2d", "kernel_sizes": [ [ 3, 3 ], [ 3, 3 ], [ 3, 3 ], [ 3, 3 ], [ 3, 3 ], [ 3, 3 ], [ 3, 3 ], [ 3, 3 ] ], "strides": [ [ 1, 1 ], [ 2, 2 ], [ 2, 2 ], [ 2, 2 ], [ 2, 2 ], [ 2, 2 ], [ 2, 2 ], [ 2, 2 ] ], "n_conv_per_stage": [ 2, 2, 2, 2, 2, 2, 2, 2 ], "n_conv_per_stage_decoder": [ 2, 2, 2, 2, 2, 2, 2 ], "conv_bias": true, "norm_op": "torch.nn.modules.instancenorm.InstanceNorm2d", "norm_op_kwargs": { "eps": 1e-05, "affine": true }, "dropout_op": null, "dropout_op_kwargs": null, "nonlin": "torch.nn.LeakyReLU", "nonlin_kwargs": { "inplace": true } }, "_kw_requires_import": [ "conv_op", "norm_op", "dropout_op", "nonlin" ] }, "batch_dice": true }, "3d_lowres": { "data_identifier": "nnUNetPlans_3d_lowres", "preprocessor_name": "DefaultPreprocessor", "batch_size": 2, "patch_size": [ 128, 128, 128 ], "median_image_size_in_voxels": [ 177, 217, 217 ], "spacing": [ 2.3565655060093813, 1.841066801569828, 1.841066801569828 ], "normalization_schemes": [ "CTNormalization" ], "use_mask_for_norm": [ false ], "resampling_fn_data": "resample_data_or_seg_to_shape", "resampling_fn_seg": "resample_data_or_seg_to_shape", "resampling_fn_data_kwargs": { "is_seg": false, "order": 3, "order_z": 0, "force_separate_z": null }, "resampling_fn_seg_kwargs": { "is_seg": true, "order": 1, "order_z": 0, "force_separate_z": null }, "resampling_fn_probabilities": "resample_data_or_seg_to_shape", "resampling_fn_probabilities_kwargs": { "is_seg": false, "order": 1, "order_z": 0, "force_separate_z": null }, "architecture": { "network_class_name": "dynamic_network_architectures.architectures.unet.PlainConvUNet", "arch_kwargs": { "n_stages": 6, "features_per_stage": [ 32, 64, 128, 256, 320, 320 ], "conv_op": "torch.nn.modules.conv.Conv3d", "kernel_sizes": [ [ 3, 3, 3 ], [ 3, 3, 3 ], [ 3, 3, 3 ], [ 3, 3, 3 ], [ 3, 3, 3 ], [ 3, 3, 3 ] ], "strides": [ [ 1, 1, 1 ], [ 2, 2, 2 ], [ 2, 2, 2 ], [ 2, 2, 2 ], [ 2, 2, 2 ], [ 2, 2, 2 ] ], "n_conv_per_stage": [ 2, 2, 2, 2, 2, 2 ], "n_conv_per_stage_decoder": [ 2, 2, 2, 2, 2 ], "conv_bias": true, "norm_op": "torch.nn.modules.instancenorm.InstanceNorm3d", "norm_op_kwargs": { "eps": 1e-05, "affine": true }, "dropout_op": null, "dropout_op_kwargs": null, "nonlin": "torch.nn.LeakyReLU", "nonlin_kwargs": { "inplace": true } }, "_kw_requires_import": [ "conv_op", "norm_op", "dropout_op", "nonlin" ] }, "batch_dice": false, "next_stage": "3d_cascade_fullres" }, "3d_fullres": { "data_identifier": "nnUNetPlans_3d_fullres", "preprocessor_name": "DefaultPreprocessor", "batch_size": 2, "patch_size": [ 128, 128, 128 ], "median_image_size_in_voxels": [ 417.0, 512.0, 512.0 ], "spacing": [ 1.0, 0.78125, 0.78125 ], "normalization_schemes": [ "CTNormalization" ], "use_mask_for_norm": [ false ], "resampling_fn_data": "resample_data_or_seg_to_shape", "resampling_fn_seg": "resample_data_or_seg_to_shape", "resampling_fn_data_kwargs": { "is_seg": false, "order": 3, "order_z": 0, "force_separate_z": null }, "resampling_fn_seg_kwargs": { "is_seg": true, "order": 1, "order_z": 0, "force_separate_z": null }, "resampling_fn_probabilities": "resample_data_or_seg_to_shape", "resampling_fn_probabilities_kwargs": { "is_seg": false, "order": 1, "order_z": 0, "force_separate_z": null }, "architecture": { "network_class_name": "dynamic_network_architectures.architectures.unet.PlainConvUNet", "arch_kwargs": { "n_stages": 6, "features_per_stage": [ 32, 64, 128, 256, 320, 320 ], "conv_op": "torch.nn.modules.conv.Conv3d", "kernel_sizes": [ [ 3, 3, 3 ], [ 3, 3, 3 ], [ 3, 3, 3 ], [ 3, 3, 3 ], [ 3, 3, 3 ], [ 3, 3, 3 ] ], "strides": [ [ 1, 1, 1 ], [ 2, 2, 2 ], [ 2, 2, 2 ], [ 2, 2, 2 ], [ 2, 2, 2 ], [ 2, 2, 2 ] ], "n_conv_per_stage": [ 2, 2, 2, 2, 2, 2 ], "n_conv_per_stage_decoder": [ 2, 2, 2, 2, 2 ], "conv_bias": true, "norm_op": "torch.nn.modules.instancenorm.InstanceNorm3d", "norm_op_kwargs": { "eps": 1e-05, "affine": true }, "dropout_op": null, "dropout_op_kwargs": null, "nonlin": "torch.nn.LeakyReLU", "nonlin_kwargs": { "inplace": true } }, "_kw_requires_import": [ "conv_op", "norm_op", "dropout_op", "nonlin" ] }, "batch_dice": true }, "3d_cascade_fullres": { "inherits_from": "3d_fullres", "previous_stage": "3d_lowres" } }, "experiment_planner_used": "ExperimentPlanner", "label_manager": "LabelManager", "foreground_intensity_properties_per_channel": { "0": { "max": 3071.0, "mean": 103.11735534667969, "median": 102.0, "min": -1022.0, "percentile_00_5": -58.0, "percentile_99_5": 302.0, "std": 73.36670684814453 } } }

It doesn't always happen in the same case, and the same error occurs on each of the five folds. I refer to the dataset used in your new paper (nnU-Net Revisited: A Call for Rigorous Validation in 3D Medical Image Segmentation). Other datasets are fine, but this data set has this strange error.

seziegler commented 2 months ago

There is the dataset conversion script for this dataset here. If you converted it differently, can you check if your error still persists if you do the dataset conversion as in the script? Especially the NibabelIOWithReorient could be important.

sahikabetul commented 1 week ago

Hello, I have the same problem. Do not understand what is the problem. Could you help me?

My dataset structure same as nnUNet structure. I also runned dataset verification via this command, and corrected the mistakes. So, I don't think it caused from my dataset:

nnUNetv2_plan_and_preprocess -d 666 --verify_dataset_integrity

The code that I am using for training process:

nnUNetv2_train 666 2d all -p nnUNetPlansSpine -device mps -tr nnUNetTrainer_10epochs

My dataset including 12 training and 3 test images and labels. The error message produced after 12nd image. The error message:


2024-07-20 23:18:25.235026: unpacking done...
2024-07-20 23:18:25.244699: Unable to plot network architecture:
2024-07-20 23:18:25.246547: No module named 'hiddenlayer'
2024-07-20 23:18:25.277605: 
2024-07-20 23:18:25.279938: Epoch 0
2024-07-20 23:18:25.282348: Current learning rate: 0.01
/Users/yayli.sahikabetul/Desktop/vestibularSchwannoma/spineSegm/.venv/lib/python3.9/site-packages/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py:1044: RuntimeWarning: invalid value encountered in scalar divide
  global_dc_per_class = [i for i in [2 * i / (2 * i + j + k) for i, j, k in zip(tp, fp, fn)]]
2024-07-20 23:28:10.494386: train_loss 0.4936
2024-07-20 23:28:10.498129: val_loss 0.2855
2024-07-20 23:28:10.499977: Pseudo dice [np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(nan), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(nan)]
2024-07-20 23:28:10.501816: Epoch time: 585.22 s
2024-07-20 23:28:10.503826: Yayy! New best EMA pseudo Dice: 0.0
2024-07-20 23:28:11.961117: 
2024-07-20 23:28:11.963388: Epoch 1
2024-07-20 23:28:11.965194: Current learning rate: 0.0091
2024-07-20 23:37:53.176200: train_loss 0.2361
2024-07-20 23:37:53.187152: val_loss 0.165
2024-07-20 23:37:53.195680: Pseudo dice [np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(nan), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(nan)]
2024-07-20 23:37:53.251974: Epoch time: 581.22 s
2024-07-20 23:37:54.172962: 
2024-07-20 23:37:54.175742: Epoch 2
2024-07-20 23:37:54.177841: Current learning rate: 0.00818
2024-07-20 23:47:38.024999: train_loss 0.142
2024-07-20 23:47:38.026308: val_loss 0.0908
2024-07-20 23:47:38.029089: Pseudo dice [np.float32(0.0), np.float32(0.0175), np.float32(0.2913), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(nan), np.float32(0.6258), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(nan)]
2024-07-20 23:47:38.031622: Epoch time: 583.85 s
2024-07-20 23:47:38.033793: Yayy! New best EMA pseudo Dice: 0.005499999970197678
2024-07-20 23:47:39.316096: 
2024-07-20 23:47:39.316566: Epoch 3
2024-07-20 23:47:39.318655: Current learning rate: 0.00725
2024-07-20 23:57:21.485537: train_loss 0.0842
2024-07-20 23:57:21.487911: val_loss 0.0554
2024-07-20 23:57:21.497697: Pseudo dice [np.float32(0.0006), np.float32(0.0423), np.float32(0.3128), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(nan), np.float32(0.8645), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(nan)]
2024-07-20 23:57:21.506062: Epoch time: 582.17 s
2024-07-20 23:57:21.512560: Yayy! New best EMA pseudo Dice: 0.01209999993443489
2024-07-20 23:57:22.835561: 
2024-07-20 23:57:22.835871: Epoch 4
2024-07-20 23:57:22.838091: Current learning rate: 0.00631
2024-07-21 00:06:59.036105: train_loss 0.0536
2024-07-21 00:06:59.037666: val_loss 0.0135
2024-07-21 00:06:59.048433: Pseudo dice [np.float32(0.1339), np.float32(0.1727), np.float32(0.3657), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(nan), np.float32(0.8807), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(nan)]
2024-07-21 00:06:59.052185: Epoch time: 576.2 s
2024-07-21 00:06:59.060036: Yayy! New best EMA pseudo Dice: 0.019999999552965164
2024-07-21 00:07:00.333239: 
2024-07-21 00:07:00.335638: Epoch 5
2024-07-21 00:07:00.337388: Current learning rate: 0.00536
2024-07-21 00:16:36.923052: train_loss -0.0007
2024-07-21 00:16:36.934732: val_loss -0.0591
2024-07-21 00:16:36.942990: Pseudo dice [np.float32(0.1958), np.float32(0.0345), np.float32(0.4511), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(nan), np.float32(0.8998), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(nan)]
2024-07-21 00:16:36.947974: Epoch time: 576.59 s
2024-07-21 00:16:36.955770: Yayy! New best EMA pseudo Dice: 0.027300000190734863
2024-07-21 00:16:38.211440: 
2024-07-21 00:16:38.213857: Epoch 6
2024-07-21 00:16:38.215755: Current learning rate: 0.00438
2024-07-21 00:26:14.436854: train_loss -0.08
2024-07-21 00:26:14.448916: val_loss -0.1325
2024-07-21 00:26:14.455781: Pseudo dice [np.float32(0.3656), np.float32(0.3317), np.float32(0.4989), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(nan), np.float32(0.883), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(nan)]
2024-07-21 00:26:14.461033: Epoch time: 576.23 s
2024-07-21 00:26:14.467067: Yayy! New best EMA pseudo Dice: 0.03680000081658363
2024-07-21 00:26:15.763127: 
2024-07-21 00:26:15.765329: Epoch 7
2024-07-21 00:26:15.767034: Current learning rate: 0.00338
2024-07-21 00:35:52.626220: train_loss -0.1356
2024-07-21 00:35:52.630180: val_loss -0.189
2024-07-21 00:35:52.632497: Pseudo dice [np.float32(0.529), np.float32(0.6236), np.float32(0.3977), np.float32(0.0), np.float32(0.0872), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(nan), np.float32(0.898), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(nan)]
2024-07-21 00:35:52.634646: Epoch time: 576.86 s
2024-07-21 00:35:52.636826: Yayy! New best EMA pseudo Dice: 0.04809999838471413
2024-07-21 00:35:53.920485: 
2024-07-21 00:35:53.923115: Epoch 8
2024-07-21 00:35:53.924836: Current learning rate: 0.00235
2024-07-21 00:45:30.268207: train_loss -0.186
2024-07-21 00:45:30.279616: val_loss -0.2283
2024-07-21 00:45:30.287670: Pseudo dice [np.float32(0.7281), np.float32(0.7562), np.float32(0.4545), np.float32(0.0082), np.float32(0.5023), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(nan), np.float32(0.9145), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(nan)]
2024-07-21 00:45:30.296341: Epoch time: 576.35 s
2024-07-21 00:45:30.303141: Yayy! New best EMA pseudo Dice: 0.06310000270605087
2024-07-21 00:45:31.745094: 
2024-07-21 00:45:31.747323: Epoch 9
2024-07-21 00:45:31.748963: Current learning rate: 0.00126
2024-07-21 00:55:08.095404: train_loss -0.2213
2024-07-21 00:55:08.107355: val_loss -0.2494
2024-07-21 00:55:08.113922: Pseudo dice [np.float32(0.8652), np.float32(0.8642), np.float32(0.5754), np.float32(0.0067), np.float32(0.5891), np.float32(1e-04), np.float32(0.0), np.float32(0.0), np.float32(nan), np.float32(0.914), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(0.0), np.float32(nan)]
2024-07-21 00:55:08.119577: Epoch time: 576.35 s
2024-07-21 00:55:08.126232: Yayy! New best EMA pseudo Dice: 0.07919999957084656
2024-07-21 00:55:10.458365: Training done.
perform_everything_on_device=True is only supported for cuda devices! Setting this to False
2024-07-21 00:55:10.604099: predicting 112
2024-07-21 00:55:10.611278: 112, shape torch.Size([1, 24, 448, 448]), rank 0
2024-07-21 00:55:15.567606: predicting 113
2024-07-21 00:55:15.590546: 113, shape torch.Size([1, 26, 449, 450]), rank 0
/Users/yayli.sahikabetul/Desktop/vestibularSchwannoma/spineSegm/.venv/lib/python3.9/site-packages/urllib3/__init__.py:35: NotOpenSSLWarning: urllib3 v2 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 2.8.3'. See: https://github.com/urllib3/urllib3/issues/3020
  warnings.warn(
/Users/yayli.sahikabetul/Desktop/vestibularSchwannoma/spineSegm/.venv/lib/python3.9/site-packages/urllib3/__init__.py:35: NotOpenSSLWarning: urllib3 v2 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 2.8.3'. See: https://github.com/urllib3/urllib3/issues/3020
  warnings.warn(
/Users/yayli.sahikabetul/Desktop/vestibularSchwannoma/spineSegm/.venv/lib/python3.9/site-packages/urllib3/__init__.py:35: NotOpenSSLWarning: urllib3 v2 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 2.8.3'. See: https://github.com/urllib3/urllib3/issues/3020
  warnings.warn(
/Users/yayli.sahikabetul/Desktop/vestibularSchwannoma/spineSegm/.venv/lib/python3.9/site-packages/urllib3/__init__.py:35: NotOpenSSLWarning: urllib3 v2 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 2.8.3'. See: https://github.com/urllib3/urllib3/issues/3020
  warnings.warn(
/Users/yayli.sahikabetul/Desktop/vestibularSchwannoma/spineSegm/.venv/lib/python3.9/site-packages/urllib3/__init__.py:35: NotOpenSSLWarning: urllib3 v2 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 2.8.3'. See: https://github.com/urllib3/urllib3/issues/3020
  warnings.warn(
/Users/yayli.sahikabetul/Desktop/vestibularSchwannoma/spineSegm/.venv/lib/python3.9/site-packages/urllib3/__init__.py:35: NotOpenSSLWarning: urllib3 v2 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 2.8.3'. See: https://github.com/urllib3/urllib3/issues/3020
  warnings.warn(
/Users/yayli.sahikabetul/Desktop/vestibularSchwannoma/spineSegm/.venv/lib/python3.9/site-packages/urllib3/__init__.py:35: NotOpenSSLWarning: urllib3 v2 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 2.8.3'. See: https://github.com/urllib3/urllib3/issues/3020
  warnings.warn(
/Users/yayli.sahikabetul/Desktop/vestibularSchwannoma/spineSegm/.venv/lib/python3.9/site-packages/urllib3/__init__.py:35: NotOpenSSLWarning: urllib3 v2 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 2.8.3'. See: https://github.com/urllib3/urllib3/issues/3020
  warnings.warn(
2024-07-21 00:55:24.675225: predicting 118
2024-07-21 00:55:24.692907: 118, shape torch.Size([1, 20, 482, 480]), rank 0
2024-07-21 00:55:31.133328: predicting 165
2024-07-21 00:55:31.147656: 165, shape torch.Size([1, 15, 448, 448]), rank 0
2024-07-21 00:55:33.602367: predicting 181
2024-07-21 00:55:33.616183: 181, shape torch.Size([1, 19, 422, 416]), rank 0
2024-07-21 00:55:36.701741: predicting 38
2024-07-21 00:55:36.716890: 38, shape torch.Size([1, 30, 456, 448]), rank 0
2024-07-21 00:55:41.589948: predicting 39
2024-07-21 00:55:41.605067: 39, shape torch.Size([1, 26, 451, 448]), rank 0
2024-07-21 00:55:45.838206: predicting 44
2024-07-21 00:55:45.855270: 44, shape torch.Size([1, 24, 448, 448]), rank 0
2024-07-21 00:55:49.741971: predicting 45
2024-07-21 00:55:49.757362: 45, shape torch.Size([1, 17, 480, 480]), rank 0
2024-07-21 00:55:55.279124: predicting 71
2024-07-21 00:55:55.304966: 71, shape torch.Size([1, 15, 448, 448]), rank 0
2024-07-21 00:55:57.754477: predicting 94
2024-07-21 00:55:57.771436: 94, shape torch.Size([1, 26, 450, 448]), rank 0
2024-07-21 00:56:01.993409: predicting 95
2024-07-21 00:56:02.007638: 95, shape torch.Size([1, 24, 448, 448]), rank 0
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 51, in starmapstar
    return list(itertools.starmap(args[0], args[1]))
  File "/Users/yayli.sahikabetul/Desktop/vestibularSchwannoma/spineSegm/.venv/lib/python3.9/site-packages/nnunetv2/inference/export_prediction.py", line 88, in export_prediction_from_logits
    ret = convert_predicted_logits_to_segmentation_with_correct_shape(
  File "/Users/yayli.sahikabetul/Desktop/vestibularSchwannoma/spineSegm/.venv/lib/python3.9/site-packages/nnunetv2/inference/export_prediction.py", line 30, in convert_predicted_logits_to_segmentation_with_correct_shape
    predicted_logits = configuration_manager.resampling_fn_probabilities(predicted_logits,
  File "/Users/yayli.sahikabetul/Desktop/vestibularSchwannoma/spineSegm/.venv/lib/python3.9/site-packages/nnunetv2/preprocessing/resampling/default_resampling.py", line 121, in resample_data_or_seg_to_shape
    data_reshaped = resample_data_or_seg(data, new_shape, is_seg, axis, order, do_separate_z, order_z=order_z)
  File "/Users/yayli.sahikabetul/Desktop/vestibularSchwannoma/spineSegm/.venv/lib/python3.9/site-packages/nnunetv2/preprocessing/resampling/default_resampling.py", line 174, in resample_data_or_seg
    reshaped_here[slice_id] = resize_fn(data[c, :, :, slice_id], new_shape_2d, order, **kwargs)
IndexError: index 26 is out of bounds for axis 0 with size 26
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/yayli.sahikabetul/Desktop/vestibularSchwannoma/spineSegm/.venv/bin/nnUNetv2_train", line 8, in <module>
    sys.exit(run_training_entry())
  File "/Users/yayli.sahikabetul/Desktop/vestibularSchwannoma/spineSegm/.venv/lib/python3.9/site-packages/nnunetv2/run/run_training.py", line 274, in run_training_entry
    run_training(args.dataset_name_or_id, args.configuration, args.fold, args.tr, args.p, args.pretrained_weights,
  File "/Users/yayli.sahikabetul/Desktop/vestibularSchwannoma/spineSegm/.venv/lib/python3.9/site-packages/nnunetv2/run/run_training.py", line 214, in run_training
    nnunet_trainer.perform_actual_validation(export_validation_probabilities)
  File "/Users/yayli.sahikabetul/Desktop/vestibularSchwannoma/spineSegm/.venv/lib/python3.9/site-packages/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py", line 1263, in perform_actual_validation
    _ = [r.get() for r in results]
  File "/Users/yayli.sahikabetul/Desktop/vestibularSchwannoma/spineSegm/.venv/lib/python3.9/site-packages/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py", line 1263, in <listcomp>
    _ = [r.get() for r in results]
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 771, in get
    raise self._value
IndexError: index 26 is out of bounds for axis 0 with size 26