Error message after replacing PlainConvUNet

[Jiaoyang45]

I replaced the original PlainConvUNet with PlainRollUNet, which is my own model.

The specific steps are to first establish the “PlainRollUNet” network, and then modify the “network_class_name” in “nnUNetPlans.json”.

But when using it for training, I encountered the following problem. I am unable to locate the problem and how to solve it. Can you help me?

############################ INFO: You are using the old nnU-Net default plans. We have updated our recommendations. Please consider using those instead! Read more here: https://github.com/MIC-DKFZ/nnUNet/blob/master/documentation/resenc_presets.md ############################

Using device: cuda:0

####################################################################### Please cite the following paper when using nnU-Net: Isensee, F., Jaeger, P. F., Kohl, S. A., Petersen, J., & Maier-Hein, K. H. (2021). nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods, 18(2), 203-211. #######################################################################

2024-09-19 10:35:09.204235: do_dummy_2d_data_aug: False 2024-09-19 10:35:09.204858: Using splits from existing split file: /home/hongchang/ttt/nnUNet/nnUNetFrame/DATASET/nnUNet_preprocessed/Dataset002_Heart/splits_final.json 2024-09-19 10:35:09.205317: The split file contains 5 splits. 2024-09-19 10:35:09.205409: Desired fold for training: 0 2024-09-19 10:35:09.205462: This split has 16 training and 4 validation cases. using pin_memory on device 0 using pin_memory on device 0 2024-09-19 10:35:21.698486: Using torch.compile... /home/hongchang/.conda/envs/tttnnu/lib/python3.9/site-packages/torch/optim/lr_scheduler.py:28: UserWarning: The verbose parameter is deprecated. Please use get_last_lr() to access the learning rate. warnings.warn("The verbose parameter is deprecated. Please use get_last_lr() "

This is the configuration used by this training: Configuration name: 2d {'data_identifier': 'nnUNetPlans_2d', 'preprocessor_name': 'DefaultPreprocessor', 'batch_size': 40, 'patch_size': [320, 256], 'median_image_size_in_voxels': [320.0, 232.0], 'spacing': [1.25, 1.25], 'normalization_schemes': ['ZScoreNormalization'], 'use_mask_for_norm': [True], 'resampling_fn_data': 'resample_data_or_seg_to_shape', 'resampling_fn_seg': 'resample_data_or_seg_to_shape', 'resampling_fn_data_kwargs': {'is_seg': False, 'order': 3, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_seg_kwargs': {'is_seg': True, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_probabilities': 'resample_data_or_seg_to_shape', 'resampling_fn_probabilities_kwargs': {'is_seg': False, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'architecture': {'network_class_name': 'dynamic_network_architectures.architectures.unet.PlainRollUNet', 'arch_kwargs': {'n_stages': 7, 'features_per_stage': [32, 64, 128, 256, 512, 512, 512], 'conv_op': 'torch.nn.modules.conv.Conv2d', 'kernel_sizes': [[3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3]], 'strides': [[1, 1], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2]], 'n_conv_per_stage': [2, 2, 2, 2, 2, 2, 2], 'n_conv_per_stage_decoder': [2, 2, 2, 2, 2, 2], 'conv_bias': True, 'norm_op': 'torch.nn.modules.instancenorm.InstanceNorm2d', 'norm_op_kwargs': {'eps': 1e-05, 'affine': True}, 'dropout_op': None, 'dropout_op_kwargs': None, 'nonlin': 'torch.nn.LeakyReLU', 'nonlin_kwargs': {'inplace': True}, 'deep_supervision': True}, '_kw_requires_import': ['conv_op', 'norm_op', 'dropout_op', 'nonlin']}, 'batch_dice': True}

These are the global plan.json settings: {'dataset_name': 'Dataset002_Heart', 'plans_name': 'nnUNetPlans', 'original_median_spacing_after_transp': [1.3700000047683716, 1.25, 1.25], 'original_median_shape_after_transp': [115, 320, 232], 'image_reader_writer': 'SimpleITKIO', 'transpose_forward': [0, 1, 2], 'transpose_backward': [0, 1, 2], 'experiment_planner_used': 'ExperimentPlanner', 'label_manager': 'LabelManager', 'foreground_intensity_properties_per_channel': {'0': {'max': 1996.0, 'mean': 1090.214599609375, 'median': 1097.0, 'min': 165.0, 'percentile_00_5': 598.0, 'percentile_99_5': 1521.0, 'std': 165.1737823486328}}}

2024-09-19 10:35:27.534549: unpacking dataset... 2024-09-19 10:35:31.345370: unpacking done... 2024-09-19 10:35:31.370673: Unable to plot network architecture: nnUNet_compile is enabled! 2024-09-19 10:35:31.727179: 2024-09-19 10:35:31.727840: Epoch 0 2024-09-19 10:35:31.728352: Current learning rate: 0.01 Traceback (most recent call last): File "/home/hongchang/.conda/envs/tttnnu/bin/nnUNetv2_train", line 8, in sys.exit(run_training_entry()) File "/home/hongchang/ttt/nnUNet/nnunetv2/run/run_training.py", line 275, in run_training_entry run_training(args.dataset_name_or_id, args.configuration, args.fold, args.tr, args.p, args.pretrained_weights, File "/home/hongchang/ttt/nnUNet/nnunetv2/run/run_training.py", line 211, in run_training nnunet_trainer.run_training() File "/home/hongchang/ttt/nnUNet/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py", line 1346, in run_training train_outputs.append(self.train_step(next(self.dataloader_train))) File "/home/hongchang/ttt/nnUNet/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py", line 973, in train_step output = self.network(data) File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl return self._call_impl(*args, kwargs) File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl return forward_call(*args, *kwargs) File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/site-packages/torch/_dynamo/eval_frame.py", line 451, in _fn return fn(args, kwargs) File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl return self._call_impl(*args, kwargs) File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl return forward_call(*args, *kwargs) File "/home/hongchang/ttt/dynamic-network-architectures/dynamic_network_architectures/architectures/unet.py", line 235, in forward skips = self.encoder(x) File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl return self._call_impl(args, kwargs) File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl return forward_call(*args, kwargs) File "/home/hongchang/ttt/dynamic-network-architectures/dynamic_network_architectures/building_blocks/plain_roll_encoder.py", line 93, in forward x = s(x) File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl return self._call_impl(*args, *kwargs) File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl return forward_call(args, kwargs) File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/site-packages/torch/nn/modules/container.py", line 217, in forward input = module(input) File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl return self._call_impl(*args, kwargs) File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl return forward_call(*args, *kwargs) File "/home/hongchang/ttt/dynamic-network-architectures/dynamic_network_architectures/building_blocks/roll_blocks.py", line 271, in forward return self.convs(x) File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl return self._call_impl(args, kwargs) File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl return forward_call(*args, kwargs) File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/site-packages/torch/nn/modules/container.py", line 217, in forward input = module(input) File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl return self._call_impl(*args, *kwargs) File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl return forward_call(args, kwargs) File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py", line 921, in catch_errors return callback(frame, cache_entry, hooks, frame_state, skip=1) File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py", line 786, in _convert_frame result = inner_convert( File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py", line 400, in _convert_frame_assert return _compile( File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/contextlib.py", line 79, in inner return func(*args, kwds) File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py", line 676, in _compile guarded_code = compile_inner(code, one_graph, hooks, transform) File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/site-packages/torch/_dynamo/utils.py", line 262, in time_wrapper r = func(*args, *kwargs) File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py", line 535, in compile_inner out_code = transform_code_object(code, transform) File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/site-packages/torch/_dynamo/bytecode_transformation.py", line 1036, in transform_code_object transformations(instructions, code_options) File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py", line 165, in _fn return fn(args, kwargs) File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py", line 500, in transform tracer.run() File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 2149, in run super().run() File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 810, in run and self.step() File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 773, in step getattr(self, inst.opname)(inst) File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 2268, in RETURN_VALUE self.output.compile_subgraph( File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/site-packages/torch/_dynamo/output_graph.py", line 971, in compile_subgraph self.compile_and_call_fx_graph(tx, list(reversed(stack_values)), root) File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/contextlib.py", line 79, in inner return func(*args, kwds) File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/site-packages/torch/_dynamo/output_graph.py", line 1168, in compile_and_call_fx_graph compiled_fn = self.call_user_compiler(gm) File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/site-packages/torch/_dynamo/utils.py", line 262, in time_wrapper r = func(*args, *kwargs) File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/site-packages/torch/_dynamo/output_graph.py", line 1241, in call_user_compiler raise BackendCompilerFailed(self.compiler_fn, e).with_traceback( File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/site-packages/torch/_dynamo/output_graph.py", line 1222, in call_user_compiler compiled_fn = compiler_fn(gm, self.example_inputs()) File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/site-packages/torch/_dynamo/repro/after_dynamo.py", line 117, in debug_wrapper compiled_gm = compiler_fn(gm, example_inputs) File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/site-packages/torch/init.py", line 1729, in call return compilefx(model, inputs_, config_patches=self.config) File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/contextlib.py", line 79, in inner return func(args, kwds) File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/site-packages/torch/_inductor/compile_fx.py", line 1330, in compile_fx return aot_autograd( File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/site-packages/torch/_dynamo/backends/common.py", line 58, in compiler_fn cg = aot_module_simplified(gm, example_inputs, kwargs) File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/site-packages/torch/_functorch/aot_autograd.py", line 903, in aot_module_simplified compiled_fn = create_aot_dispatcher_function( File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/site-packages/torch/_dynamo/utils.py", line 262, in time_wrapper r = func(*args, *kwargs) File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/site-packages/torch/_functorch/aot_autograd.py", line 628, in create_aot_dispatcher_function compiled_fn = compiler_fn(flat_fn, fake_flat_args, aot_config, fw_metadata=fw_metadata) File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 443, in aot_wrapper_dedupe return compiler_fn(flat_fn, leaf_flat_args, aot_config, fw_metadata=fw_metadata) File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 648, in aot_wrapper_synthetic_base return compiler_fn(flat_fn, flat_args, aot_config, fw_metadata=fw_metadata) File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/site-packages/torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py", line 352, in aot_dispatch_autograd compiled_fw_func = aot_config.fw_compiler(fw_module, adjusted_flat_args) File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/site-packages/torch/_dynamo/utils.py", line 262, in time_wrapper r = func(args, kwargs) File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/site-packages/torch/_inductor/compile_fx.py", line 1257, in fw_compiler_base return inner_compile( File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/site-packages/torch/_dynamo/repro/after_aot.py", line 83, in debug_wrapper inner_compiled_fn = compiler_fn(gm, example_inputs) File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/site-packages/torch/_inductor/debug.py", line 304, in inner return fn(*args, kwargs) File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/contextlib.py", line 79, in inner return func(*args, *kwds) File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/contextlib.py", line 79, in inner return func(args, kwds) File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/site-packages/torch/_dynamo/utils.py", line 262, in time_wrapper r = func(*args, *kwargs) File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/site-packages/torch/_inductor/compile_fx.py", line 438, in compile_fx_inner compiled_graph = fx_codegen_and_compile( File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/site-packages/torch/_inductor/compile_fx.py", line 714, in fx_codegen_and_compile compiled_fn = graph.compile_to_fn() File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/site-packages/torch/_inductor/graph.py", line 1307, in compile_to_fn return self.compile_to_module().call File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/site-packages/torch/_dynamo/utils.py", line 262, in time_wrapper r = func(args, *kwargs) File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/site-packages/torch/_inductor/graph.py", line 1254, in compile_to_module mod = PyCodeCache.load_by_key_path( File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/site-packages/torch/_inductor/codecache.py", line 2160, in load_by_key_path exec(code, mod.dict, mod.dict) File "/tmp/torchinductor_hongchang/ib/cib3qt6iz7xzw55k37jclecxjwfvxhkxq7u6kyryqhf6xfdylfma.py", line 389, in async_compile.wait(globals()) File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/site-packages/torch/_inductor/codecache.py", line 2715, in wait scope[key] = result.result() File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/site-packages/torch/_inductor/codecache.py", line 2522, in result self.future.result() File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/concurrent/futures/_base.py", line 446, in result return self.get_result() File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/concurrent/futures/_base.py", line 391, in get_result raise self._exception torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised: CompilationError: at 17:2422: rbase = tl.arange(0, RBLOCK)[None, :] x0 = xindex % 2 x1 = (xindex // 2) x3 = xindex tmp3_mean = tl.zeros([XBLOCK, RBLOCK], tl.float32) tmp3_m2 = tl.zeros([XBLOCK, RBLOCK], tl.float32) tmp3_weight = tl.zeros([XBLOCK, RBLOCK], tl.float32) for roffset in range(0, rnumel, RBLOCK): rindex = roffset + rbase rmask = rindex < rnumel r2 = rindex tmp0 = tl.load(in_ptr0 + ((512x1) + ((r2 // (1 + ((((-1) + ks0) // 2)(((-1) + ks1) // 2)) + (((-1) + ks0) // 2) + (((-1) + ks1) // 2)))(((-1) + ks0) // 2)) + ((r2 // (1 + ((((-1) + ks0) // 2)(((-1) + ks1) // 2)) + (((-1) + ks0) // 2) + (((-1) + ks1) // 2)))(((-1) + ks1) // 2)) + ((((-1) + ks1) // 2)(((r2 + (256x0) + (256x0(((-1) + ks0) // 2)) + (256x0(((-1) + ks1) // 2)) + (256x0(((-1) + ks0) // 2)(((-1) + ks1) // 2))) // (1 + (((-1) + ks1) // 2))) % (1 + (((-1) + ks0) // 2)))) + (256x0(1/(1 + ((((-1) + ks0) // 2)(((-1) + ks1) // 2)) + (((-1) + ks0) // 2) + (((-1) + ks1) // 2)))) + (512x1(((-1) + ks0) // 2)) + (512x1(((-1) + ks1) // 2)) + ((r2 // (1 + ((((-1) + ks0) // 2)(((-1) + ks1) // 2)) + (((-1) + ks0) // 2) + (((-1) + ks1) // 2)))(((-1) + ks0) // 2)(((-1) + ks1) // 2)) + (256x0(1/(1 + ((((-1) + ks0) // 2)(((-1) + ks1) // 2)) + (((-1) + ks0) // 2) + (((-1) + ks1) // 2)))((((-1) + ks0) // 2)(((-1) + ks0) // 2))) + (256x0(1/(1 + ((((-1) + ks0) // 2)(((-1) + ks1) // 2)) + (((-1) + ks0) // 2) + (((-1) + ks1) // 2)))((((-1) + ks1) // 2)(((-1) + ks1) // 2))) + (512x0(1/(1 + ((((-1) + ks0) // 2)(((-1) + ks1) // 2)) + (((-1) + ks0) // 2) + (((-1) + ks1) // 2)))(((-1) + ks0) // 2)) + (512x0(1/(1 + ((((-1) + ks0) // 2)(((-1) + ks1) // 2)) + (((-1) + ks0) // 2) + (((-1) + ks1) // 2)))(((-1) + ks1) // 2)) + (512x1(((-1) + ks0) // 2)(((-1) + ks1) // 2)) + (256x0(1/(1 + ((((-1) + ks0) // 2)(((-1) + ks1) // 2)) + (((-1) + ks0) // 2) + (((-1) + ks1) // 2)))((((-1) + ks0) // 2)(((-1) + ks0) // 2))((((-1) + ks1) // 2)(((-1) + ks1) // 2))) + (512x0(1/(1 + ((((-1) + ks0) // 2)(((-1) + ks1) // 2)) + (((-1) + ks0) // 2) + (((-1) + ks1) // 2)))((((-1) + ks0) // 2)(((-1) + ks0) // 2))(((-1) + ks1) // 2)) + (512x0(1/(1 + ((((-1) + ks0) // 2)(((-1) + ks1) // 2)) + (((-1) + ks0) // 2) + (((-1) + ks1) // 2)))((((-1) + ks1) // 2)(((-1) + ks1) // 2))(((-1) + ks0) // 2)) + (1024x0(1/(1 + ((((-1) + ks0) // 2)(((-1) + ks1) // 2)) + (((-1) + ks0) // 2) + (((-1) + ks1) // 2)))(((-1) + ks0) // 2)(((-1) + ks1) // 2)) + (r2 // (1 + ((((-1) + ks0) // 2)(((-1) + ks1) // 2)) + (((-1) + ks0) // 2) + (((-1) + ks1) // 2))) + (r2 % (1 + (((-1) + ks1) // 2))) + (((r2 + (256x0) + (256x0(((-1) + ks0) // 2)) + (256x0(((-1) + ks1) // 2)) + (256x0(((-1) + ks0) // 2)*(((-1) + ks1) // 2))) // (1 + (((-1) + ks1) // 2))) % (1 + (((-1) + ks0) // 2)))), rmask & xmask, eviction_policy='evict_last', other=0.0).to(tl.float32) ^ IncompatibleTypeErrorImpl('invalid operands of type pointer and triton.language.fp32')

Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information

You can suppress this exception and fall back to eager by setting: import torch._dynamo torch._dynamo.config.suppress_errors = True

Exception in thread Thread-2: Traceback (most recent call last): File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/threading.py", line 980, in _bootstrap_inner self.run() File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/threading.py", line 917, in run self._target(*self._args, *self._kwargs) File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 125, in results_loop raise e File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 103, in results_loop raise RuntimeError("One or more background workers are no longer alive. Exiting. Please check the " RuntimeError: One or more background workers are no longer alive. Exiting. Please check the print statements above for the actual error message Exception in thread Thread-1: Traceback (most recent call last): File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/threading.py", line 980, in _bootstrap_inner self.run() File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/threading.py", line 917, in run self._target(self._args, **self._kwargs) File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 125, in results_loop raise e File "/home/hongchang/.conda/envs/tttnnu/lib/python3.9/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 103, in results_loop raise RuntimeError("One or more background workers are no longer alive. Exiting. Please check the " RuntimeError: One or more background workers are no longer alive. Exiting. Please check the print statements above for the actual error message