MIC-DKFZ / nnUNet

Apache License 2.0
5.72k stars 1.73k forks source link

local variable 'region_labels' referenced before assignment #2181

Closed IEbokai closed 4 months ago

IEbokai commented 5 months ago

I get this error when I run training: nnUNetv2_train 137 3d_fullres 0

############################ INFO: You are using the old nnU-Net default plans. We have updated our recommendations. Please consider using those instead! Read more here: https://github.com/MIC-DKFZ/nnUNet/blob/master/documentation/resenc_presets.md ############################

Using device: cuda:0

####################################################################### Please cite the following paper when using nnU-Net: Isensee, F., Jaeger, P. F., Kohl, S. A., Petersen, J., & Maier-Hein, K. H. (2021). nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods, 18(2), 203-211. #######################################################################

2024-05-14 20:41:51.961834: do_dummy_2d_data_aug: False 2024-05-14 20:41:52.117781: Using splits from existing split file: ./Data/nnUNetdataset/nnUNetv2/nnUNet_preprocessed/Dataset137_BraTS2021/splits_final.json 2024-05-14 20:41:52.122275: The split file contains 5 splits. 2024-05-14 20:41:52.123136: Desired fold for training: 0 2024-05-14 20:41:52.123788: This split has 1000 training and 251 validation cases. using pin_memory on device 0 Exception in background worker 10: local variable 'region_labels' referenced before assignment Traceback (most recent call last): File "./softwares/anaconda3/envs/nnUNetv2/lib/python3.9/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 53, in producer item = next(data_loader) File "./softwares/anaconda3/envs/nnUNetv2/lib/python3.9/site-packages/batchgenerators/dataloading/data_loader.py", line 126, in next return self.generate_train_batch() File "./package/nnUNet/nnunetv2/training/dataloading/data_loader_3d.py", line 63, in generate_train_batch tmp = self.transforms({'image': data_all[b], 'segmentation': seg_all[b]}) File "./softwares/anaconda3/envs/nnUNetv2/lib/python3.9/site-packages/batchgeneratorsv2/transforms/base/basic_transform.py", line 18, in call return self.apply(data_dict, params) File "./softwares/anaconda3/envs/nnUNetv2/lib/python3.9/site-packages/batchgeneratorsv2/transforms/utils/compose.py", line 13, in apply data_dict = t(data_dict) File "./softwares/anaconda3/envs/nnUNetv2/lib/python3.9/site-packages/batchgeneratorsv2/transforms/base/basic_transform.py", line 18, in call return self.apply(data_dict, params) File "./softwares/anaconda3/envs/nnUNetv2/lib/python3.9/site-packages/batchgeneratorsv2/transforms/base/basic_transform.py", line 67, in apply data_dict['segmentation'] = self._apply_to_segmentation(data_dict['segmentation'], **params) File "./softwares/anaconda3/envs/nnUNetv2/lib/python3.9/site-packages/batchgeneratorsv2/transforms/utils/seg_to_regions.py", line 17, in _apply_to_segmentation if isinstance(region_labels, int) or len(region_labels) == 1: UnboundLocalError: local variable 'region_labels' referenced before assignment Traceback (most recent call last): File "./softwares/anaconda3/envs/nnUNetv2/bin/nnUNetv2_train", line 8, in sys.exit(run_training_entry()) File "./package/nnUNet/nnunetv2/run/run_training.py", line 275, in run_training_entry run_training(args.dataset_name_or_id, args.configuration, args.fold, args.tr, args.p, args.pretrained_weights, File "./package/nnUNet/nnunetv2/run/run_training.py", line 211, in run_training nnunet_trainer.run_training() File "./package/nnUNet/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py", line 1338, in run_training self.on_train_start() File "./package/nnUNet/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py", line 882, in on_train_start self.dataloader_train, self.dataloader_val = self.get_dataloaders() File "./package/nnUNet/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py", line 675, in getdataloaders = next(mt_gen_train) File "./softwares/anaconda3/envs/nnUNetv2/lib/python3.9/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 196, in next item = self.__get_next_item() File "./softwares/anaconda3/envs/nnUNetv2/lib/python3.9/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 181, in __get_next_item raise RuntimeError("One or more background workers are no longer alive. Exiting. Please check the " RuntimeError: One or more background workers are no longer alive. Exiting. Please check the print statements above for the actual error message

IEbokai commented 5 months ago

The wrong fuction code is :

class ConvertSegmentationToRegionsTransform(SegOnlyTransform):
    def __init__(self, regions: Union[List, Tuple], channel_in_seg: int = 0):
        super().__init__()
        self.regions = regions
        self.channel_in_seg = channel_in_seg

    def _apply_to_segmentation(self, segmentation: torch.Tensor, **params) -> torch.Tensor:
        num_regions = len(self.regions)
        region_output = torch.zeros((num_regions, *segmentation.shape[1:]), dtype=torch.bool, device=segmentation.device)
        if isinstance(region_labels, int) or len(region_labels) == 1:
            if not isinstance(region_labels, int):
                region_labels = region_labels[0]
            region_output[:, region_id] = seg[:, self.seg_channel] == region_labels
        else:
            region_output[:, region_id] |= np.isin(seg[:, self.seg_channel], region_labels)
        return region_output.to(segmentation.dtype)
jiashizuo commented 4 months ago

Excuse me, have you solved it? I also encountered this error report.

htcwf89 commented 4 months ago

I'm running into the same error when trying to run region-based training. I traced the issue back to the "ConvertSegmentationToRegionsTransform" class in the batchgeneratorsv2 package (batchgeneratorsv2/batchgeneratorsv2/transforms/utils/seg_to_regions.py). It appears the latest update by @FabianIsensee that was pushed as the result of this comment https://github.com/MIC-DKFZ/nnUNet/issues/2136#issuecomment-2085004267 has broken the region-based training in nnunetv2. However, I tried manually reverting the "ConvertSegmentationToRegionsTransform" class to previous versions visible in the history there but those introduced different errors instead. @FabianIsensee & @GregorKoehler could you guys please take a look?

p.s. I'm using nnunetv2 V2.5, torch 2.1.2+cu118, and batchgeneratorsv2 0.1.1 My OS is Linux and my GPUs are RTX A6000s.

FabianIsensee commented 4 months ago

Hey, yeah that was dumbdumb. Fixed it now - please install nnUNetv2 and batchgeneratorsv2 from their respective master branches

Nastii22 commented 2 months ago

I'm running into the same error when trying to run region-based training. I traced the issue back to the "ConvertSegmentationToRegionsTransform" class in the batchgeneratorsv2 package (batchgeneratorsv2/batchgeneratorsv2/transforms/utils/seg_to_regions.py). It appears the latest update by @FabianIsensee that was pushed as the result of this comment #2136 (comment) has broken the region-based training in nnunetv2. However, I tried manually reverting the "ConvertSegmentationToRegionsTransform" class to previous versions visible in the history there but those introduced different errors instead. @FabianIsensee & @GregorKoehler could you guys please take a look?

p.s. I'm using nnunetv2 V2.5, torch 2.1.2+cu118, and batchgeneratorsv2 0.1.1 My OS is Linux and my GPUs are RTX A6000s.

Hi @htcwf89 could you solve this issue?