MultiThreadedAugmenter issues

[designer00]

Problem:

Traceback (most recent call last): File "/data/home/zhangyinglin/anaconda3/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/data/home/zhangyinglin/anaconda3/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, *self._kwargs) File "/data/home/zhangyinglin/anaconda3/lib/python3.7/site-packages/batchgenerators/dataloading/multi_threaded_augmenter.py", line 36, in producer data_loader.set_thread_id(thread_id) AttributeError: '_MultiProcessingDataLoaderIter' object has no attribute 'set_thread_id' Exception in thread Thread-2: Traceback (most recent call last): File "/data/home/zhangyinglin/anaconda3/lib/python3.7/threading.py", line 926, in _bootstrap_inner self.run() File "/data/home/zhangyinglin/anaconda3/lib/python3.7/threading.py", line 870, in run self._target(self._args, **self._kwargs) File "/data/home/zhangyinglin/anaconda3/lib/python3.7/site-packages/batchgenerators/dataloading/multi_threaded_augmenter.py", line 92, in results_loop raise RuntimeError("Abort event was set. So someone died and we should end this madness. \nIMPORTANT: " RuntimeError: Abort event was set. So someone died and we should end this madness. IMPORTANT: This is not the actual error message! Look further up to see what caused the error. Please also check whether your RAM was full

Traceback (most recent call last): File "/data/home/zhangyinglin/workspace/pytorch-unet-cornea/transform.py", line 126, in multithreaded_generator.next() File "/data/home/zhangyinglin/anaconda3/lib/python3.7/site-packages/batchgenerators/dataloading/multi_threaded_augmenter.py", line 182, in next return self.next() File "/data/home/zhangyinglin/anaconda3/lib/python3.7/site-packages/batchgenerators/dataloading/multi_threaded_augmenter.py", line 206, in next item = self.get_next_item() File "/data/home/zhangyinglin/anaconda3/lib/python3.7/site-packages/batchgenerators/dataloading/multi_threaded_augmenter.py", line 190, in get_next_item raise RuntimeError("MultiThreadedAugmenter.abort_event was set, something went wrong. Maybe one of " RuntimeError: MultiThreadedAugmenter.abort_event was set, something went wrong. Maybe one of your workers crashed. This is not the actual error message! Look further up your stdout to see what caused the error. Please also check whether your RAM was full

Process finished with exit code 1

Code as follow:

from batchgenerators.transforms.spatial_transforms import SpatialTransform, MirrorTransform from batchgenerators.dataloading.nondet_multi_threaded_augmenter import NonDetMultiThreadedAugmenter from batchgenerators.dataloading.multi_threaded_augmenter import MultiThreadedAugmenter from batchgenerators.dataloading.data_loader import DataLoaderFromDataset from batchgenerators.transforms.abstract_transforms import Compose import numpy as np from utils.dataset import BasicDataset from torch.utils.data import DataLoader import os import matplotlib.pyplot as plt

-------------------------------------------------------

data_path = '/data2/imed-data/zhangyinglin/Other_Dataset/' data_name = 'Ciliary_split2' train_dir = os.path.join(data_path, data_name, '', 'train/') val_dir = os.path.join(data_path, data_name, '', 'val/') final_weight = os.path.join(data_path, data_name, '', 'weights/final/') body_weight = os.path.join(data_path, data_name, '', 'weights/body/') canny_weight = os.path.join(data_path, data_name, '', 'weights/edge/') dir_mask = os.path.join(data_path, data_name, "", "final_mask/") canny_mask = os.path.join(data_path, data_name, "", "canny_mask/") body_mask = os.path.join(data_path, data_name, "", "body_mask/")

img_size = 192 img_scale = 1

-------------------------------------------------------

transform = []

params = {'do_elastic': False, 'elastic_deform_alpha': (0., 200.), 'elastic_deform_sigma': (9., 13.), 'rotation_x': (-30. / 360 2. np.pi, 30. / 360 2. np.pi), 'rotation_y': (-30. / 360 2. np.pi, 30. / 360 2. np.pi), 'rotation_z': (-30. / 360 2. np.pi, 30. / 360 2. np.pi), 'rotation_p_per_axis': 1, 'do_scaling': True, 'scale_range': (0.7, 1.4), 'border_mode_data': 'constant', 'random_crop': False, 'p_eldef': 0.2, 'p_scale': 0.2, 'p_rot': 0.2, 'independent_scale_factor_for_each_axis': 1, 'patch_size': np.asarray([192, 192]) }

transform.append(SpatialTransform( patch_size=params['patch_size'], patch_center_dist_from_border=None, do_elastic_deform=False, alpha=params['elastic_deform_alpha'], sigma=params['elastic_deform_sigma'], do_rotation=True, angle_x=params["rotation_x"], angle_y=params["rotation_y"], angle_z=params["rotation_z"], p_rot_per_axis=params["rotation_p_per_axis"], do_scale=params["do_scaling"], scale=params["scale_range"], border_mode_data=params["border_mode_data"], border_cval_data=0, order_data=3, border_mode_seg="constant", border_cval_seg=-1, order_seg=1, random_crop=params["random_crop"], data_key='data', label_key='seg', p_el_per_sample=params["p_eldef"], p_scale_per_sample=params["p_scale"], p_rot_per_sample=params["p_rot"], independent_scale_for_each_axis=params["independent_scale_factor_for_each_axis"] ))

if name == 'main': tr_transform = Compose(transform) train_loader = DataLoader(train, batch_size=1, shuffle=True, num_workers=4, pin_memory=True) batchgen = iter(train_loader)

multithreaded_generator = MultiThreadedAugmenter(batchgen, tr_transform, 1, 1, seeds=None) multithreaded_generator.next()

[designer00]

I use the MultiThreadedAugmenter to apply transform, but suffer problems. How can I solve this? I wish the library can be modify to be compatiable with the transform interface of pytorch, so that it can be applied easily.

[FabianIsensee]

Hi, can you please post the entire error message? There are parts missing, all I see is the workers telling me that something happened but the important bit isn't there. Simple way to debug would be to use SingleThreadedAugmenter for development. It will give clearer error messages. Once that works you can just replace it with MultiThreadedAugmenter and it will work

[designer00]

ok, thanks for you reply, I will give it a try. If there is still a problem, I will return the full error message.		Yinglin Zhang

@. | 签名由网易邮箱大师定制 On 3/28/2022 15:48，Fabian @.> wrote：

Hi, can you please post the entire error message? There are parts missing, all I see is the workers telling me that something happened but the important bit isn't there. Simple way to debug would be to use SingleThreadedAugmenter for development. It will give clearer error messages. Once that works you can just replace it with MultiThreadedAugmenter and it will work

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

[FabianIsensee]

does it work now?