MIC-DKFZ / batchgenerators

A framework for data augmentation for 2D and 3D image classification and segmentation
Apache License 2.0
1.1k stars 221 forks source link

Multiprocessing problem under Windows #92

Open ttdeshuo opened 2 years ago

ttdeshuo commented 2 years ago

I have modified the code according to Windows Support is very experimental! However, the problem also occurs as follows. Have any method to solve it?

Traceback (most recent call last): File "F:/ki67/batchgenerator_example.py", line 106, in main() File "F:/ki67/batchgenerator_example.py", line 97, in main tr_gen.restart() File "C:\Users\Admin.conda\envs\py352\lib\site-packages\batchgenerators\dataloading\multi_threaded_augmenter.py", line 292, in restart self._start() File "C:\Users\Admin.conda\envs\py352\lib\site-packages\batchgenerators\dataloading\multi_threaded_augmenter.py", line 244, in _start self._processes[-1].start() File "C:\Users\Admin.conda\envs\py352\lib\multiprocessing\process.py", line 105, in start self._popen = self._Popen(self) File "C:\Users\Admin.conda\envs\py352\lib\multiprocessing\context.py", line 212, in _Popen return _default_context.get_context().Process._Popen(process_obj) File "C:\Users\Admin.conda\envs\py352\lib\multiprocessing\context.py", line 313, in _Popen return Popen(process_obj) File "C:\Users\Admin.conda\envs\py352\lib\multiprocessing\popen_spawn_win32.py", line 66, in init reduction.dump(process_obj, to_child) File "C:\Users\Admin.conda\envs\py352\lib\multiprocessing\reduction.py", line 59, in dump ForkingPickler(file, protocol).dump(obj) ValueError: ctypes objects containing pointers cannot be pickled Exception ignored in: <bound method MultiThreadedAugmenter.del of <batchgenerators.dataloading.multi_threaded_augmenter.MultiThreadedAugmenter object at 0x0000022D094A66D8>> Traceback (most recent call last): File "C:\Users\Admin.conda\envs\py352\lib\site-packages\batchgenerators\dataloading\multi_threaded_augmenter.py", line 296, in del self._finish() File "C:\Users\Admin.conda\envs\py352\lib\site-packages\batchgenerators\dataloading\multi_threaded_augmenter.py", line 275, in _finish [i.terminate() for i in self._processes] File "C:\Users\Admin.conda\envs\py352\lib\site-packages\batchgenerators\dataloading\multi_threaded_augmenter.py", line 275, in [i.terminate() for i in self._processes] File "C:\Users\Admin.conda\envs\py352\lib\multiprocessing\process.py", line 113, in terminate self._popen.terminate() AttributeError: 'NoneType' object has no attribute 'terminate'

FabianIsensee commented 2 years ago

Hi, the error you are looking for is ValueError: ctypes objects containing pointers cannot be pickled which points to a typical windows problem with python and multiprocessing, see also https://stackoverflow.com/questions/8804830/python-multiprocessing-picklingerror-cant-pickle-type-function I cannot really help you here because I don't use Windows for work-related stuff. Sorry :-/ Best, Fabian