MIC-DKFZ / medicaldetectiontoolkit

The Medical Detection Toolkit contains 2D + 3D implementations of prevalent object detectors such as Mask R-CNN, Retina Net, Retina U-Net, as well as a training and inference framework focused on dealing with medical images.
Apache License 2.0
1.31k stars 297 forks source link

troubles with MultiThreadedAugmenter #114

Closed theVmagnificient closed 4 years ago

theVmagnificient commented 4 years ago

Greetings! When I'm starting training - I'm getting the next error, how could it be fixed?

data set loaded with: 621 train / 207 val / 207 test patients starting training epoch 1 Traceback (most recent call last): File "exec.py", line 224, in <module> train(logger) File "exec.py", line 90, in train batch = next(batch_gen['train']) File "/data/adel_volume/kirill/lungs/medicaldetectiontoolkit/mdt/lib/python3.6/site-packages/batchgenerators-0.20.1-py3.6.egg/batchgenerators/dataloading/multi_threaded_augmenter.py", line 201, in __next__ File "/data/adel_volume/kirill/lungs/medicaldetectiontoolkit/mdt/lib/python3.6/site-packages/batchgenerators-0.20.1-py3.6.egg/batchgenerators/dataloading/multi_threaded_augmenter.py", line 234, in _start File "/data/adel_volume/kirill/lungs/medicaldetectiontoolkit/mdt/lib/python3.6/site-packages/threadpoolctl-2.1.0-py3.6.egg/threadpoolctl.py", line 171, in __init__ self._original_info = self._set_threadpool_limits() File "/data/adel_volume/kirill/lungs/medicaldetectiontoolkit/mdt/lib/python3.6/site-packages/threadpoolctl-2.1.0-py3.6.egg/threadpoolctl.py", line 269, in _set_threadpool_limits user_api=self._user_api) File "/data/adel_volume/kirill/lungs/medicaldetectiontoolkit/mdt/lib/python3.6/site-packages/threadpoolctl-2.1.0-py3.6.egg/threadpoolctl.py", line 340, in __init__ self._load_modules() File "/data/adel_volume/kirill/lungs/medicaldetectiontoolkit/mdt/lib/python3.6/site-packages/threadpoolctl-2.1.0-py3.6.egg/threadpoolctl.py", line 375, in _load_modules self._find_modules_with_dl_iterate_phdr() File "/data/adel_volume/kirill/lungs/medicaldetectiontoolkit/mdt/lib/python3.6/site-packages/threadpoolctl-2.1.0-py3.6.egg/threadpoolctl.py", line 387, in _find_modules_with_dl_iterate_phdr libc = self._get_libc() File "/data/adel_volume/kirill/lungs/medicaldetectiontoolkit/mdt/lib/python3.6/site-packages/threadpoolctl-2.1.0-py3.6.egg/threadpoolctl.py", line 556, in _get_libc libc = ctypes.CDLL(libc_name, mode=_RTLD_NOLOAD) File "/usr/lib/python3.6/ctypes/__init__.py", line 348, in __init__ self._handle = _dlopen(self._name, mode) OSError: dlopen() error Exception ignored in: <bound method MultiThreadedAugmenter.__del__ of <batchgenerators.dataloading.multi_threaded_augmenter.MultiThreadedAugmenter object at 0x7f26f94e0390>> Traceback (most recent call last): File "/data/adel_volume/kirill/lungs/medicaldetectiontoolkit/mdt/lib/python3.6/site-packages/batchgenerators-0.20.1-py3.6.egg/batchgenerators/dataloading/multi_threaded_augmenter.py", line 287, in __del__ File "/data/adel_volume/kirill/lungs/medicaldetectiontoolkit/mdt/lib/python3.6/site-packages/batchgenerators-0.20.1-py3.6.egg/batchgenerators/dataloading/multi_threaded_augmenter.py", line 262, in _finish AttributeError: 'NoneType' object has no attribute 'is_alive' Exception ignored in: <bound method MultiThreadedAugmenter.__del__ of <batchgenerators.dataloading.multi_threaded_augmenter.MultiThreadedAugmenter object at 0x7f2767cdecf8>> Traceback (most recent call last): File "/data/adel_volume/kirill/lungs/medicaldetectiontoolkit/mdt/lib/python3.6/site-packages/batchgenerators-0.20.1-py3.6.egg/batchgenerators/dataloading/multi_threaded_augmenter.py", line 287, in __del__ File "/data/adel_volume/kirill/lungs/medicaldetectiontoolkit/mdt/lib/python3.6/site-packages/batchgenerators-0.20.1-py3.6.egg/batchgenerators/dataloading/multi_threaded_augmenter.py", line 262, in _finish AttributeError: 'NoneType' object has no attribute 'is_alive'

theVmagnificient commented 4 years ago

The issue was in threadctl framework

lovelyyutian commented 3 years ago

Hi,theVmagnificient! I had the same problem. How did you solve it