DCBIA-OrthoLab / SlicerAutomatedDentalTools

A 3D Slicer extension to use AMASSS, ALI-CBCT and ALI-IOS
Other
86 stars 26 forks source link

RuntimeError: DataLoader worker (pid(s) 22552) exited unexpectedly #33

Closed fridayghost closed 1 year ago

fridayghost commented 1 year ago

I tried running the extension on two separate devices, bone with GTX 1070ti and other with RTX 3050. Both gave the same error. Here is the error that i get in the python console.

Loading scan : C:/Users/frida/OneDrive/Desktop/CBCT/MG_test_scan.nii.gz Correcting scan contrast : C:/Users/frida/OneDrive/Desktop/CBCT/MG_test_scan.nii.gz Importing librairies... No module named 'logic' Found existing installation: monai 0.7.0 Uninstalling monai-0.7.0: ERROR: Exception: Traceback (most recent call last): File "C:\Users\frida\AppData\Local\slicer.org\Slicer 5.3.0-2023-03-27\lib\Python\Lib\shutil.py", line 815, in move os.rename(src, real_dst) FileNotFoundError: [WinError 2] The system cannot find the file specified: 'c:\users\frida\appdata\local\slicer.org\slicer 5.3.0-2023-03-27\lib\python\lib\site-packages\monai-0.7.0.dist-info\' -> 'C:\Users\frida\AppData\Local\Temp\pip-uninstall-gg2psnif\monai-0.7.0.dist-info'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "C:\Users\frida\AppData\Local\slicer.org\Slicer 5.3.0-2023-03-27\lib\Python\Lib\site-packages\pip_internal\cli\base_command.py", line 160, in exc_logging_wrapper status = run_func(*args) File "C:\Users\frida\AppData\Local\slicer.org\Slicer 5.3.0-2023-03-27\lib\Python\Lib\site-packages\pip_internal\commands\uninstall.py", line 105, in run uninstall_pathset = req.uninstall( File "C:\Users\frida\AppData\Local\slicer.org\Slicer 5.3.0-2023-03-27\lib\Python\Lib\site-packages\pip_internal\req\req_install.py", line 664, in uninstall uninstalled_pathset.remove(auto_confirm, verbose) File "C:\Users\frida\AppData\Local\slicer.org\Slicer 5.3.0-2023-03-27\lib\Python\Lib\site-packages\pip_internal\req\req_uninstall.py", line 373, in remove moved.stash(path) File "C:\Users\frida\AppData\Local\slicer.org\Slicer 5.3.0-2023-03-27\lib\Python\Lib\site-packages\pip_internal\req\req_uninstall.py", line 271, in stash renames(path, new_path) File "C:\Users\frida\AppData\Local\slicer.org\Slicer 5.3.0-2023-03-27\lib\Python\Lib\site-packages\pip_internal\utils\misc.py", line 312, in renames shutil.move(old, new) File "C:\Users\frida\AppData\Local\slicer.org\Slicer 5.3.0-2023-03-27\lib\Python\Lib\shutil.py", line 835, in move copy_function(src, real_dst) File "C:\Users\frida\AppData\Local\slicer.org\Slicer 5.3.0-2023-03-27\lib\Python\Lib\shutil.py", line 444, in copy2 copyfile(src, dst, follow_symlinks=follow_symlinks) File "C:\Users\frida\AppData\Local\slicer.org\Slicer 5.3.0-2023-03-27\lib\Python\Lib\shutil.py", line 264, in copyfile with open(src, 'rb') as fsrc: FileNotFoundError: [Errno 2] No such file or directory: 'c:\users\frida\appdata\local\slicer.org\slicer 5.3.0-2023-03-27\lib\python\lib\site-packages\monai-0.7.0.dist-info\'

========= ERROR =========

CLI execution failed:

Traceback (most recent call last): File "", line 1, in File "C:\Users\frida\AppData\Local\slicer.org\Slicer 5.3.0-2023-03-27\lib\Python\Lib\multiprocessing\spawn.py", line 116, in spawn_main exitcode = _main(fd, parent_sentinel) File "C:\Users\frida\AppData\Local\slicer.org\Slicer 5.3.0-2023-03-27\lib\Python\Lib\multiprocessing\spawn.py", line 125, in _main prepare(preparation_data) File "C:\Users\frida\AppData\Local\slicer.org\Slicer 5.3.0-2023-03-27\lib\Python\Lib\multiprocessing\spawn.py", line 236, in prepare _fixup_main_from_path(data['init_main_from_path']) File "C:\Users\frida\AppData\Local\slicer.org\Slicer 5.3.0-2023-03-27\lib\Python\Lib\multiprocessing\spawn.py", line 287, in _fixup_main_from_path main_content = runpy.run_path(main_path, File "C:\Users\frida\AppData\Local\slicer.org\Slicer 5.3.0-2023-03-27\lib\Python\Lib\runpy.py", line 268, in run_path return _run_module_code(code, init_globals, run_name, File "C:\Users\frida\AppData\Local\slicer.org\Slicer 5.3.0-2023-03-27\lib\Python\Lib\runpy.py", line 97, in _run_module_code _run_code(code, mod_globals, init_globals, File "C:\Users\frida\AppData\Local\slicer.org\Slicer 5.3.0-2023-03-27\lib\Python\Lib\runpy.py", line 87, in _run_code exec(code, run_globals) File "C:\Users\frida\AppData\Local\slicer.org\Slicer 5.3.0-2023-03-27\slicer.org\Extensions-31693\SlicerAutomatedDentalTools\lib\Slicer-5.3\cli-modules\AMASSS_CLI.py", line 71, in pip_uninstall('monai') File "C:\Users\frida\AppData\Local\slicer.org\Slicer 5.3.0-2023-03-27\bin\Python\slicer\util.py", line 3622, in pip_uninstall _executePythonModule('pip', args) File "C:\Users\frida\AppData\Local\slicer.org\Slicer 5.3.0-2023-03-27\bin\Python\slicer\util.py", line 3548, in _executePythonModule logProcessOutput(proc) File "C:\Users\frida\AppData\Local\slicer.org\Slicer 5.3.0-2023-03-27\bin\Python\slicer\util.py", line 3517, in logProcessOutput raise CalledProcessError(retcode, proc.args, output=proc.stdout, stderr=proc.stderr) subprocess.CalledProcessError: Command '['C:\Users\frida\AppData\Local\slicer.org\Slicer 5.3.0-2023-03-27\bin/PythonSlicer.exe', '-m', 'pip', 'uninstall', 'monai', '--yes']' returned non-zero exit status 2. Traceback (most recent call last): File "C:\Users\frida\AppData\Local\slicer.org\Slicer 5.3.0-2023-03-27\lib\Python\Lib\site-packages\torch\utils\data\dataloader.py", line 1133, in _try_get_data data = self._data_queue.get(timeout=timeout) File "C:\Users\frida\AppData\Local\slicer.org\Slicer 5.3.0-2023-03-27\lib\Python\Lib\multiprocessing\queues.py", line 114, in get raise Empty _queue.Empty

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "C:\Users\frida\AppData\Local\slicer.org\Slicer 5.3.0-2023-03-27\slicer.org\Extensions-31693\SlicerAutomatedDentalTools\lib\Slicer-5.3\cli-modules\AMASSS_CLI.py", line 1220, in main(args) File "C:\Users\frida\AppData\Local\slicer.org\Slicer 5.3.0-2023-03-27\slicer.org\Extensions-31693\SlicerAutomatedDentalTools\lib\Slicer-5.3\cli-modules\AMASSS_CLI.py", line 908, in main for step, batch in enumerate(pred_loader): File "C:\Users\frida\AppData\Local\slicer.org\Slicer 5.3.0-2023-03-27\lib\Python\Lib\site-packages\torch\utils\data\dataloader.py", line 634, in next data = self._next_data() File "C:\Users\frida\AppData\Local\slicer.org\Slicer 5.3.0-2023-03-27\lib\Python\Lib\site-packages\torch\utils\data\dataloader.py", line 1329, in _next_data idx, data = self._get_data() File "C:\Users\frida\AppData\Local\slicer.org\Slicer 5.3.0-2023-03-27\lib\Python\Lib\site-packages\torch\utils\data\dataloader.py", line 1295, in _get_data success, data = self._try_get_data() File "C:\Users\frida\AppData\Local\slicer.org\Slicer 5.3.0-2023-03-27\lib\Python\Lib\site-packages\torch\utils\data\dataloader.py", line 1146, in _try_get_data raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) from e RuntimeError: DataLoader worker (pid(s) 22552) exited unexpectedly

lucanchling commented 1 year ago

Can you check which version of monai you have on the Slicer's python, it should be 0.7.0 @fridayghost It seems like the error is coming from monai installation (here uninstallation of the version that you had to replace it by monai 0.7.0)

fridayghost commented 1 year ago

The installed version of monai is 0.7.0

juangmagic commented 1 year ago

I found the same error using 3D Slicer 5.2.2 and 5.3.0 when using the amass extension. The problem appears running on windows an MacOS.

========= ERROR =========

CLI execution failed:

AMASSS_CLI standard error:

Traceback (most recent call last): File "", line 1, in File "/Applications/Slicer.app/Contents/lib/Python/lib/python3.9/multiprocessing/spawn.py", line 116, in spawn_main exitcode = _main(fd, parent_sentinel) File "/Applications/Slicer.app/Contents/lib/Python/lib/python3.9/multiprocessing/spawn.py", line 125, in _main prepare(preparation_data) File "/Applications/Slicer.app/Contents/lib/Python/lib/python3.9/multiprocessing/spawn.py", line 236, in prepare _fixup_main_from_path(data['init_main_from_path']) File "/Applications/Slicer.app/Contents/lib/Python/lib/python3.9/multiprocessing/spawn.py", line 287, in _fixup_main_from_path main_content = runpy.run_path(main_path, File "/Applications/Slicer.app/Contents/lib/Python/lib/python3.9/runpy.py", line 268, in run_path return _run_module_code(code, init_globals, run_name, File "/Applications/Slicer.app/Contents/lib/Python/lib/python3.9/runpy.py", line 97, in _run_module_code _run_code(code, mod_globals, init_globals, File "/Applications/Slicer.app/Contents/lib/Python/lib/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/Applications/Slicer.app/Contents/Extensions-31382/SlicerAutomatedDentalTools/lib/Slicer-5.2/cli-modules/AMASSS_CLI.py", line 71, in pip_uninstall('monai -q') File "/Applications/Slicer.app/Contents/bin/Python/slicer/util.py", line 3614, in pip_uninstall _executePythonModule('pip', args) File "/Applications/Slicer.app/Contents/bin/Python/slicer/util.py", line 3540, in _executePythonModule logProcessOutput(proc) File "/Applications/Slicer.app/Contents/bin/Python/slicer/util.py", line 3509, in logProcessOutput raise CalledProcessError(retcode, proc.args, output=proc.stdout, stderr=proc.stderr) subprocess.CalledProcessError: Command '['/Applications/Slicer.app/Contents/bin/PythonSlicer', '-m', 'pip', 'uninstall', 'monai', '-q', '--yes']' returned non-zero exit status 2. Traceback (most recent call last): File "", line 1, in File "/Applications/Slicer.app/Contents/lib/Python/lib/python3.9/multiprocessing/spawn.py", line 116, in spawn_main exitcode = _main(fd, parent_sentinel) File "/Applications/Slicer.app/Contents/lib/Python/lib/python3.9/multiprocessing/spawn.py", line 125, in _main prepare(preparation_data) File "/Applications/Slicer.app/Contents/lib/Python/lib/python3.9/multiprocessing/spawn.py", line 236, in prepare _fixup_main_from_path(data['init_main_from_path']) File "/Applications/Slicer.app/Contents/lib/Python/lib/python3.9/multiprocessing/spawn.py", line 287, in _fixup_main_from_path main_content = runpy.run_path(main_path, File "/Applications/Slicer.app/Contents/lib/Python/lib/python3.9/runpy.py", line 268, in run_path Traceback (most recent call last): File "", line 1, in return _run_module_code(code, init_globals, run_name, File "/Applications/Slicer.app/Contents/lib/Python/lib/python3.9/runpy.py", line 97, in _run_module_code _run_code(code, mod_globals, init_globals, File "/Applications/Slicer.app/Contents/lib/Python/lib/python3.9/runpy.py", line 87, in _run_code File "/Applications/Slicer.app/Contents/lib/Python/lib/python3.9/multiprocessing/spawn.py", line 116, in spawn_main exec(code, run_globals) File "/Applications/Slicer.app/Contents/Extensions-31382/SlicerAutomatedDentalTools/lib/Slicer-5.2/cli-modules/AMASSS_CLI.py", line 71, in pip_uninstall('monai -q') File "/Applications/Slicer.app/Contents/bin/Python/slicer/util.py", line 3614, in pip_uninstall exitcode = _main(fd, parent_sentinel) File "/Applications/Slicer.app/Contents/lib/Python/lib/python3.9/multiprocessing/spawn.py", line 125, in _main prepare(preparation_data) File "/Applications/Slicer.app/Contents/lib/Python/lib/python3.9/multiprocessing/spawn.py", line 236, in prepare _fixup_main_from_path(data['init_main_from_path']) File "/Applications/Slicer.app/Contents/lib/Python/lib/python3.9/multiprocessing/spawn.py", line 287, in _fixup_main_from_path main_content = runpy.run_path(main_path, File "/Applications/Slicer.app/Contents/lib/Python/lib/python3.9/runpy.py", line 268, in run_path return _run_module_code(code, init_globals, run_name, File "/Applications/Slicer.app/Contents/lib/Python/lib/python3.9/runpy.py", line 97, in _run_module_code _run_code(code, mod_globals, init_globals, File "/Applications/Slicer.app/Contents/lib/Python/lib/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/Applications/Slicer.app/Contents/Extensions-31382/SlicerAutomatedDentalTools/lib/Slicer-5.2/cli-modules/AMASSS_CLI.py", line 71, in _executePythonModule('pip', args) File "/Applications/Slicer.app/Contents/bin/Python/slicer/util.py", line 3540, in _executePythonModule pip_uninstall('monai -q') File "/Applications/Slicer.app/Contents/bin/Python/slicer/util.py", line 3614, in pip_uninstall logProcessOutput(proc) File "/Applications/Slicer.app/Contents/bin/Python/slicer/util.py", line 3509, in logProcessOutput _executePythonModule('pip', args) File "/Applications/Slicer.app/Contents/bin/Python/slicer/util.py", line 3540, in _executePythonModule raise CalledProcessError(retcode, proc.args, output=proc.stdout, stderr=proc.stderr) logProcessOutput(proc) File "/Applications/Slicer.app/Contents/bin/Python/slicer/util.py", line 3509, in logProcessOutput subprocess.CalledProcessError: Command '['/Applications/Slicer.app/Contents/bin/PythonSlicer', '-m', 'pip', 'uninstall', 'monai', '-q', '--yes']' returned non-zero exit status 2. raise CalledProcessError(retcode, proc.args, output=proc.stdout, stderr=proc.stderr) subprocess.CalledProcessError: Command '['/Applications/Slicer.app/Contents/bin/PythonSlicer', '-m', 'pip', 'uninstall', 'monai', '-q', '--yes']' returned non-zero exit status 2. Traceback (most recent call last): File "", line 1, in File "/Applications/Slicer.app/Contents/lib/Python/lib/python3.9/multiprocessing/spawn.py", line 116, in spawn_main exitcode = _main(fd, parent_sentinel) File "/Applications/Slicer.app/Contents/lib/Python/lib/python3.9/multiprocessing/spawn.py", line 125, in _main prepare(preparation_data) File "/Applications/Slicer.app/Contents/lib/Python/lib/python3.9/multiprocessing/spawn.py", line 236, in prepare _fixup_main_from_path(data['init_main_from_path']) File "/Applications/Slicer.app/Contents/lib/Python/lib/python3.9/multiprocessing/spawn.py", line 287, in _fixup_main_from_path main_content = runpy.run_path(main_path, File "/Applications/Slicer.app/Contents/lib/Python/lib/python3.9/runpy.py", line 268, in run_path return _run_module_code(code, init_globals, run_name, File "/Applications/Slicer.app/Contents/lib/Python/lib/python3.9/runpy.py", line 97, in _run_module_code _run_code(code, mod_globals, init_globals, File "/Applications/Slicer.app/Contents/lib/Python/lib/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/Applications/Slicer.app/Contents/Extensions-31382/SlicerAutomatedDentalTools/lib/Slicer-5.2/cli-modules/AMASSS_CLI.py", line 71, in pip_uninstall('monai -q') File "/Applications/Slicer.app/Contents/bin/Python/slicer/util.py", line 3614, in pip_uninstall _executePythonModule('pip', args) File "/Applications/Slicer.app/Contents/bin/Python/slicer/util.py", line 3540, in _executePythonModule logProcessOutput(proc) File "/Applications/Slicer.app/Contents/bin/Python/slicer/util.py", line 3509, in logProcessOutput raise CalledProcessError(retcode, proc.args, output=proc.stdout, stderr=proc.stderr) subprocess.CalledProcessError: Command '['/Applications/Slicer.app/Contents/bin/PythonSlicer', '-m', 'pip', 'uninstall', 'monai', '-q', '--yes']' returned non-zero exit status 2. Traceback (most recent call last): File "/Applications/Slicer.app/Contents/lib/Python/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1132, in _try_get_data data = self._data_queue.get(timeout=timeout) File "/Applications/Slicer.app/Contents/lib/Python/lib/python3.9/multiprocessing/queues.py", line 113, in get if not self._poll(timeout): File "/Applications/Slicer.app/Contents/lib/Python/lib/python3.9/multiprocessing/connection.py", line 262, in poll return self._poll(timeout) File "/Applications/Slicer.app/Contents/lib/Python/lib/python3.9/multiprocessing/connection.py", line 429, in _poll r = wait([self], timeout) File "/Applications/Slicer.app/Contents/lib/Python/lib/python3.9/multiprocessing/connection.py", line 936, in wait ready = selector.select(timeout) File "/Applications/Slicer.app/Contents/lib/Python/lib/python3.9/selectors.py", line 416, in select fd_event_list = self._selector.poll(timeout) File "/Applications/Slicer.app/Contents/lib/Python/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler _error_if_any_worker_fails() RuntimeError: DataLoader worker (pid 51709) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/Applications/Slicer.app/Contents/Extensions-31382/SlicerAutomatedDentalTools/lib/Slicer-5.2/cli-modules/AMASSS_CLI.py", line 1228, in main(args) File "/Applications/Slicer.app/Contents/Extensions-31382/SlicerAutomatedDentalTools/lib/Slicer-5.2/cli-modules/AMASSS_CLI.py", line 914, in main for step, batch in enumerate(pred_loader): File "/Applications/Slicer.app/Contents/lib/Python/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 633, in next data = self._next_data() File "/Applications/Slicer.app/Contents/lib/Python/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1328, in _next_data idx, data = self._get_data() File "/Applications/Slicer.app/Contents/lib/Python/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1294, in _get_data success, data = self._try_get_data() File "/Applications/Slicer.app/Contents/lib/Python/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1145, in _try_get_data raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) from e RuntimeError: DataLoader worker (pid(s) 51709, 51710, 51711, 51713) exited unexpectedly

Same version of monai print(monai.version) 0.7.0

lucanchling commented 1 year ago

So the issue was due to the number of CPU workers selected as input. After multiple checkings, the computational time isn't impacted by only using one. The #36 PR has been created to fix this!

juangmagic commented 1 year ago

Awesome, using just one CPU unit solved the problem.... thank you very much.!