gwm17
commented
1 month ago Unfortunately, the os.system("taskset...") call will only work in linux. On any other platform it may throw an error as the taskset command most likely doesn't exist. So we can't just put this in the default example. However, we can add it to the FAQ as a solution to a commonly asked question!
Utilizing multiprocessing with scipy sometimes returns an error (running on Debian 4.19.304-1) that is caused by the OpenBLAS library used by scipy's linalg libraries. Importing these libraries changes the CPU affinity which affects the parent process's ability to interact with its child cores. More on this issue can be found at: https://stackoverflow.com/questions/23537716/importing-scipy-breaks-multiprocessing-support-in-python
Error Message:
OpenBLAS blas_thread_init: pthread_create failed for thread 63 of 64: Resource temporarily unavailable OpenBLAS blas_thread_init: RLIMIT_NPROC 512 current, 512 max Traceback (most recent call last)::00, ?it/s] File "", line 1, in ?it/s]
File "/user/turi/.conda/envs/spyral-env/lib/python3.11/multiprocessing/spawn.py", line 122, in spawn_main
exitcode = _main(fd, parent_sentinel)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/user/turi/.conda/envs/spyral-env/lib/python3.11/multiprocessing/spawn.py", line 131, in _main
prepare(preparation_data)
File "/user/turi/.conda/envs/spyral-env/lib/python3.11/multiprocessing/spawn.py", line 246, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/user/turi/.conda/envs/spyral-env/lib/python3.11/multiprocessing/spawn.py", line 297, in _fixup_main_from_path
main_content = runpy.run_path(main_path,
^^^^^^^^^^^^^^^^^^^^^^^^^
File "", line 291, in run_path
File "", line 98, in _run_module_code
File "", line 88, in _run_code
File "/user/turi/Analysis/a2091/RunScript.py", line 1, in
from spyral import (
File "/user/turi/.conda/envs/spyral-env/lib/python3.11/site-packages/spyral/init.py", line 20, in
from .phases.pointcloud_phase import PointcloudPhase
File "/user/turi/.conda/envs/spyral-env/lib/python3.11/site-packages/spyral/phases/pointcloud_phase.py", line 17, in
from ..trace.get_event import GetEvent
File "/user/turi/.conda/envs/spyral-env/lib/python3.11/site-packages/spyral/trace/get_event.py", line 1, in
from .get_trace import GetTrace
File "/user/turi/.conda/envs/spyral-env/lib/python3.11/site-packages/spyral/trace/get_trace.py", line 6, in
from scipy import signal
File "", line 1229, in _handle_fromlist
File "/user/turi/.conda/envs/spyral-env/lib/python3.11/site-packages/scipy/init.py", line 134, in getattr
return _importlib.import_module(f'scipy.{name}')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/user/turi/.conda/envs/spyral-env/lib/python3.11/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/user/turi/.conda/envs/spyral-env/lib/python3.11/site-packages/scipy/signal/init.py", line 315, in
from . import _sigtools, windows
File "/user/turi/.conda/envs/spyral-env/lib/python3.11/site-packages/scipy/signal/windows/init.py", line 42, in
from ._windows import
File "/user/turi/.conda/envs/spyral-env/lib/python3.11/site-packages/scipy/signal/windows/_windows.py", line 7, in
from scipy import linalg, special, fft as sp_fft
File "", line 1229, in _handle_fromlist
File "/user/turi/.conda/envs/spyral-env/lib/python3.11/site-packages/scipy/init.py", line 134, in getattr
return _importlib.import_module(f'scipy.{name}')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/user/turi/.conda/envs/spyral-env/lib/python3.11/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/user/turi/.conda/envs/spyral-env/lib/python3.11/site-packages/scipy/linalg/init.py", line 206, in
from ._misc import
File "/user/turi/.conda/envs/spyral-env/lib/python3.11/site-packages/scipy/linalg/_misc.py", line 3, in
from .blas import get_blas_funcs
File "/user/turi/.conda/envs/spyral-env/lib/python3.11/site-packages/scipy/linalg/blas.py", line 213, in
from scipy.linalg import _fblas
FIX:
This can be fixed by explicitly defining the CPU affinity at the top of the main file. In the main file only one new line needs to be included.
import os
if name == 'main': os.system('taskset -cp 0-%d %s' % (n_processes, os.getpid()))