mowshon / bounded_pool_executor

Bounded Process&Thread Pool Executor
MIT License
61 stars 5 forks source link

Is windows not supported ? #5

Open KTN1990 opened 4 years ago

KTN1990 commented 4 years ago

i got this error when i run the test script :

`Traceback (most recent call last): File "", line 1, in File "C:\Program Files\Python37\lib\multiprocessing\spawn.py", line 105, in spawn_main exitcode = _main(fd)

0 Worker initialization

Traceback (most recent call last): File "", line 1, in File "C:\Program Files\Python37\lib\multiprocessing\spawn.py", line 105, in spawn_main exitcode = _main(fd) File "C:\Program Files\Python37\lib\multiprocessing\spawn.py", line 114, in _main prepare(preparation_data) File "C:\Program Files\Python37\lib\multiprocessing\spawn.py", line 225, in prepare _fixup_main_from_path(data['init_main_from_path']) File "C:\Program Files\Python37\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path run_name="_mpmain") File "C:\Program Files\Python37\lib\runpy.py", line 263, in run_path pkg_name=pkg_name, script_name=fname) File "C:\Program Files\Python37\lib\runpy.py", line 96, in _run_module_code

0 Worker initialization

Traceback (most recent call last): File "", line 1, in File "C:\Program Files\Python37\lib\multiprocessing\spawn.py", line 105, in spawn_main exitcode = _main(fd) File "C:\Program Files\Python37\lib\multiprocessing\spawn.py", line 114, in _main prepare(preparation_data) File "C:\Program Files\Python37\lib\multiprocessing\spawn.py", line 225, in prepare _fixup_main_from_path(data['init_main_from_path']) File "C:\Program Files\Python37\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path run_name="_mpmain") File "C:\Program Files\Python37\lib\runpy.py", line 263, in run_path pkg_name=pkg_name, script_name=fname) File "C:\Program Files\Python37\lib\runpy.py", line 96, in _run_module_code mod_name, mod_spec, pkg_name, script_name) File "C:\Program Files\Python37\lib\runpy.py", line 85, in _run_code exec(code, run_globals) File "C:\Users\Administrator\Downloads\Compressed\ProExp\0\jomla\test.py", line 13, in worker.submit(do_job, num) File "C:\Program Files\Python37\lib\site-packages\bounded_pool_executor__init__.py", line 20, in submit future = super().submit(fn, *args, **kwargs) File "C:\Program Files\Python37\lib\concurrent\futures\process.py", line 641, in submit self._start_queue_management_thread() File "C:\Program Files\Python37\lib\concurrent\futures\process.py", line 583, in _start_queue_management_thread self._adjust_process_count() File "C:\Program Files\Python37\lib\concurrent\futures\process.py", line 607, in _adjust_process_count p.start() File "C:\Program Files\Python37\lib\multiprocessing\process.py", line 112, in start self._popen = self._Popen(self) File "C:\Program Files\Python37\lib\multiprocessing\context.py", line 322, in _Popen return Popen(process_obj) File "C:\Program Files\Python37\lib\multiprocessing\popen_spawn_win32.py", line 46, in init Traceback (most recent call last): File "", line 1, in mod_name, mod_spec, pkg_name, script_name)

0 Worker initialization

prep_data = spawn.get_preparation_data(process_obj._name)

File "C:\Program Files\Python37\lib\multiprocessing\spawn.py", line 143, in get_preparation_data Traceback (most recent call last): File "", line 1, in File "C:\Program Files\Python37\lib\multiprocessing\spawn.py", line 114, in _main File "C:\Program Files\Python37\lib\multiprocessing\spawn.py", line 105, in spawn_main _check_not_importing_main() File "C:\Program Files\Python37\lib\multiprocessing\spawn.py", line 136, in _check_not_importing_main File "C:\Program Files\Python37\lib\multiprocessing\spawn.py", line 105, in spawn_main prepare(preparation_data) File "C:\Program Files\Python37\lib\multiprocessing\spawn.py", line 225, in prepare File "C:\Program Files\Python37\lib\runpy.py", line 85, in _run_code is not going to be frozen to produce an executable.''') RuntimeError: An attempt has been made to start a new process before the current process has finished its bootstrapping phase.

    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:

        if _name_ == '_main_':
            freeze_support()
            ...

    The "freeze_support()" line can be omitted if the program
    is not going to be frozen to produce an executable.
_fixup_main_from_path(data['init_main_from_path'])

File "C:\Program Files\Python37\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path run_name="_mpmain") File "C:\Program Files\Python37\lib\runpy.py", line 263, in run_path pkg_name=pkg_name, script_name=fname) File "C:\Program Files\Python37\lib\runpy.py", line 96, in _run_module_code mod_name, mod_spec, pkg_name, script_name) File "C:\Program Files\Python37\lib\runpy.py", line 85, in _run_code exec(code, run_globals) File "C:\Users\Administrator\Downloads\Compressed\ProExp\0\jomla\test.py", line 13, in worker.submit(do_job, num) File "C:\Program Files\Python37\lib\site-packages\bounded_pool_executor__init__.py", line 20, in submit future = super().submit(fn, *args, *kwargs) File "C:\Program Files\Python37\lib\concurrent\futures\process.py", line 641, in submit self._start_queue_management_thread() File "C:\Program Files\Python37\lib\concurrent\futures\process.py", line 583, in _start_queue_management_thread self._adjust_process_count() File "C:\Program Files\Python37\lib\concurrent\futures\process.py", line 607, in _adjust_process_count exitcode = _main(fd) File "C:\Program Files\Python37\lib\multiprocessing\spawn.py", line 114, in _main prepare(preparation_data) File "C:\Program Files\Python37\lib\multiprocessing\spawn.py", line 225, in prepare _fixup_main_from_path(data['init_main_from_path']) File "C:\Program Files\Python37\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path run_name="_mpmain") File "C:\Program Files\Python37\lib\runpy.py", line 263, in run_path pkg_name=pkg_name, script_name=fname) File "C:\Program Files\Python37\lib\runpy.py", line 96, in _run_module_code mod_name, mod_spec, pkg_name, script_name) File "C:\Program Files\Python37\lib\runpy.py", line 85, in _run_code exec(code, run_globals) File "C:\Users\Administrator\Downloads\Compressed\ProExp\0\jomla\test.py", line 13, in worker.submit(do_job, num) File "C:\Program Files\Python37\lib\site-packages\bounded_pool_executor__init.py", line 20, in submit exec(code, run_globals) Traceback (most recent call last): File "test.py", line 13, in worker.submit(do_job, num) File "C:\Program Files\Python37\lib\site-packages\bounded_pool_executor\init__.py", line 20, in submit future = super().submit(fn, args, **kwargs) File "C:\Program Files\Python37\lib\concurrent\futures\process.py", line 625, in submit raise BrokenProcessPool(self._broken) concurrent.futures.process.BrokenProcessPool: A child process terminated abruptly, the process pool is not usable anymore`

mowshon commented 4 years ago

Is this error still persisting? Unfortunately, I do not have Windows to test this error. Have you tried running the code from the example?

AndrewLt commented 3 years ago

I have the same problem on Windows.

GabrielBigardi commented 3 years ago

Hi, i was facing the same problem on Windows, the fix is pretty simple, instead of running the code directly you need to run it this way:

if __name__ == '__main__':
    #Code Here

So the example on the main page would be:

from bounded_pool_executor import BoundedProcessPoolExecutor
from time import sleep
from random import randint

def do_job(num):
    sleep_sec = randint(1, 10)
    print('value: %d, sleep: %d sec.' % (num, sleep_sec))
    sleep(sleep_sec)

if __name__ == '__main__':
    with BoundedProcessPoolExecutor(max_workers=5) as worker:
        for num in range(10000):
            print('#%d Worker initialization' % num)
            worker.submit(do_job, num)

I will make a pull request to add it on the readme

GunGunGun commented 2 years ago

This is a Window-specific's multiprocess problem, because Windows doesn't support os.fork, and os.spawn is a pretty bad alternative so multiprocessing usually doesn't work as good as on Linux.