search

joblib / loky

Robust and reusable Executor for joblib
http://loky.readthedocs.io/en/stable/
BSD 3-Clause "New" or "Revised" License
530 stars 45 forks source link

Random deadlock in test_in_callback_submit_with_crash[func15-args15-UnpicklingError] at executor shutdown during test teardown #92

Closed ogrisel closed 6 years ago

ogrisel commented 7 years ago

Observed on Python 2.7 (32 bit) under Windows (on appveyor):

tests/test_reusable_executor.py::TestExecutorDeadLock::test_in_callback_submit_with_crash[func15-args15-UnpicklingError] PASSED
tests/test_reusable_executor.py::TestExecutorDeadLock::test_callback_crash_on_submit PASSED
tests/test_reusable_executor.py::TestExecutorDeadLock::test_deadlock_kill PASSED
tests/test_reusable_executor.py::TestExecutorDeadLock::test_crash_races[1] PASSED
tests/test_reusable_executor.py::TestExecutorDeadLock::test_crash_races[2] PASSED
+++++++++++++++++++++++++++++++++++ Timeout ++++++++++++++++++++++++++++++++++++
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Captured stderr ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[DEBUG:MainProcess:QueueManager] queue management thread shutting down
[DEBUG:MainProcess:QueueManager] closing call_queue
[DEBUG:MainProcess:QueueManager] telling queue thread to quit
[DEBUG:MainProcess:QueueManager] joining processes
[DEBUG:MainProcess:MainThread] Creating a new executor with max_workers=2 as the previous instance cannot be reused (broken).
[DEBUG:MainProcess:QueueFeederThread] feeder thread got sentinel -- exiting
[DEBUG:MainProcess:QueueManager] queue management thread clean shutdown of worker processes: []
[DEBUG:MainProcess:MainThread] shutting down executor <loky.reusable_executor.ReusablePoolExecutor object at 0x03E377B0>
[DEBUG:MainProcess:ThreadManager] shutting down
~~~~~~~~~~~~~~~~~~~~~~~~~~ Stack of Thread-29 (2760) ~~~~~~~~~~~~~~~~~~~~~~~~~~~
  File "C:\Python27\Lib\threading.py", line 774, in __bootstrap
    self.__bootstrap_inner()
  File "C:\Python27\Lib\threading.py", line 801, in __bootstrap_inner
    self.run()
  File "C:\Python27\Lib\threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "C:\Python27\Lib\multiprocessing\reduction.py", line 124, in _serve
    conn = _listener.accept()
  File "C:\Python27\Lib\multiprocessing\connection.py", line 145, in accept
    c = self._listener.accept()
  File "C:\Python27\Lib\multiprocessing\connection.py", line 365, in accept
    win32.ConnectNamedPipe(handle, win32.NULL)
~~~~~~~~~~~~~~~~~~~~~~~~~~ Stack of MainThread (2136) ~~~~~~~~~~~~~~~~~~~~~~~~~~
  File "C:\Python27\Lib\runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "C:\Python27\Lib\runpy.py", line 72, in _run_code
    exec code in run_globals
  File "C:\projects\loky\.tox\py27\Scripts\py.test.EXE\__main__.py", line 9, in <module>
    sys.exit(main())
  File "c:\projects\loky\.tox\py27\lib\site-packages\_pytest\config.py", line 58, in main
    return config.hook.pytest_cmdline_main(config=config)
  File "c:\projects\loky\.tox\py27\lib\site-packages\_pytest\vendored_packages\pluggy.py", line 745, in __call__
    return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
  File "c:\projects\loky\.tox\py27\lib\site-packages\_pytest\vendored_packages\pluggy.py", line 339, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "c:\projects\loky\.tox\py27\lib\site-packages\_pytest\vendored_packages\pluggy.py", line 334, in <lambda>
    _MultiCall(methods, kwargs, hook.spec_opts).execute()
  File "c:\projects\loky\.tox\py27\lib\site-packages\_pytest\vendored_packages\pluggy.py", line 614, in execute
    res = hook_impl.function(*args)
  File "c:\projects\loky\.tox\py27\lib\site-packages\_pytest\main.py", line 139, in pytest_cmdline_main
    return wrap_session(config, _main)
  File "c:\projects\loky\.tox\py27\lib\site-packages\_pytest\main.py", line 110, in wrap_session
    session.exitstatus = doit(config, session) or 0
  File "c:\projects\loky\.tox\py27\lib\site-packages\_pytest\main.py", line 146, in _main
    config.hook.pytest_runtestloop(session=session)
  File "c:\projects\loky\.tox\py27\lib\site-packages\_pytest\vendored_packages\pluggy.py", line 745, in __call__
    return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
  File "c:\projects\loky\.tox\py27\lib\site-packages\_pytest\vendored_packages\pluggy.py", line 339, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "c:\projects\loky\.tox\py27\lib\site-packages\_pytest\vendored_packages\pluggy.py", line 334, in <lambda>
    _MultiCall(methods, kwargs, hook.spec_opts).execute()
  File "c:\projects\loky\.tox\py27\lib\site-packages\_pytest\vendored_packages\pluggy.py", line 614, in execute
    res = hook_impl.function(*args)
  File "c:\projects\loky\.tox\py27\lib\site-packages\_pytest\main.py", line 169, in pytest_runtestloop
    item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
  File "c:\projects\loky\.tox\py27\lib\site-packages\_pytest\vendored_packages\pluggy.py", line 745, in __call__
    return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
  File "c:\projects\loky\.tox\py27\lib\site-packages\_pytest\vendored_packages\pluggy.py", line 339, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "c:\projects\loky\.tox\py27\lib\site-packages\_pytest\vendored_packages\pluggy.py", line 334, in <lambda>
    _MultiCall(methods, kwargs, hook.spec_opts).execute()
  File "c:\projects\loky\.tox\py27\lib\site-packages\_pytest\vendored_packages\pluggy.py", line 613, in execute
    return _wrapped_call(hook_impl.function(*args), self.execute)
  File "c:\projects\loky\.tox\py27\lib\site-packages\_pytest\vendored_packages\pluggy.py", line 248, in _wrapped_call
    call_outcome = _CallOutcome(func)
  File "c:\projects\loky\.tox\py27\lib\site-packages\_pytest\vendored_packages\pluggy.py", line 265, in __init__
    self.result = func()
  File "c:\projects\loky\.tox\py27\lib\site-packages\_pytest\vendored_packages\pluggy.py", line 613, in execute
    return _wrapped_call(hook_impl.function(*args), self.execute)
  File "c:\projects\loky\.tox\py27\lib\site-packages\_pytest\vendored_packages\pluggy.py", line 248, in _wrapped_call
    call_outcome = _CallOutcome(func)
  File "c:\projects\loky\.tox\py27\lib\site-packages\_pytest\vendored_packages\pluggy.py", line 265, in __init__
    self.result = func()
  File "c:\projects\loky\.tox\py27\lib\site-packages\_pytest\vendored_packages\pluggy.py", line 613, in execute
    return _wrapped_call(hook_impl.function(*args), self.execute)
  File "c:\projects\loky\.tox\py27\lib\site-packages\_pytest\vendored_packages\pluggy.py", line 248, in _wrapped_call
    call_outcome = _CallOutcome(func)
  File "c:\projects\loky\.tox\py27\lib\site-packages\_pytest\vendored_packages\pluggy.py", line 265, in __init__
    self.result = func()
  File "c:\projects\loky\.tox\py27\lib\site-packages\_pytest\vendored_packages\pluggy.py", line 614, in execute
    res = hook_impl.function(*args)
  File "c:\projects\loky\.tox\py27\lib\site-packages\_pytest\runner.py", line 67, in pytest_runtest_protocol
    runtestprotocol(item, nextitem=nextitem)
  File "c:\projects\loky\.tox\py27\lib\site-packages\_pytest\runner.py", line 83, in runtestprotocol
    nextitem=nextitem))
  File "c:\projects\loky\.tox\py27\lib\site-packages\_pytest\runner.py", line 157, in call_and_report
    call = call_runtest_hook(item, when, **kwds)
  File "c:\projects\loky\.tox\py27\lib\site-packages\_pytest\runner.py", line 177, in call_runtest_hook
    return CallInfo(lambda: ihook(item=item, **kwds), when=when)
  File "c:\projects\loky\.tox\py27\lib\site-packages\_pytest\runner.py", line 191, in __init__
    self.result = func()
  File "c:\projects\loky\.tox\py27\lib\site-packages\_pytest\runner.py", line 177, in <lambda>
    return CallInfo(lambda: ihook(item=item, **kwds), when=when)
  File "c:\projects\loky\.tox\py27\lib\site-packages\_pytest\vendored_packages\pluggy.py", line 745, in __call__
    return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
  File "c:\projects\loky\.tox\py27\lib\site-packages\_pytest\vendored_packages\pluggy.py", line 339, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "c:\projects\loky\.tox\py27\lib\site-packages\_pytest\vendored_packages\pluggy.py", line 334, in <lambda>
    _MultiCall(methods, kwargs, hook.spec_opts).execute()
  File "c:\projects\loky\.tox\py27\lib\site-packages\_pytest\vendored_packages\pluggy.py", line 613, in execute
    return _wrapped_call(hook_impl.function(*args), self.execute)
  File "c:\projects\loky\.tox\py27\lib\site-packages\_pytest\vendored_packages\pluggy.py", line 248, in _wrapped_call
    call_outcome = _CallOutcome(func)
  File "c:\projects\loky\.tox\py27\lib\site-packages\_pytest\vendored_packages\pluggy.py", line 265, in __init__
    self.result = func()
  File "c:\projects\loky\.tox\py27\lib\site-packages\_pytest\vendored_packages\pluggy.py", line 614, in execute
    res = hook_impl.function(*args)
  File "c:\projects\loky\.tox\py27\lib\site-packages\_pytest\runner.py", line 125, in pytest_runtest_teardown
    item.session._setupstate.teardown_exact(item, nextitem)
  File "c:\projects\loky\.tox\py27\lib\site-packages\_pytest\runner.py", line 474, in teardown_exact
    self._teardown_towards(needed_collectors)
  File "c:\projects\loky\.tox\py27\lib\site-packages\_pytest\runner.py", line 480, in _teardown_towards
    self._pop_and_teardown()
  File "c:\projects\loky\.tox\py27\lib\site-packages\_pytest\runner.py", line 440, in _pop_and_teardown
    self._teardown_with_finalization(colitem)
  File "c:\projects\loky\.tox\py27\lib\site-packages\_pytest\runner.py", line 458, in _teardown_with_finalization
    self._callfinalizers(colitem)
  File "c:\projects\loky\.tox\py27\lib\site-packages\_pytest\runner.py", line 448, in _callfinalizers
    fin()
  File "c:\projects\loky\.tox\py27\lib\site-packages\_pytest\python.py", line 465, in <lambda>
    return lambda: result(param_obj)
  File "C:\projects\loky\tests\_executor_mixin.py", line 181, in teardown_method
    executor = get_reusable_executor(max_workers=2)
  File "C:\projects\loky\loky\reusable_executor.py", line 106, in get_reusable_executor
    executor.shutdown(wait=True, kill_workers=kill_workers)
  File "C:\projects\loky\loky\process_executor.py", line 978, in shutdown
    self._queue_management_thread.join()
  File "C:\Python27\Lib\threading.py", line 940, in join
    self.__block.wait()
  File "C:\Python27\Lib\threading.py", line 340, in wait
    waiter.acquire()
ogrisel commented 7 years ago

https://ci.appveyor.com/project/tomMoral/loky/build/1.0.610/job/4o750xoparelm3vu

tomMoral commented 6 years ago

This can be closed as the structure of the ProcessPoolExecutor has changed (there is no ThreadManager anymore). Let's reopen it if it occurs again.