kevlened / pytest-parallel

A pytest plugin for parallel and concurrent testing
https://github.com/browsertron/pytest-parallel/issues/104#issuecomment-1293941066
MIT License
313 stars 59 forks source link

Fatal Python error: _enter_buffered_busy: could not acquire lock for <_io.BufferedWriter name=5> at interpreter shutdown, possibly due to daemon threads #114

Open omerXfaruq opened 2 years ago

omerXfaruq commented 2 years ago

Hello, while trying this library on Gradio project, I encountered this error, will share the reproduction below.

commit 98242fe3632c20511300ac63b774290e4fdf8313

➜  gradio git:(queue-refactor-backend) ✗ pytest --tests-per-worker 5 test/test_event_queue.py
================================================================================================================== test session starts ===================================================================================================================
platform win32 -- Python 3.9.10, pytest-7.0.0, pluggy-1.0.0
rootdir: F:\SecondaryDownloads\git_repos\gradio
plugins: anyio-3.5.0, asyncio-0.18.3, cov-3.0.0, parallel-0.1.1
asyncio: mode=legacy
collected 1 item                                                                                                                                                                                                                                          
pytest-parallel: 1 worker (process), 1 test per worker (thread)
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "f:\secondarydownloads\git_repos\gradio\venv\lib\site-packages\_pytest\main.py", line 268, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "f:\secondarydownloads\git_repos\gradio\venv\lib\site-packages\_pytest\main.py", line 322, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "f:\secondarydownloads\git_repos\gradio\venv\lib\site-packages\pluggy\_hooks.py", line 265, in __call__
INTERNALERROR>     return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
INTERNALERROR>   File "f:\secondarydownloads\git_repos\gradio\venv\lib\site-packages\pluggy\_manager.py", line 80, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "f:\secondarydownloads\git_repos\gradio\venv\lib\site-packages\pluggy\_callers.py", line 60, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "f:\secondarydownloads\git_repos\gradio\venv\lib\site-packages\pluggy\_result.py", line 60, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "f:\secondarydownloads\git_repos\gradio\venv\lib\site-packages\pluggy\_callers.py", line 39, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "f:\secondarydownloads\git_repos\gradio\venv\lib\site-packages\pytest_parallel\__init__.py", line 313, in pytest_runtestloop
INTERNALERROR>     process.start()
INTERNALERROR>   File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.2800.0_x64__qbz5n2kfra8p0\lib\multiprocessing\process.py", line 121, in start
INTERNALERROR>     self._popen = self._Popen(self)
INTERNALERROR>   File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.2800.0_x64__qbz5n2kfra8p0\lib\multiprocessing\context.py", line 224, in _Popen
INTERNALERROR>     return _default_context.get_context().Process._Popen(process_obj)
INTERNALERROR>   File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.2800.0_x64__qbz5n2kfra8p0\lib\multiprocessing\context.py", line 327, in _Popen
INTERNALERROR>     return Popen(process_obj)
INTERNALERROR>   File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.2800.0_x64__qbz5n2kfra8p0\lib\multiprocessing\popen_spawn_win32.py", line 93, in __init__
INTERNALERROR>     reduction.dump(process_obj, to_child)
INTERNALERROR>   File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.2800.0_x64__qbz5n2kfra8p0\lib\multiprocessing\reduction.py", line 60, in dump
INTERNALERROR>     ForkingPickler(file, protocol).dump(obj)
INTERNALERROR> AttributeError: Can't pickle local object 'ArgumentParser.__init__.<locals>.identity'

================================================================================================================== 2 warnings in 2.62s =================================================================================================================== 
Exception in thread Thread-1:
Traceback (most recent call last):
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.2800.0_x64__qbz5n2kfra8p0\lib\multiprocessing\connection.py", line 317, in _recv_bytes
    nread, err = ov.GetOverlappedResult(True)
BrokenPipeError: [WinError 109] The pipe has been ended

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.2800.0_x64__qbz5n2kfra8p0\lib\threading.py", line 973, in _bootstrap_inner
    self.run()
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.2800.0_x64__qbz5n2kfra8p0\lib\threading.py", line 910, in run
    self._target(*self._args, **self._kwargs)
  File "f:\secondarydownloads\git_repos\gradio\venv\lib\site-packages\pytest_parallel\__init__.py", line 359, in process_responses
    event_name, kwargs = queue.get()
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.2800.0_x64__qbz5n2kfra8p0\lib\multiprocessing\spawn.py", line 107, in spawn_main
    new_handle = reduction.duplicate(pipe_handle,
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.2800.0_x64__qbz5n2kfra8p0\lib\multiprocessing\reduction.py", line 79, in duplicate
    return _winapi.DuplicateHandle(
OSError: [WinError 6] The handle is invalid
Fatal Python error: _enter_buffered_busy: could not acquire lock for <_io.BufferedWriter name=5> at interpreter shutdown, possibly due to daemon threads
Python runtime state: finalizing (tstate=0000022F10A55240)

Current thread 0x000049a4 (most recent call first):
<no Python frame>