search

kevlened / pytest-parallel

A pytest plugin for parallel and concurrent testing
https://github.com/browsertron/pytest-parallel/issues/104#issuecomment-1293941066
MIT License
313 stars 60 forks source link

AttributeError: Can't pickle local object when Running with Python 3.8 #41

Closed laike9m closed 4 years ago

laike9m commented 4 years ago

To reproduce, first have a Python 3.8 enviroment, then

pip install pytest pytest-parallel
touch test.py
pytest --workers 10 test.py

will give

platform darwin -- Python 3.8.0, pytest-5.2.1, py-1.8.0, pluggy-0.13.0
rootdir: /Users/laike9m/Dev/Python/TestPython
plugins: parallel-0.0.9
collected 0 items
pytest-parallel: 10 workers (processes), 1 test per worker (thread)
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/Users/laike9m/.pyenv/versions/3.8.0/lib/python3.8/site-packages/_pytest/main.py", line 191, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/Users/laike9m/.pyenv/versions/3.8.0/lib/python3.8/site-packages/_pytest/main.py", line 235, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/Users/laike9m/.pyenv/versions/3.8.0/lib/python3.8/site-packages/pluggy/hooks.py", line 286, in __call__
INTERNALERROR>     return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR>   File "/Users/laike9m/.pyenv/versions/3.8.0/lib/python3.8/site-packages/pluggy/manager.py", line 92, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/Users/laike9m/.pyenv/versions/3.8.0/lib/python3.8/site-packages/pluggy/manager.py", line 83, in <lambda>
INTERNALERROR>     self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
INTERNALERROR>   File "/Users/laike9m/.pyenv/versions/3.8.0/lib/python3.8/site-packages/pluggy/callers.py", line 208, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/Users/laike9m/.pyenv/versions/3.8.0/lib/python3.8/site-packages/pluggy/callers.py", line 80, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/Users/laike9m/.pyenv/versions/3.8.0/lib/python3.8/site-packages/pluggy/callers.py", line 187, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/Users/laike9m/.pyenv/versions/3.8.0/lib/python3.8/site-packages/pytest_parallel/__init__.py", line 342, in pytest_runtestloop
INTERNALERROR>     process.start()
INTERNALERROR>   File "/Users/laike9m/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/process.py", line 121, in start
INTERNALERROR>     self._popen = self._Popen(self)
INTERNALERROR>   File "/Users/laike9m/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
INTERNALERROR>     return _default_context.get_context().Process._Popen(process_obj)
INTERNALERROR>   File "/Users/laike9m/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/context.py", line 283, in _Popen
INTERNALERROR>     return Popen(process_obj)
INTERNALERROR>   File "/Users/laike9m/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
INTERNALERROR>     super().__init__(process_obj)
INTERNALERROR>   File "/Users/laike9m/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
INTERNALERROR>     self._launch(process_obj)
INTERNALERROR>   File "/Users/laike9m/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
INTERNALERROR>     reduction.dump(process_obj, fp)
INTERNALERROR>   File "/Users/laike9m/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
INTERNALERROR>     ForkingPickler(file, protocol).dump(obj)
INTERNALERROR> AttributeError: Can't pickle local object 'ArgumentParser.__init__.<locals>.identity'
laike9m commented 4 years ago

Ping?

laike9m commented 4 years ago

Related https://bugs.python.org/issue33884

francbartoli commented 4 years ago

Confirm the issue also with pytest-parallel==0.0.10 but exclusively on Darwin. It is still safe on linux-based OS

max-block commented 4 years ago

I have the same problem on this platform: platform darwin -- Python 3.8.1, pytest-5.3.5, py-1.8.1, pluggy-0.13.1

blueyed commented 4 years ago

From https://docs.python.org/3/library/multiprocessing.html:

Changed in version 3.8: On macOS, the spawn start method is now the default. The fork start method should be considered unsafe as it can lead to crashes of the subprocess. See bpo-33725.

So you could set it manually to "forkserver" I guess.

import multiprocessing as mp; mp.set_start_method("forkserver")

Apart from that pytest-xdist might be a good alternative for executing tests in parallel.

kevlened commented 4 years ago

I force fork on macOS in 0.1.0, which fixes the problem. This sacrifices safety to keep usability, so I've added a notice to the README.