Unable to run parallel tests with pytest-xdist - can't serialize <class 'pytest_bdd.parser.Step'>

[aypingchewsky]

I'm unable to run pytest-bdd (6.1.1) with pytest-xdist (3.5.0)

I'm getting the following error:

                    INTERNALERROR> def worker_internal_error(self, node, formatted_error):
INTERNALERROR>         """
INTERNALERROR>         pytest_internalerror() was called on the worker.
INTERNALERROR>     
INTERNALERROR>         pytest_internalerror() arguments are an excinfo and an excrepr, which can't
INTERNALERROR>         be serialized, so we go with a poor man's solution of raising an exception
INTERNALERROR>         here ourselves using the formatted message.
INTERNALERROR>         """
INTERNALERROR>         self._active_nodes.remove(node)
INTERNALERROR>         try:
INTERNALERROR> >           assert False, formatted_error
INTERNALERROR> E           AssertionError: Traceback (most recent call last):
INTERNALERROR> E               File "C:\Users\JSMITH\repos\project\venv\Lib\site-packages\execnet\gateway_base.py", line 1498, in _save
INTERNALERROR> E                 dispatch = self._dispatch[tp]
INTERNALERROR> E                            ~~~~~~~~~~~~~~^^^^
INTERNALERROR> E             KeyError: <class 'pytest_bdd.parser.Step'>
INTERNALERROR> E             
INTERNALERROR> E             During handling of the above exception, another exception occurred:
INTERNALERROR> E             
INTERNALERROR> E             Traceback (most recent call last):
INTERNALERROR> E               File "C:\Users\JSMITH\repos\project\venv\Lib\site-packages\_pytest\main.py", line 271, in wrap_session
INTERNALERROR> E                 session.exitstatus = doit(config, session) or 0
INTERNALERROR> E                                      ^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "C:\Users\JSMITH\repos\project\venv\Lib\site-packages\_pytest\main.py", line 325, in _main
INTERNALERROR> E                 config.hook.pytest_runtestloop(session=session)
INTERNALERROR> E               File "C:\Users\JSMITH\repos\project\venv\Lib\site-packages\pluggy\_hooks.py", line 501, in __call__
INTERNALERROR> E                 return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR> E                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "C:\Users\JSMITH\repos\project\venv\Lib\site-packages\pluggy\_manager.py", line 119, in _hookexec
INTERNALERROR> E                 return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR> E                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "C:\Users\JSMITH\repos\project\venv\Lib\site-packages\pluggy\_callers.py", line 181, in _multicall
INTERNALERROR> E                 return outcome.get_result()
INTERNALERROR> E                        ^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "C:\Users\JSMITH\repos\project\venv\Lib\site-packages\pluggy\_result.py", line 99, in get_result
INTERNALERROR> E                 raise exc.with_traceback(exc.__traceback__)
INTERNALERROR> E               File "C:\Users\JSMITH\repos\project\venv\Lib\site-packages\pluggy\_callers.py", line 102, in _multicall
INTERNALERROR> E                 res = hook_impl.function(*args)
INTERNALERROR> E                       ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "C:\Users\JSMITH\repos\project\venv\Lib\site-packages\xdist\remote.py", line 157, in pytest_runtestloop
INTERNALERROR> E                 self.run_one_test()
INTERNALERROR> E               File "C:\Users\JSMITH\repos\project\venv\Lib\site-packages\xdist\remote.py", line 174, in run_one_test
INTERNALERROR> E                 self.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
INTERNALERROR> E               File "C:\Users\JSMITH\repos\project\venv\Lib\site-packages\pluggy\_hooks.py", line 501, in __call__
INTERNALERROR> E                 return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR> E                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "C:\Users\JSMITH\repos\project\venv\Lib\site-packages\pluggy\_manager.py", line 119, in _hookexec
INTERNALERROR> E                 return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR> E                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "C:\Users\JSMITH\repos\project\venv\Lib\site-packages\pluggy\_callers.py", line 181, in _multicall
INTERNALERROR> E                 return outcome.get_result()
INTERNALERROR> E                        ^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "C:\Users\JSMITH\repos\project\venv\Lib\site-packages\pluggy\_result.py", line 99, in get_result
INTERNALERROR> E                 raise exc.with_traceback(exc.__traceback__)
INTERNALERROR> E               File "C:\Users\JSMITH\repos\project\venv\Lib\site-packages\pluggy\_callers.py", line 102, in _multicall
INTERNALERROR> E                 res = hook_impl.function(*args)
INTERNALERROR> E                       ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "C:\Users\JSMITH\repos\project\venv\Lib\site-packages\_pytest\runner.py", line 114, in pytest_runtest_protocol
INTERNALERROR> E                 runtestprotocol(item, nextitem=nextitem)
INTERNALERROR> E               File "C:\Users\JSMITH\repos\project\venv\Lib\site-packages\_pytest\runner.py", line 133, in runtestprotocol
INTERNALERROR> E                 reports.append(call_and_report(item, "call", log))
INTERNALERROR> E                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "C:\Users\JSMITH\repos\project\venv\Lib\site-packages\_pytest\runner.py", line 226, in call_and_report
INTERNALERROR> E                 hook.pytest_runtest_logreport(report=report)
INTERNALERROR> E               File "C:\Users\JSMITH\repos\project\venv\Lib\site-packages\pluggy\_hooks.py", line 501, in __call__
INTERNALERROR> E                 return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR> E                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "C:\Users\JSMITH\repos\project\venv\Lib\site-packages\pluggy\_manager.py", line 119, in _hookexec
INTERNALERROR> E                 return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR> E                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "C:\Users\JSMITH\repos\project\venv\Lib\site-packages\pluggy\_callers.py", line 138, in _multicall
INTERNALERROR> E                 raise exception.with_traceback(exception.__traceback__)
INTERNALERROR> E               File "C:\Users\JSMITH\repos\project\venv\Lib\site-packages\pluggy\_callers.py", line 102, in _multicall
INTERNALERROR> E                 res = hook_impl.function(*args)
INTERNALERROR> E                       ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "C:\Users\JSMITH\repos\project\venv\Lib\site-packages\xdist\remote.py", line 227, in pytest_runtest_logreport
INTERNALERROR> E                 self.sendevent("testreport", data=data)
INTERNALERROR> E               File "C:\Users\JSMITH\repos\project\venv\Lib\site-packages\xdist\remote.py", line 87, in sendevent
INTERNALERROR> E                 self.channel.send((name, kwargs))
INTERNALERROR> E               File "C:\Users\JSMITH\repos\project\venv\Lib\site-packages\execnet\gateway_base.py", line 811, in send
INTERNALERROR> E                 self.gateway._send(Message.CHANNEL_DATA, self.id, dumps_internal(item))
INTERNALERROR> E                                                                   ^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "C:\Users\JSMITH\repos\project\venv\Lib\site-packages\execnet\gateway_base.py", line 1469, in dumps_internal
INTERNALERROR> E                 return _Serializer().save(obj)
INTERNALERROR> E                        ^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "C:\Users\JSMITH\repos\project\venv\Lib\site-packages\execnet\gateway_base.py", line 1487, in save
INTERNALERROR> E                 self._save(obj)
INTERNALERROR> E               File "C:\Users\JSMITH\repos\project\venv\Lib\site-packages\execnet\gateway_base.py", line 1505, in _save
INTERNALERROR> E                 dispatch(self, obj)
INTERNALERROR> E               File "C:\Users\JSMITH\repos\project\venv\Lib\site-packages\execnet\gateway_base.py", line 1580, in save_tuple
INTERNALERROR> E                 self._save(item)
INTERNALERROR> E               File "C:\Users\JSMITH\repos\project\venv\Lib\site-packages\execnet\gateway_base.py", line 1505, in _save
INTERNALERROR> E                 dispatch(self, obj)
INTERNALERROR> E               File "C:\Users\JSMITH\repos\project\venv\Lib\site-packages\execnet\gateway_base.py", line 1576, in save_dict
INTERNALERROR> E                 self._write_setitem(key, value)
INTERNALERROR> E               File "C:\Users\JSMITH\repos\project\venv\Lib\site-packages\execnet\gateway_base.py", line 1570, in _write_setitem
INTERNALERROR> E                 self._save(value)
INTERNALERROR> E               File "C:\Users\JSMITH\repos\project\venv\Lib\site-packages\execnet\gateway_base.py", line 1505, in _save
INTERNALERROR> E                 dispatch(self, obj)
INTERNALERROR> E               File "C:\Users\JSMITH\repos\project\venv\Lib\site-packages\execnet\gateway_base.py", line 1576, in save_dict
INTERNALERROR> E                 self._write_setitem(key, value)
INTERNALERROR> E               File "C:\Users\JSMITH\repos\project\venv\Lib\site-packages\execnet\gateway_base.py", line 1570, in _write_setitem
INTERNALERROR> E                 self._save(value)
INTERNALERROR> E               File "C:\Users\JSMITH\repos\project\venv\Lib\site-packages\execnet\gateway_base.py", line 1505, in _save
INTERNALERROR> E                 dispatch(self, obj)
INTERNALERROR> E               File "C:\Users\JSMITH\repos\project\venv\Lib\site-packages\execnet\gateway_base.py", line 1566, in save_list
INTERNALERROR> E                 self._write_setitem(i, item)
INTERNALERROR> E               File "C:\Users\JSMITH\repos\project\venv\Lib\site-packages\execnet\gateway_base.py", line 1570, in _write_setitem
INTERNALERROR> E                 self._save(value)
INTERNALERROR> E               File "C:\Users\JSMITH\repos\project\venv\Lib\site-packages\execnet\gateway_base.py", line 1505, in _save
INTERNALERROR> E                 dispatch(self, obj)
INTERNALERROR> E               File "C:\Users\JSMITH\repos\project\venv\Lib\site-packages\execnet\gateway_base.py", line 1580, in save_tuple
INTERNALERROR> E                 self._save(item)
INTERNALERROR> E               File "C:\Users\JSMITH\repos\project\venv\Lib\site-packages\execnet\gateway_base.py", line 1505, in _save
INTERNALERROR> E                 dispatch(self, obj)
INTERNALERROR> E               File "C:\Users\JSMITH\repos\project\venv\Lib\site-packages\execnet\gateway_base.py", line 1576, in save_dict
INTERNALERROR> E                 self._write_setitem(key, value)
INTERNALERROR> E               File "C:\Users\JSMITH\repos\project\venv\Lib\site-packages\execnet\gateway_base.py", line 1570, in _write_setitem
INTERNALERROR> E                 self._save(value)
INTERNALERROR> E               File "C:\Users\JSMITH\repos\project\venv\Lib\site-packages\execnet\gateway_base.py", line 1505, in _save
INTERNALERROR> E                 dispatch(self, obj)
INTERNALERROR> E               File "C:\Users\JSMITH\repos\project\venv\Lib\site-packages\execnet\gateway_base.py", line 1566, in save_list
INTERNALERROR> E                 self._write_setitem(i, item)
INTERNALERROR> E               File "C:\Users\JSMITH\repos\project\venv\Lib\site-packages\execnet\gateway_base.py", line 1570, in _write_setitem
INTERNALERROR> E                 self._save(value)
INTERNALERROR> E               File "C:\Users\JSMITH\repos\project\venv\Lib\site-packages\execnet\gateway_base.py", line 1503, in _save
INTERNALERROR> E                 raise DumpError(f"can't serialize {tp}")
INTERNALERROR> E             execnet.gateway_base.DumpError: can't serialize <class 'pytest_bdd.parser.Step'>
INTERNALERROR> E           assert False
INTERNALERROR> 
INTERNALERROR> venv\Lib\site-packages\xdist\dsession.py:200: AssertionError

Example Scenario:

@demo
Scenario: Passing Demo
    When I'm on Google
    Then I can see the correct URL

#tests/step_defs/common/test_demo_steps.py

from pytest_bdd import scenarios, when, then

scenarios('../tests/features/demo/demo.feature')
@when("I'm on Google")
@logger_wrapper()
def visit_google_homepage():
    """test_demo_steps"""
    pass

@then("I can see the correct URL")
@logger_wrapper()
def verify_correct_url_google():
    """test_demo_steps"""
    pass

# conftest.py
import pytest

import tests.step_defs.common

def get_packages_in_module(m: ModuleType) -> Iterable[ModuleInfo]:
    return walk_packages(m.__path__, prefix=m.__name__ + '.')  # type: ignore

def get_package_paths_in_module(m: ModuleType) -> Iterable[str]:
    return [package.name for package in get_packages_in_module(m)]

pytest_plugins = [*get_package_paths_in_module(tests.step_defs.common)]

Command: pytest -m demo -n 2

This works normally if I run tests sequentially using pytest -m demo

I've tried the following:

• running with --dist=loadscope - same errors wrapped up in "RuntimeError: Unexpectedly no active workers available"
• running with -n auto - same errors
• Upgrading my poetry packages - same errors
• Moving all of my test step files to a common folder and including them in pytest_plugins in conftest.py
• Running with pytest-parallel (this project hasn't been updated in 3 years and I was getting different errors but I was still unable to get them to run , no surprise)
• Commented out the pytest_runtest_makereport hook in conftest.py
• Running a pytest test without BDD steps - this works perfectly and tests pass, which makes me think the issue is pytest-bdd related

Poetry dependencies:

[tool.poetry.dependencies]
python = ">=3.12,<4.0"
pytest = "^7.3.1"
selenium = "^4.9.1"
assertpy = "^1.1"
webdriver-manager = "^3.8.6"
pytest-bdd = "^7.0.1"
pytest-repeat = "^0.9.1"
pytest-html = "^3.2.0"
pytest-bdd-html = "^0.1.14a0"
pytest-loguru = "0.2.0"
pyyaml = "^6.0"
pyodbc = "^5.0.1"
pandas = "^2.0.1"
requests-oauthlib = "^1.3.1"
semantic-release = "^0.1.0"
loguru = "0.7.0"
jinja2 = "^3.1.2"
google-cloud-secret-manager = "^2.16.4"
pytest-xdist = "^3.5.0"

Python version: 3.12.1 OS: Windows 10

Please help.

[youtux]

Hi, please try removing pytest-bdd-html and try again, as I was able to reproduce the issue only by installing that library.Also, there is an issue about pytest-xdist in that repo: https://github.com/slavos1/pytest-bdd-html/issues/6

Closing this unless for now. If that doesn't fix the issue, we can reopen this.