search

reportportal / agent-python-pytest

Framework integration with PyTest
Apache License 2.0
94 stars 103 forks source link

report-portal plugin causes RuntimeError when using xdist #341

Closed kkonicki-hpe closed 1 year ago

kkonicki-hpe commented 1 year ago

Describe the bug Check the log below:

[2023-06-19T11:45:47.389Z] plugins: reportportal-5.1.7, testrail-2.9.0, cov-4.0.0, Faker-15.3.4, order-1.0.0, md-report-0.3.0, xdist-3.3.0, dependency-0.5.1

[2023-06-19T11:45:47.389Z] created: 42/42 workers

[2023-06-19T11:45:47.389Z] INTERNALERROR> Traceback (most recent call last):

[2023-06-19T11:45:47.389Z] INTERNALERROR>   File "/usr/local/lib/python3.10/site-packages/_pytest/main.py", line 267, in wrap_session

[2023-06-19T11:45:47.389Z] INTERNALERROR>     config.hook.pytest_sessionstart(session=session)

[2023-06-19T11:45:47.389Z] INTERNALERROR>   File "/usr/local/lib/python3.10/site-packages/pluggy/_hooks.py", line 265, in __call__

[2023-06-19T11:45:47.389Z] INTERNALERROR>     return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)

[2023-06-19T11:45:47.389Z] INTERNALERROR>   File "/usr/local/lib/python3.10/site-packages/pluggy/_manager.py", line 80, in _hookexec

[2023-06-19T11:45:47.389Z] INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)

[2023-06-19T11:45:47.389Z] INTERNALERROR>   File "/usr/local/lib/python3.10/site-packages/pluggy/_callers.py", line 60, in _multicall

[2023-06-19T11:45:47.389Z] INTERNALERROR>     return outcome.get_result()

[2023-06-19T11:45:47.389Z] INTERNALERROR>   File "/usr/local/lib/python3.10/site-packages/pluggy/_result.py", line 60, in get_result

[2023-06-19T11:45:47.389Z] INTERNALERROR>     raise ex[1].with_traceback(ex[2])

[2023-06-19T11:45:47.389Z] INTERNALERROR>   File "/usr/local/lib/python3.10/site-packages/pluggy/_callers.py", line 39, in _multicall

[2023-06-19T11:45:47.389Z] INTERNALERROR>     res = hook_impl.function(*args)

[2023-06-19T11:45:47.389Z] INTERNALERROR>   File "/usr/local/lib/python3.10/site-packages/xdist/dsession.py", line 83, in pytest_sessionstart

[2023-06-19T11:45:47.389Z] INTERNALERROR>     nodes = self.nodemanager.setup_nodes(putevent=self.queue.put)

[2023-06-19T11:45:47.389Z] INTERNALERROR>   File "/usr/local/lib/python3.10/site-packages/xdist/workermanage.py", line 68, in setup_nodes

[2023-06-19T11:45:47.389Z] INTERNALERROR>     return [self.setup_node(spec, putevent) for spec in self.specs]

[2023-06-19T11:45:47.389Z] INTERNALERROR>   File "/usr/local/lib/python3.10/site-packages/xdist/workermanage.py", line 68, in <listcomp>

[2023-06-19T11:45:47.390Z] INTERNALERROR>     return [self.setup_node(spec, putevent) for spec in self.specs]

[2023-06-19T11:45:47.390Z] INTERNALERROR>   File "/usr/local/lib/python3.10/site-packages/xdist/workermanage.py", line 76, in setup_node

[2023-06-19T11:45:47.390Z] INTERNALERROR>     node.setup()

[2023-06-19T11:45:47.390Z] INTERNALERROR>   File "/usr/local/lib/python3.10/site-packages/xdist/workermanage.py", line 274, in setup

[2023-06-19T11:45:47.390Z] INTERNALERROR>     self.config.hook.pytest_configure_node(node=self)

[2023-06-19T11:45:47.390Z] INTERNALERROR>   File "/usr/local/lib/python3.10/site-packages/pluggy/_hooks.py", line 265, in __call__

[2023-06-19T11:45:47.390Z] INTERNALERROR>     return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)

[2023-06-19T11:45:47.390Z] INTERNALERROR>   File "/usr/local/lib/python3.10/site-packages/pluggy/_manager.py", line 80, in _hookexec

[2023-06-19T11:45:47.390Z] INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)

[2023-06-19T11:45:47.390Z] INTERNALERROR>   File "/usr/local/lib/python3.10/site-packages/pluggy/_callers.py", line 60, in _multicall

[2023-06-19T11:45:47.390Z] INTERNALERROR>     return outcome.get_result()

[2023-06-19T11:45:47.390Z] INTERNALERROR>   File "/usr/local/lib/python3.10/site-packages/pluggy/_result.py", line 60, in get_result

[2023-06-19T11:45:47.390Z] INTERNALERROR>     raise ex[1].with_traceback(ex[2])

[2023-06-19T11:45:47.390Z] INTERNALERROR>   File "/usr/local/lib/python3.10/site-packages/pluggy/_callers.py", line 39, in _multicall

[2023-06-19T11:45:47.390Z] INTERNALERROR>     res = hook_impl.function(*args)

[2023-06-19T11:45:47.390Z] INTERNALERROR>   File "/usr/local/lib/python3.10/site-packages/pytest_reportportal/plugin.py", line 56, in pytest_configure_node

[2023-06-19T11:45:47.390Z] INTERNALERROR>     node.workerinput['py_test_service'] = pickle.dumps(

[2023-06-19T11:45:47.390Z] INTERNALERROR>   File "/usr/local/lib/python3.10/site-packages/dill/_dill.py", line 263, in dumps

[2023-06-19T11:45:47.390Z] INTERNALERROR>     dump(obj, file, protocol, byref, fmode, recurse, **kwds)#, strictio)

[2023-06-19T11:45:47.390Z] INTERNALERROR>   File "/usr/local/lib/python3.10/site-packages/dill/_dill.py", line 235, in dump

[2023-06-19T11:45:47.390Z] INTERNALERROR>     Pickler(file, protocol, **_kwds).dump(obj)

[2023-06-19T11:45:47.390Z] INTERNALERROR>   File "/usr/local/lib/python3.10/site-packages/dill/_dill.py", line 394, in dump

[2023-06-19T11:45:47.390Z] INTERNALERROR>     StockPickler.dump(self, obj)

[2023-06-19T11:45:47.390Z] INTERNALERROR>   File "/usr/local/lib/python3.10/pickle.py", line 487, in dump

[2023-06-19T11:45:47.390Z] INTERNALERROR>     self.save(obj)

[2023-06-19T11:45:47.390Z] INTERNALERROR>   File "/usr/local/lib/python3.10/site-packages/dill/_dill.py", line 388, in save

[2023-06-19T11:45:47.390Z] INTERNALERROR>     StockPickler.save(self, obj, save_persistent_id)

[2023-06-19T11:45:47.390Z] INTERNALERROR>   File "/usr/local/lib/python3.10/pickle.py", line 603, in save

[2023-06-19T11:45:47.390Z] INTERNALERROR>     self.save_reduce(obj=obj, *rv)

[2023-06-19T11:45:47.390Z] INTERNALERROR>   File "/usr/local/lib/python3.10/pickle.py", line 717, in save_reduce

[2023-06-19T11:45:47.390Z] INTERNALERROR>     save(state)

[2023-06-19T11:45:47.390Z] INTERNALERROR>   File "/usr/local/lib/python3.10/site-packages/dill/_dill.py", line 388, in save

[2023-06-19T11:45:47.390Z] INTERNALERROR>     StockPickler.save(self, obj, save_persistent_id)

[2023-06-19T11:45:47.390Z] INTERNALERROR>   File "/usr/local/lib/python3.10/pickle.py", line 560, in save

[2023-06-19T11:45:47.390Z] INTERNALERROR>     f(self, obj)  # Call unbound method with explicit self

[2023-06-19T11:45:47.390Z] INTERNALERROR>   File "/usr/local/lib/python3.10/site-packages/dill/_dill.py", line 1186, in save_module_dict

[2023-06-19T11:45:47.390Z] INTERNALERROR>     StockPickler.save_dict(pickler, obj)

[2023-06-19T11:45:47.390Z] INTERNALERROR>   File "/usr/local/lib/python3.10/pickle.py", line 972, in save_dict

[2023-06-19T11:45:47.390Z] INTERNALERROR>     self._batch_setitems(obj.items())

[2023-06-19T11:45:47.390Z] INTERNALERROR>   File "/usr/local/lib/python3.10/pickle.py", line 998, in _batch_setitems

[2023-06-19T11:45:47.390Z] INTERNALERROR>     save(v)

[2023-06-19T11:45:47.390Z] INTERNALERROR>   File "/usr/local/lib/python3.10/site-packages/dill/_dill.py", line 388, in save

[2023-06-19T11:45:47.390Z] INTERNALERROR>     StockPickler.save(self, obj, save_persistent_id)

[2023-06-19T11:45:47.390Z] INTERNALERROR>   File "/usr/local/lib/python3.10/pickle.py", line 603, in save

[2023-06-19T11:45:47.390Z] INTERNALERROR>     self.save_reduce(obj=obj, *rv)

[2023-06-19T11:45:47.390Z] INTERNALERROR>   File "/usr/local/lib/python3.10/pickle.py", line 717, in save_reduce

[2023-06-19T11:45:47.390Z] INTERNALERROR>     save(state)

[2023-06-19T11:45:47.390Z] INTERNALERROR>   File "/usr/local/lib/python3.10/site-packages/dill/_dill.py", line 388, in save

[2023-06-19T11:45:47.390Z] INTERNALERROR>     StockPickler.save(self, obj, save_persistent_id)

[2023-06-19T11:45:47.390Z] INTERNALERROR>   File "/usr/local/lib/python3.10/pickle.py", line 560, in save

[2023-06-19T11:45:47.390Z] INTERNALERROR>     f(self, obj)  # Call unbound method with explicit self

[2023-06-19T11:45:47.390Z] INTERNALERROR>   File "/usr/local/lib/python3.10/site-packages/dill/_dill.py", line 1186, in save_module_dict

[2023-06-19T11:45:47.390Z] INTERNALERROR>     StockPickler.save_dict(pickler, obj)

[2023-06-19T11:45:47.390Z] INTERNALERROR>   File "/usr/local/lib/python3.10/pickle.py", line 972, in save_dict

[2023-06-19T11:45:47.390Z] INTERNALERROR>     self._batch_setitems(obj.items())

[2023-06-19T11:45:47.390Z] INTERNALERROR>   File "/usr/local/lib/python3.10/pickle.py", line 998, in _batch_setitems

[2023-06-19T11:45:47.390Z] INTERNALERROR>     save(v)

[2023-06-19T11:45:47.390Z] INTERNALERROR>   File "/usr/local/lib/python3.10/site-packages/dill/_dill.py", line 388, in save

[2023-06-19T11:45:47.390Z] INTERNALERROR>     StockPickler.save(self, obj, save_persistent_id)

[2023-06-19T11:45:47.390Z] INTERNALERROR>   File "/usr/local/lib/python3.10/pickle.py", line 603, in save

[2023-06-19T11:45:47.390Z] INTERNALERROR>     self.save_reduce(obj=obj, *rv)

[2023-06-19T11:45:47.390Z] INTERNALERROR>   File "/usr/local/lib/python3.10/pickle.py", line 717, in save_reduce

[2023-06-19T11:45:47.390Z] INTERNALERROR>     save(state)

[2023-06-19T11:45:47.390Z] INTERNALERROR>   File "/usr/local/lib/python3.10/site-packages/dill/_dill.py", line 388, in save

[2023-06-19T11:45:47.390Z] INTERNALERROR>     StockPickler.save(self, obj, save_persistent_id)

[2023-06-19T11:45:47.390Z] INTERNALERROR>   File "/usr/local/lib/python3.10/pickle.py", line 560, in save

[2023-06-19T11:45:47.390Z] INTERNALERROR>     f(self, obj)  # Call unbound method with explicit self

[2023-06-19T11:45:47.390Z] INTERNALERROR>   File "/usr/local/lib/python3.10/site-packages/dill/_dill.py", line 1186, in save_module_dict

[2023-06-19T11:45:47.390Z] INTERNALERROR>     StockPickler.save_dict(pickler, obj)

[2023-06-19T11:45:47.390Z] INTERNALERROR>   File "/usr/local/lib/python3.10/pickle.py", line 972, in save_dict

[2023-06-19T11:45:47.390Z] INTERNALERROR>     self._batch_setitems(obj.items())

[2023-06-19T11:45:47.391Z] INTERNALERROR>   File "/usr/local/lib/python3.10/pickle.py", line 998, in _batch_setitems

[2023-06-19T11:45:47.391Z] INTERNALERROR>     save(v)

[2023-06-19T11:45:47.391Z] INTERNALERROR>   File "/usr/local/lib/python3.10/site-packages/dill/_dill.py", line 388, in save

[2023-06-19T11:45:47.391Z] INTERNALERROR>     StockPickler.save(self, obj, save_persistent_id)

[2023-06-19T11:45:47.391Z] INTERNALERROR>   File "/usr/local/lib/python3.10/pickle.py", line 603, in save

[2023-06-19T11:45:47.391Z] INTERNALERROR>     self.save_reduce(obj=obj, *rv)

[2023-06-19T11:45:47.391Z] INTERNALERROR>   File "/usr/local/lib/python3.10/pickle.py", line 717, in save_reduce

[2023-06-19T11:45:47.391Z] INTERNALERROR>     save(state)

[2023-06-19T11:45:47.391Z] INTERNALERROR>   File "/usr/local/lib/python3.10/site-packages/dill/_dill.py", line 388, in save

[2023-06-19T11:45:47.391Z] INTERNALERROR>     StockPickler.save(self, obj, save_persistent_id)

[2023-06-19T11:45:47.391Z] INTERNALERROR>   File "/usr/local/lib/python3.10/pickle.py", line 560, in save

[2023-06-19T11:45:47.660Z] /usr/local/lib/python3.10/site-packages/requests/__init__.py:102: RequestsDependencyWarning: urllib3 (1.26.16) or chardet (5.1.0)/charset_normalizer (2.0.12) doesn't match a supported version!

[2023-06-19T11:45:47.660Z]   warnings.warn("urllib3 ({}) or chardet ({})/charset_normalizer ({}) doesn't match a supported "

[2023-06-19T11:45:47.660Z] INTERNALERROR>     f(self, obj)  # Call unbound method with explicit self

[2023-06-19T11:45:47.660Z] INTERNALERROR>   File "/usr/local/lib/python3.10/site-packages/dill/_dill.py", line 1186, in save_module_dict

[2023-06-19T11:45:47.660Z] INTERNALERROR>     StockPickler.save_dict(pickler, obj)

[2023-06-19T11:45:47.660Z] INTERNALERROR>   File "/usr/local/lib/python3.10/pickle.py", line 972, in save_dict

[2023-06-19T11:45:47.660Z] INTERNALERROR>     self._batch_setitems(obj.items())

[2023-06-19T11:45:47.660Z] INTERNALERROR>   File "/usr/local/lib/python3.10/pickle.py", line 998, in _batch_setitems

[2023-06-19T11:45:47.660Z] INTERNALERROR>     save(v)

[2023-06-19T11:45:47.660Z] INTERNALERROR>   File "/usr/local/lib/python3.10/site-packages/dill/_dill.py", line 388, in save

[2023-06-19T11:45:47.660Z] INTERNALERROR>     StockPickler.save(self, obj, save_persistent_id)

[2023-06-19T11:45:47.660Z] INTERNALERROR>   File "/usr/local/lib/python3.10/pickle.py", line 603, in save

[2023-06-19T11:45:47.660Z] INTERNALERROR>     self.save_reduce(obj=obj, *rv)

[2023-06-19T11:45:47.660Z] INTERNALERROR>   File "/usr/local/lib/python3.10/pickle.py", line 717, in save_reduce

[2023-06-19T11:45:47.660Z] INTERNALERROR>     save(state)

[2023-06-19T11:45:47.660Z] INTERNALERROR>   File "/usr/local/lib/python3.10/site-packages/dill/_dill.py", line 388, in save

[2023-06-19T11:45:47.660Z] INTERNALERROR>     StockPickler.save(self, obj, save_persistent_id)

[2023-06-19T11:45:47.660Z] INTERNALERROR>   File "/usr/local/lib/python3.10/pickle.py", line 560, in save

[2023-06-19T11:45:47.660Z] INTERNALERROR>     f(self, obj)  # Call unbound method with explicit self

[2023-06-19T11:45:47.661Z] INTERNALERROR>   File "/usr/local/lib/python3.10/site-packages/dill/_dill.py", line 1186, in save_module_dict

[2023-06-19T11:45:47.661Z] INTERNALERROR>     StockPickler.save_dict(pickler, obj)

[2023-06-19T11:45:47.661Z] INTERNALERROR>   File "/usr/local/lib/python3.10/pickle.py", line 972, in save_dict

[2023-06-19T11:45:47.661Z] INTERNALERROR>     self._batch_setitems(obj.items())

[2023-06-19T11:45:47.661Z] INTERNALERROR>   File "/usr/local/lib/python3.10/pickle.py", line 998, in _batch_setitems

[2023-06-19T11:45:47.661Z] INTERNALERROR>     save(v)

[2023-06-19T11:45:47.661Z] INTERNALERROR>   File "/usr/local/lib/python3.10/site-packages/dill/_dill.py", line 388, in save

[2023-06-19T11:45:47.661Z] INTERNALERROR>     StockPickler.save(self, obj, save_persistent_id)

[2023-06-19T11:45:47.661Z] INTERNALERROR>   File "/usr/local/lib/python3.10/pickle.py", line 603, in save

[2023-06-19T11:45:47.661Z] INTERNALERROR>     self.save_reduce(obj=obj, *rv)

[2023-06-19T11:45:47.661Z] INTERNALERROR>   File "/usr/local/lib/python3.10/pickle.py", line 717, in save_reduce

[2023-06-19T11:45:47.661Z] INTERNALERROR>     save(state)

[2023-06-19T11:45:47.661Z] INTERNALERROR>   File "/usr/local/lib/python3.10/site-packages/dill/_dill.py", line 388, in save

[2023-06-19T11:45:47.661Z] INTERNALERROR>     StockPickler.save(self, obj, save_persistent_id)

[2023-06-19T11:45:47.661Z] INTERNALERROR>   File "/usr/local/lib/python3.10/pickle.py", line 560, in save

[2023-06-19T11:45:47.661Z] INTERNALERROR>     f(self, obj)  # Call unbound method with explicit self

[2023-06-19T11:45:47.661Z] INTERNALERROR>   File "/usr/local/lib/python3.10/site-packages/dill/_dill.py", line 1186, in save_module_dict

[2023-06-19T11:45:47.661Z] INTERNALERROR>     StockPickler.save_dict(pickler, obj)

[2023-06-19T11:45:47.661Z] INTERNALERROR>   File "/usr/local/lib/python3.10/pickle.py", line 972, in save_dict

[2023-06-19T11:45:47.661Z] INTERNALERROR>     self._batch_setitems(obj.items())

[2023-06-19T11:45:47.661Z] INTERNALERROR>   File "/usr/local/lib/python3.10/pickle.py", line 998, in _batch_setitems

[2023-06-19T11:45:47.661Z] INTERNALERROR>     save(v)

[2023-06-19T11:45:47.661Z] INTERNALERROR>   File "/usr/local/lib/python3.10/site-packages/dill/_dill.py", line 388, in save

[2023-06-19T11:45:47.661Z] INTERNALERROR>     StockPickler.save(self, obj, save_persistent_id)

[2023-06-19T11:45:47.661Z] INTERNALERROR>   File "/usr/local/lib/python3.10/pickle.py", line 603, in save

[2023-06-19T11:45:47.661Z] INTERNALERROR>     self.save_reduce(obj=obj, *rv)

[2023-06-19T11:45:47.661Z] INTERNALERROR>   File "/usr/local/lib/python3.10/pickle.py", line 710, in save_reduce

[2023-06-19T11:45:47.661Z] INTERNALERROR>     self._batch_appends(listitems)

[2023-06-19T11:45:47.661Z] INTERNALERROR>   File "/usr/local/lib/python3.10/pickle.py", line 951, in _batch_appends

[2023-06-19T11:45:47.661Z] INTERNALERROR>     tmp = list(islice(it, self._BATCHSIZE))

[2023-06-19T11:45:47.661Z] INTERNALERROR> RuntimeError: deque mutated during iteration

Steps to Reproduce Steps to reproduce the behavior: Run the command: pytest -sv test --rootdir . --reportportal --rp-launch-description "description"

Used lib versions: pytest-reportportal==5.1.7 pytest==7.3.1 pytest-xdist==3.3.0 reportportal-client==5.3.1

Expected behavior No errors during test execution

Actual behavior [2023-06-19T11:45:47.661Z] INTERNALERROR> RuntimeError: deque mutated during iteration

Package versions Include version info of the following packages: reportportal-client, pytest-reportportal pytest-reportportal==5.1.7 pytest==7.3.1 pytest-xdist==3.3.0 reportportal-client==5.3.1

Additional context Add any other context about the problem here.

HardNorth commented 1 year ago

@kkonicki-hpe I can't reproduce it myself, could you please try using the latest version of the agent (5.1.9)?

kkonicki-hpe commented 1 year ago

@kkonicki-hpe I can't reproduce it myself, could you please try using the latest version of the agent (5.1.9)?

Maybe the pytest.ini would be helpful 

[pytest]
rp_uuid = 
rp_endpoint = 
rp_project = 
rp_launch = 
rp_launch_attributes = 
rp_launch_description = 
rp_ignore_errors = True
rp_ignore_attributes = 'xfail' 'usefixture'
rp_hierarchy_module = True
rp_hierarchy_dirs = True
rp_hierarchy_parametrize = True
rp_is_skipped_an_issue = False

filterwarnings =
    ignore::DeprecationWarning
addopts = -svra --md-report --disable-warnings -p no:allure_pytest --ignore=steps -n 42 --dist loadscope
HardNorth commented 1 year ago

@kkonicki-hpe Excuse me, but please do what I ask, if you want to get support.

HardNorth commented 1 year ago

Abandoned

kkonicki-hpe commented 1 year ago

Problem was caused by xdist. If we declared more workers than we got tests then problem disapeared.

Just changed -n 42 to -n 62