search

seleniumbase / SeleniumBase

📊 Python's all-in-one framework for web crawling, scraping, testing, and reporting. Supports pytest. UC Mode provides stealth. Includes many tools.
https://seleniumbase.io
MIT License
4.67k stars 929 forks source link

Race condition when using parallelism `-n` with `--dashboard` and `--html` #1766

Closed lhole closed 1 year ago

lhole commented 1 year ago

When running tests with pytest in our CI with the dashboard options enabled, we often receive an INTERNALERROR. I think the relevant part of the stack trace is:

INTERNALERROR> E               File "/builds/tc/software/.cache/venvs/marketplace-iZsxknbI-py3.9/lib/python3.9/site-packages/seleniumbase/plugins/pytest_plugin.py", line 1750, in pytest_collection_finish
INTERNALERROR> E                 _create_dashboard_assets_()
INTERNALERROR> E               File "/builds/tc/software/.cache/venvs/marketplace-iZsxknbI-py3.9/lib/python3.9/site-packages/seleniumbase/plugins/pytest_plugin.py", line 1685, in _create_dashboard_assets_
INTERNALERROR> E                 os.makedirs(assets_folder)
INTERNALERROR> E               File "/usr/local/lib/python3.9/os.py", line 225, in makedirs
INTERNALERROR> E                 mkdir(name, mode)
INTERNALERROR> E             FileExistsError: [Errno 17] File exists: '/builds/tc/software/services/marketplace/assets'
INTERNALERROR> E           assert False

Looking at the code, it seems like there's a race condition where the existence of the directory is checked before actually doing the write. I assume that because there are multiple processes contending, two of them can check whether the directory exists and both find it doesn't, then the one process would be able to create the dir and the other wouldn't. I think the solution is to just add exist_ok=True to the os.makedirs call.

It looks like you've already added that protection in patcher.py. I also see that there are a bunch of places in the code calling os.makedirs without it specified. I can submit a PR for just the line causing the problem but I'll need some guidance as to whether it should be applied everywhere or not.

Full stack trace:

poetry run pytest -m "e2e" --protocol=http --chrome --headless --server=tc-e2e-chrome  --port=4444 --list-fail-page -n 8 --ignore post_deal --dashboard --html=public/report.html --reruns 3 --reruns-delay 5
Dashboard: file:///builds/tc/software/services/marketplace/dashboard.html
*********************************************************************************
============================= test session starts ==============================
platform linux -- Python 3.9.16, pytest-7.2.1, pluggy-1.0.0
django: settings: marketplace.settings.test (from ini)
rootdir: /builds/tc/software/services/marketplace, configfile: pyproject.toml
plugins: html-2.0.1, seleniumbase-4.12.8, lazy-fixture-0.6.3, django-4.5.2, deadfixtures-2.2.1, metadata-2.0.4, httpx-0.21.3, mock-3.10.0, subtests-0.10.0, xdist-3.2.0, ordering-0.6, anyio-3.6.2, asyncio-0.20.3, forked-1.4.0, requests-mock-1.10.0, tc-django-common-1.0.0, rerunfailures-11.0, cov-4.0.0
asyncio: mode=strict
gw0 I / gw1 I / gw2 I / gw3 I / gw4 I / gw5 I / gw6 I / gw7 I
INTERNALERROR> def worker_internal_error(self, node, formatted_error):
INTERNALERROR>         """
INTERNALERROR>         pytest_internalerror() was called on the worker.
INTERNALERROR>     
INTERNALERROR>         pytest_internalerror() arguments are an excinfo and an excrepr, which can't
INTERNALERROR>         be serialized, so we go with a poor man's solution of raising an exception
INTERNALERROR>         here ourselves using the formatted message.
INTERNALERROR>         """
INTERNALERROR>         self._active_nodes.remove(node)
INTERNALERROR>         try:
INTERNALERROR> >           assert False, formatted_error
INTERNALERROR> E           AssertionError: Traceback (most recent call last):
INTERNALERROR> E               File "/builds/tc/software/.cache/venvs/marketplace-iZsxknbI-py3.9/lib/python3.9/site-packages/_pytest/main.py", line 270, in wrap_session
INTERNALERROR> E                 session.exitstatus = doit(config, session) or 0
INTERNALERROR> E               File "/builds/tc/software/.cache/venvs/marketplace-iZsxknbI-py3.9/lib/python3.9/site-packages/_pytest/main.py", line 323, in _main
INTERNALERROR> E                 config.hook.pytest_collection(session=session)
INTERNALERROR> E               File "/builds/tc/software/.cache/venvs/marketplace-iZsxknbI-py3.9/lib/python3.9/site-packages/pluggy/_hooks.py", line 265, in __call__
INTERNALERROR> E                 return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
INTERNALERROR> E               File "/builds/tc/software/.cache/venvs/marketplace-iZsxknbI-py3.9/lib/python3.9/site-packages/pluggy/_manager.py", line 80, in _hookexec
INTERNALERROR> E                 return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR> E               File "/builds/tc/software/.cache/venvs/marketplace-iZsxknbI-py3.9/lib/python3.9/site-packages/pluggy/_callers.py", line 60, in _multicall
INTERNALERROR> E                 return outcome.get_result()
INTERNALERROR> E               File "/builds/tc/software/.cache/venvs/marketplace-iZsxknbI-py3.9/lib/python3.9/site-packages/pluggy/_result.py", line 60, in get_result
INTERNALERROR> E                 raise ex[1].with_traceback(ex[2])
INTERNALERROR> E               File "/builds/tc/software/.cache/venvs/marketplace-iZsxknbI-py3.9/lib/python3.9/site-packages/pluggy/_callers.py", line 39, in _multicall
INTERNALERROR> E                 res = hook_impl.function(*args)
INTERNALERROR> E               File "/builds/tc/software/.cache/venvs/marketplace-iZsxknbI-py3.9/lib/python3.9/site-packages/_pytest/main.py", line 334, in pytest_collection
INTERNALERROR> E                 session.perform_collect()
INTERNALERROR> E               File "/builds/tc/software/.cache/venvs/marketplace-iZsxknbI-py3.9/lib/python3.9/site-packages/_pytest/main.py", line 671, in perform_collect
INTERNALERROR> E                 hook.pytest_collection_finish(session=self)
INTERNALERROR> E               File "/builds/tc/software/.cache/venvs/marketplace-iZsxknbI-py3.9/lib/python3.9/site-packages/pluggy/_hooks.py", line 265, in __call__
INTERNALERROR> E                 return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
INTERNALERROR> E               File "/builds/tc/software/.cache/venvs/marketplace-iZsxknbI-py3.9/lib/python3.9/site-packages/pluggy/_manager.py", line 80, in _hookexec
INTERNALERROR> E                 return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR> E               File "/builds/tc/software/.cache/venvs/marketplace-iZsxknbI-py3.9/lib/python3.9/site-packages/pluggy/_callers.py", line 60, in _multicall
INTERNALERROR> E                 return outcome.get_result()
INTERNALERROR> E               File "/builds/tc/software/.cache/venvs/marketplace-iZsxknbI-py3.9/lib/python3.9/site-packages/pluggy/_result.py", line 60, in get_result
INTERNALERROR> E                 raise ex[1].with_traceback(ex[2])
INTERNALERROR> E               File "/builds/tc/software/.cache/venvs/marketplace-iZsxknbI-py3.9/lib/python3.9/site-packages/pluggy/_callers.py", line 39, in _multicall
INTERNALERROR> E                 res = hook_impl.function(*args)
INTERNALERROR> E               File "/builds/tc/software/.cache/venvs/marketplace-iZsxknbI-py3.9/lib/python3.9/site-packages/seleniumbase/plugins/pytest_plugin.py", line 1750, in pytest_collection_finish
INTERNALERROR> E                 _create_dashboard_assets_()
INTERNALERROR> E               File "/builds/tc/software/.cache/venvs/marketplace-iZsxknbI-py3.9/lib/python3.9/site-packages/seleniumbase/plugins/pytest_plugin.py", line 1685, in _create_dashboard_assets_
INTERNALERROR> E                 os.makedirs(assets_folder)
INTERNALERROR> E               File "/usr/local/lib/python3.9/os.py", line 225, in makedirs
INTERNALERROR> E                 mkdir(name, mode)
INTERNALERROR> E             FileExistsError: [Errno 17] File exists: '/builds/tc/software/services/marketplace/assets'
INTERNALERROR> E           assert False
INTERNALERROR> 
INTERNALERROR> ../../.cache/venvs/marketplace-iZsxknbI-py3.9/lib/python3.9/site-packages/xdist/dsession.py:192: AssertionError
gw0 [18] / gw1 [18] / gw2 [18] / gw3 [18] / gw4 [18] / gw5 [18] / gw6 [18] / gw7 [18]
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/builds/tc/software/.cache/venvs/marketplace-iZsxknbI-py3.9/lib/python3.9/site-packages/_pytest/main.py", line 270, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/builds/tc/software/.cache/venvs/marketplace-iZsxknbI-py3.9/lib/python3.9/site-packages/_pytest/main.py", line 324, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/builds/tc/software/.cache/venvs/marketplace-iZsxknbI-py3.9/lib/python3.9/site-packages/pluggy/_hooks.py", line 265, in __call__
INTERNALERROR>     return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
INTERNALERROR>   File "/builds/tc/software/.cache/venvs/marketplace-iZsxknbI-py3.9/lib/python3.9/site-packages/pluggy/_manager.py", line 80, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/builds/tc/software/.cache/venvs/marketplace-iZsxknbI-py3.9/lib/python3.9/site-packages/pluggy/_callers.py", line 60, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/builds/tc/software/.cache/venvs/marketplace-iZsxknbI-py3.9/lib/python3.9/site-packages/pluggy/_result.py", line 60, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/builds/tc/software/.cache/venvs/marketplace-iZsxknbI-py3.9/lib/python3.9/site-packages/pluggy/_callers.py", line 39, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/builds/tc/software/.cache/venvs/marketplace-iZsxknbI-py3.9/lib/python3.9/site-packages/xdist/dsession.py", line 117, in pytest_runtestloop
INTERNALERROR>     self.loop_once()
INTERNALERROR>   File "/builds/tc/software/.cache/venvs/marketplace-iZsxknbI-py3.9/lib/python3.9/site-packages/xdist/dsession.py", line 140, in loop_once
INTERNALERROR>     call(**kwargs)
mdmintz commented 1 year ago

@lhole Nice catch. Thanks for finding this. I can ship a fix for this in a new release in the next hour or so.

mdmintz commented 1 year ago

This was fixed in 4.13.3 - https://github.com/seleniumbase/SeleniumBase/releases/tag/v4.13.3 (was just uploaded to PyPI)