schireson / pytest-mock-resources

Pytest Fixtures that let you actually test against external resource (Postgres, Mongo, Redshift...) dependent code.
https://pytest-mock-resources.readthedocs.io/en/latest/quickstart.html
MIT License
179 stars 19 forks source link

Exception trying to unlink pmr.json when when ran with --pmr-multiprocess-safe #139

Closed JonM0 closed 2 years ago

JonM0 commented 2 years ago

Describe the bug When running tests with pytest-xdist, using the flag --pmr-multiprocess-safe, the tests are executed correctly but pytest_sessionfinish throws an error: PermissionError: [WinError 32] The process cannot access the file because it is being used by another process when trying to unlink pmr.json while holding its lock.

Environment

[tool.poetry.dev-dependencies] pytest = "^7.0" pytest-xdist = "^2.5" pytest-mock-resources = { extras = ["postgres-binary"], version = "^2.2" } pywin32 = ">227"


**To Reproduce**
Steps to reproduce the behavior:
1. Create the following files:
```py
# conftest.py
from pytest_mock_resources import create_postgres_fixture
pg_engine = create_postgres_fixture()
# test_main.py
def test(pg_engine):
    with pg_engine.connect() as conn:
        assert conn.execute('select 1').scalar() == 1
  1. Run 'pytest -n2 --pmr-multiprocess-safe'
  2. Test executes correctly, but at the end the error is printed

Expected behavior No exception is raised and pmr.json is removed from the tmp folder.

Actual Behavior

Testing started at 20:37 ...
Launching pytest with arguments -n2 --pmr-multiprocess-safe --no-header --no-summary -q in F:\code\pmr_bug_repr

============================= test session starts =============================
gw0 I / gw1 I
[gw0] win32 Python 3.10.2 cwd: F:\code\pmr_bug_repr
[gw1] win32 Python 3.10.2 cwd: F:\code\pmr_bug_repr
[gw0] Python 3.10.2 (tags/v3.10.2:a58ebcc, Jan 17 2022, 14:12:15) [MSC v.1929 64 bit (AMD64)]
[gw1] Python 3.10.2 (tags/v3.10.2:a58ebcc, Jan 17 2022, 14:12:15) [MSC v.1929 64 bit (AMD64)]
gw0 [1] / gw1 [1]

scheduling tests via LoadScheduling

test_main.py::test 
[gw0] [100%] PASSED test_main.py::test 
Traceback (most recent call last):
  File "C:\Program Files\JetBrains\PyCharm 2021.1\plugins\python\helpers\pycharm\_jb_pytest_runner.py", line 51, in <module>
    sys.exit(pytest.main(args, plugins_to_load + [Plugin]))
  File "C:\Users\[me]\AppData\Local\pypoetry\Cache\virtualenvs\pmr-bug-repr-pLX3u5c4-py3.10\lib\site-packages\_pytest\config\__init__.py", line 165, in main
    ret: Union[ExitCode, int] = config.hook.pytest_cmdline_main(
  File "C:\Users\[me]\AppData\Local\pypoetry\Cache\virtualenvs\pmr-bug-repr-pLX3u5c4-py3.10\lib\site-packages\pluggy\_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  File "C:\Users\[me]\AppData\Local\pypoetry\Cache\virtualenvs\pmr-bug-repr-pLX3u5c4-py3.10\lib\site-packages\pluggy\_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "C:\Users\[me]\AppData\Local\pypoetry\Cache\virtualenvs\pmr-bug-repr-pLX3u5c4-py3.10\lib\site-packages\pluggy\_callers.py", line 60, in _multicall
    return outcome.get_result()
  File "C:\Users\[me]\AppData\Local\pypoetry\Cache\virtualenvs\pmr-bug-repr-pLX3u5c4-py3.10\lib\site-packages\pluggy\_result.py", line 60, in get_result
    raise ex[1].with_traceback(ex[2])
  File "C:\Users\[me]\AppData\Local\pypoetry\Cache\virtualenvs\pmr-bug-repr-pLX3u5c4-py3.10\lib\site-packages\pluggy\_callers.py", line 39, in _multicall
    res = hook_impl.function(*args)
  File "C:\Users\[me]\AppData\Local\pypoetry\Cache\virtualenvs\pmr-bug-repr-pLX3u5c4-py3.10\lib\site-packages\_pytest\main.py", line 315, in pytest_cmdline_main
    return wrap_session(config, _main)
  File "C:\Users\[me]\AppData\Local\pypoetry\Cache\virtualenvs\pmr-bug-repr-pLX3u5c4-py3.10\lib\site-packages\_pytest\main.py", line 303, in wrap_session
    config.hook.pytest_sessionfinish(
  File "C:\Users\[me]\AppData\Local\pypoetry\Cache\virtualenvs\pmr-bug-repr-pLX3u5c4-py3.10\lib\site-packages\pluggy\_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  File "C:\Users\[me]\AppData\Local\pypoetry\Cache\virtualenvs\pmr-bug-repr-pLX3u5c4-py3.10\lib\site-packages\pluggy\_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "C:\Users\[me]\AppData\Local\pypoetry\Cache\virtualenvs\pmr-bug-repr-pLX3u5c4-py3.10\lib\site-packages\pluggy\_callers.py", line 55, in _multicall
    gen.send(outcome)
  File "C:\Users\[me]\AppData\Local\pypoetry\Cache\virtualenvs\pmr-bug-repr-pLX3u5c4-py3.10\lib\site-packages\_pytest\terminal.py", line 792, in pytest_sessionfinish
    outcome.get_result()
  File "C:\Users\[me]\AppData\Local\pypoetry\Cache\virtualenvs\pmr-bug-repr-pLX3u5c4-py3.10\lib\site-packages\pluggy\_result.py", line 60, in get_result
    raise ex[1].with_traceback(ex[2])
  File "C:\Users\[me]\AppData\Local\pypoetry\Cache\virtualenvs\pmr-bug-repr-pLX3u5c4-py3.10\lib\site-packages\pluggy\_callers.py", line 39, in _multicall
    res = hook_impl.function(*args)
  File "C:\Users\[me]\AppData\Local\pypoetry\Cache\virtualenvs\pmr-bug-repr-pLX3u5c4-py3.10\lib\site-packages\pytest_mock_resources\hooks.py", line 103, in pytest_sessionfinish
    fn.unlink()
  File "C:\Users\[me]\AppData\Local\Programs\Python\Python310\lib\pathlib.py", line 1204, in unlink
    self._accessor.unlink(self)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\[me]\\AppData\\Local\\Temp\\pytest-of-[me]\\pmr.json'

Process finished with exit code 1

Additional context I was able to fix it by tabbing left by 1 the call to py fn.unlink() in hooks.py, moving it outside the lockfile. Turning this

def pytest_sessionfinish(session, exitstatus):
    ...
    with load_container_lockfile(fn) as containers:
        ...
        fn.unlink()

into this

def pytest_sessionfinish(session, exitstatus):
    ...
    with load_container_lockfile(fn) as containers:
        ...
    fn.unlink()
DanCardin commented 2 years ago

Thanks for the great issue report! Interesting that I wasn't hitting this issue on linux/macos.

DanCardin commented 2 years ago

Just merged and released as 2.2.1. Thanks again!

We're internally testing this flag on our end, but it's brand new so let us know if you run into anything else!

JonM0 commented 2 years ago

No problem, glad to help! This package is looking pretty good!