Fixtures not found in Windows Server 2019 after upgrade to Pytest v5.4.1

[muzaffar-omer]

Thanks for your efforts on the great pytest framework. We are using it to test most of our modules. And it works perfectly. Recently, we started using Pytest v5.4.1 in our tests, unfortunately, we observed issues related to fixtures defined in conftest.py files in Windows Server 2019 (Linux and MacOS are perfectly fine)

Background:

We are using Pytest v4.1.1 with Python 3.6.5 to test some internal packages. Below is the directory structure:

.
├── mypy.ini
├── setup.cfg
├── setup.py
├── tests
│   ├── __init__.py
│   └── unit
│       ├── conftest.py
│       ├── test_project.py
│
├── tox.ini

Unit tests are executed using tox, which uses the below command:

pytest -v -rs --color=yes tests/unit 

Tests used to PASS successfully using Pytest v4.1.1, but after the upgrade to Pytest v5.4.1, now tests FAIL because test files are not able to see the fixtures defined in conftest.py file:

E       fixture 'proj_latest_p4d' not found

Tests now PASS only when we use the --rootdir option as below:

pytest -v -rs --color=yes tests/unit --rootdir=tests/unit

[RonnyPfannschmidt]

can you pinpoint the exact pytest release that started it, that would simplify debugging

[muzaffar-omer]

Thanks @RonnyPfannschmidt for your feedback.

We tried to upgrade directly from 4.1.1 to 5.4.1, unfortunately, we did not try another release in-between ... do you have any recommended release in mind?

[RonnyPfannschmidt]

off hand no, i would suggest to try the major numbers first, then go with releases in between so start with the latest in the 4.x series, to see if it started in 5.x, then walk the numbers

[muzaffar-omer]

Ok, will start with the latest 4.x and try multiple releases in between. Will let you know about the result,

Thanks for the suggestion @RonnyPfannschmidt

[muzaffar-omer]

We tried many versions, this issue started to appear in version 5.1.2.

Version 5.1.1 worked perfectly.

[RonnyPfannschmidt]

i strongly suspect https://github.com/pytest-dev/pytest/pull/5792 as reason in particular https://github.com/pytest-dev/pytest/pull/5792/files#diff-b80bb977f96695d5bfe21ddd2680c4efL412 could be a possible reason for the details

@nicoddemus may know more on the windows details

[s0undt3ch]

I also hit a problem regarding #5792

       ________________________ ERROR collecting test session ________________________
       .nox\pytest-parametrized-3-coverage-false-crypto-none-transport-zeromq\lib\site-packages\_pytest\config\__init__.py:495: in _importconftest
           return self._conftestpath2mod[key]
       E   KeyError: WindowsPath('C:/Users/Administrator/AppData/Local/Temp/kitchen/testing/tests/integration/conftest.py')

       During handling of the above exception, another exception occurred:
       .nox\pytest-parametrized-3-coverage-false-crypto-none-transport-zeromq\lib\site-packages\_pytest\config\__init__.py:501: in _importconftest
           mod = conftestpath.pyimport()
       .nox\pytest-parametrized-3-coverage-false-crypto-none-transport-zeromq\lib\site-packages\py\_path\local.py:722: in pyimport
           raise self.ImportMismatchError(modname, modfile, self)
       E   py._path.local.LocalPath.ImportMismatchError: ('tests.integration.conftest', 'C:\\Users\\ADMINI~1\\AppData\\Local\\Temp\\kitchen\\testing\\tests\\integration\\conftest.py', local('C:\\Users\\Administrator\\AppData\\Local\\Temp\\kitchen\\testing\\tests\\integration\\conftest.py'))

       During handling of the above exception, another exception occurred:
       .nox\pytest-parametrized-3-coverage-false-crypto-none-transport-zeromq\lib\site-packages\py\_path\common.py:377: in visit
           for x in Visitor(fil, rec, ignore, bf, sort).gen(self):
       .nox\pytest-parametrized-3-coverage-false-crypto-none-transport-zeromq\lib\site-packages\py\_path\common.py:418: in gen
           dirs = self.optsort([p for p in entries
       .nox\pytest-parametrized-3-coverage-false-crypto-none-transport-zeromq\lib\site-packages\py\_path\common.py:419: in <listcomp>
           if p.check(dir=1) and (rec is None or rec(p))])
       .nox\pytest-parametrized-3-coverage-false-crypto-none-transport-zeromq\lib\site-packages\_pytest\nodes.py:506: in _recurse
           ihook = self._gethookproxy(dirpath)
       .nox\pytest-parametrized-3-coverage-false-crypto-none-transport-zeromq\lib\site-packages\_pytest\nodes.py:487: in _gethookproxy
           my_conftestmodules = pm._getconftestmodules(fspath)
       .nox\pytest-parametrized-3-coverage-false-crypto-none-transport-zeromq\lib\site-packages\_pytest\config\__init__.py:473: in _getconftestmodules
           mod = self._importconftest(conftestpath)
       .nox\pytest-parametrized-3-coverage-false-crypto-none-transport-zeromq\lib\site-packages\_pytest\config\__init__.py:509: in _importconftest
           raise ConftestImportFailure(conftestpath, sys.exc_info())
       E   _pytest.config.ConftestImportFailure: (local('C:\\Users\\Administrator\\AppData\\Local\\Temp\\kitchen\\testing\\tests\\integration\\conftest.py'), (<class 'py._path.local.LocalPath.ImportMismatchError'>, ImportMismatchError('tests.integration.conftest', 'C:\\Users\\ADMINI~1\\AppData\\Local\\Temp\\kitchen\\testing\\tests\\integration\\conftest.py', local('C:\\Users\\Administrator\\AppData\\Local\\Temp\\kitchen\\testing\\tests\\integration\\conftest.py')), <traceback object at 0x00000181F507CD88>))
       - generated xml file: C:\Users\ADMINI~1\AppData\Local\Temp\kitchen\testing\artifacts\xml-unittests-output\test-results-20200522153151.132.xml -
       =========================== short test summary info ===========================
       ERROR ..\..\..\..\..\..\ADMINI~1\AppData\Local\Temp\kitchen\testing - _pytest...
       !!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
       ============================== 1 error in 3.12s ===============================
       Command python -m pytest --rootdir C:\Users\ADMINI~1\AppData\Local\Temp\kitchen\testing --log-file=artifacts\logs\runtests-20200522143158.601175.log --log-file-level=debug --show-capture=no -ra -s --transport=zeromq --output-columns=120 --sysinfo --junitxml=C:\Users\ADMINI~1\AppData\Local\Temp\kitchen/testing/artifacts/xml-unittests-output/test-results-20200522153151.132.xml -v --run-destructive --log-cli-level=debug -vv --run-destructive --run-slow tests/ failed with exit code 2
       Session pytest-parametrized-3(coverage=False, crypto=None, transport='zeromq') failed.
       Ran multiple sessions:
       * pytest-zeromq-3(coverage=False): success
       * pytest-parametrized-3(coverage=False, crypto=None, transport='zeromq'): failed
       Copying $env:KitchenTestingDir/artifacts/ to /home/vampas/projects/SaltStack/salt/pytest/1

It's, in fact, the same file:

E   py._path.local.LocalPath.ImportMismatchError: ('tests.integration.conftest', 'C:\\Users\\ADMINI~1\\AppData\\Local\\Temp\\kitchen\\testing\\tests\\integration\\conftest.py', local('C:\\Users\\Administrator\\AppData\\Local\\Temp\\kitchen\\testing\\tests\\integration\\conftest.py'))

[nicoddemus]

Hmmm seems like pyimport doesn't handle shortened paths properly on Windows due to ImportMismatchError.

We might decide to get rid of pyimport altogether and roll out our own based on importlib.

[nicoddemus]

@s0undt3ch can you confirm the pytest version you are using?

[s0undt3ch]

@nicoddemus 5.4.2

[nicoddemus]

Thanks!

[RonnyPfannschmidt]

@nicoddemus pyimport should be replaced by something pathlib supporting