Regression in 7.1 ? Windows CI started to fail with "assert mod not in mods" since 7.1

Carreau commented 2 years ago

Now sorry I know this is not a minimal reproducer, but I just want to log this here, and will investigate later:

You can see one Failure here

ERROR collecting test session 
C:\hostedtoolcache\windows\Python\3.9.10\x64\lib\site-packages\_pytest\runner.py:338: in from_call
    result: Optional[TResult] = func()
C:\hostedtoolcache\windows\Python\3.9.10\x64\lib\site-packages\_pytest\runner.py:369: in <lambda>
    call = CallInfo.from_call(lambda: list(collector.collect()), "collect")
C:\hostedtoolcache\windows\Python\3.9.10\x64\lib\site-packages\_pytest\main.py:7[23](https://github.com/ipython/ipython/runs/5535021197?check_suite_focus=true#step:8:23): in collect
    for x in self._collectfile(pkginit):
C:\hostedtoolcache\windows\Python\3.9.10\x64\lib\site-packages\_pytest\main.py:575: in _collectfile
    ihook = self.gethookproxy(fspath)
C:\hostedtoolcache\windows\Python\3.9.10\x64\lib\site-packages\_pytest\main.py:539: in gethookproxy
    my_conftestmodules = pm._getconftestmodules(
C:\hostedtoolcache\windows\Python\3.9.10\x64\lib\site-packages\_pytest\config\__init__.py:579: in _getconftestmodules
    mod = self._importconftest(conftestpath, importmode, rootpath)
C:\hostedtoolcache\windows\Python\3.9.10\x64\lib\site-packages\_pytest\config\__init__.py:6[24](https://github.com/ipython/ipython/runs/5535021197?check_suite_focus=true#step:8:24): in _importconftest
    assert mod not in mods
E   AssertionError

Now I'm not sure what this does, but my quick understanding is that it checks whether conftest.py itself is not in the list of collected modules ? We might definitely do something wrong on IPython side, but it's strange as other non-windows CI are passing and regardless of whether this is something I did wrong, can the assert get an explanation messages as second argument ?

Downgrading to <7.1 fixes the issue.

Again I'll try to get a MVP/small example, but it may take me some time, and I guessed maybe one of you will immediately know why this is happening.

[ ] a detailed description of the bug or problem you are having
[ ] output of pip list from the virtual environment you are using
[ ] pytest and operating system versions
[ ] minimal example if possible

bluetech commented 2 years ago

Thanks for the report, I will take a look when I get a chance. 7.1 had some internal conftest refactorings. If you manage to provide a reproduction that'd be great.

The-Compiler commented 2 years ago

One thing that seems somewhat suspicious is how IPython's pytest_ipdoctest.py seems to have some custom code around importing conftest files:

https://github.com/ipython/ipython/blob/d7cc335a36cdfe4159c9b88c94311866c923a547/IPython/testing/plugin/pytest_ipdoctest.py#L635-L646

Carreau commented 2 years ago

Yeah, it might be something on our end, we have to do some pretty weird things seeing we also an interpreter. So I would not be surprised if the problem is from our plugins.

Carreau commented 2 years ago

It does seem like removing the plugin fixes the issue, I'm not quite sure why it's windows only, and I haven't written it, so I'm not sure how I can fix it.

Feel free to close, worse case I'll run the plugin only on linux CI.

gonzalocasas commented 2 years ago

Not sure if it helps, I know it's not a slim repro setup, but this repo was just pinned to <7.1 because it was consistently reproducible in there: https://github.com/compas-dev/compas

bashtage commented 2 years ago

Seeing same in statsmodels on Windows.

https://dev.azure.com/statsmodels/statsmodels-testing/_build/results?buildId=4410&view=logs&j=9c2ba6b6-9ef1-5300-06cd-4db9cc34ebf3&t=f07685ef-4dee-5864-e861-bb3b9ab4fc6b

Carreau commented 2 years ago

@gonzalocasas and @bashtage I see both your CI install IPython in them, could you try pinning IPython to <8 to see if it's a problem with the IPython doctest plugin itself (that should not be present in 7.x)

bashtage commented 2 years ago

@Carreau I get the same error when I pip install ipython<8. I also get the same error when I pip uninstall ipython. Works find with 7.0.x, broken in 7.1.x.

bashtage commented 2 years ago

I can also get this error when I reun specific tests:

 ValueError: Plugin already registered: c:\git\statsmodels\statsmodels\conftest.py=<module 'statsmodels.conftest' from 'c:\\git\\statsmodels\\statsmodels\\conftest.py'>

jgb commented 2 years ago

I've got the same error, but on linux! Can't run any test on 7.1.x, reverting back to 7.0.x fixes the issue:

root@f786a0bfb51b:/srv/vidicenter# pytest -n4
WARNING No rollbar access token provided
Traceback (most recent call last):
  File "/usr/local/bin/pytest", line 8, in <module>
    sys.exit(console_main())
  File "/usr/local/lib/python3.8/dist-packages/_pytest/config/__init__.py", line 187, in console_main
    code = main()
  File "/usr/local/lib/python3.8/dist-packages/_pytest/config/__init__.py", line 145, in main
    config = _prepareconfig(args, plugins)
  File "/usr/local/lib/python3.8/dist-packages/_pytest/config/__init__.py", line 324, in _prepareconfig
    config = pluginmanager.hook.pytest_cmdline_parse(
  File "/usr/local/lib/python3.8/dist-packages/pluggy/_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  File "/usr/local/lib/python3.8/dist-packages/pluggy/_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/usr/local/lib/python3.8/dist-packages/pluggy/_callers.py", line 55, in _multicall
    gen.send(outcome)
  File "/usr/local/lib/python3.8/dist-packages/_pytest/helpconfig.py", line 102, in pytest_cmdline_parse
    config: Config = outcome.get_result()
  File "/usr/local/lib/python3.8/dist-packages/pluggy/_result.py", line 60, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/usr/local/lib/python3.8/dist-packages/pluggy/_callers.py", line 39, in _multicall
    res = hook_impl.function(*args)
  File "/usr/local/lib/python3.8/dist-packages/_pytest/config/__init__.py", line 1016, in pytest_cmdline_parse
    self.parse(args)
  File "/usr/local/lib/python3.8/dist-packages/_pytest/config/__init__.py", line 1304, in parse
    self._preparse(args, addopts=addopts)
  File "/usr/local/lib/python3.8/dist-packages/_pytest/config/__init__.py", line 1206, in _preparse
    self.hook.pytest_load_initial_conftests(
  File "/usr/local/lib/python3.8/dist-packages/pluggy/_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  File "/usr/local/lib/python3.8/dist-packages/pluggy/_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/usr/local/lib/python3.8/dist-packages/pluggy/_callers.py", line 60, in _multicall
    return outcome.get_result()
  File "/usr/local/lib/python3.8/dist-packages/pluggy/_result.py", line 60, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/usr/local/lib/python3.8/dist-packages/pluggy/_callers.py", line 39, in _multicall
    res = hook_impl.function(*args)
  File "/usr/local/lib/python3.8/dist-packages/_pytest/config/__init__.py", line 1083, in pytest_load_initial_conftests
    self.pluginmanager._set_initial_conftests(
  File "/usr/local/lib/python3.8/dist-packages/_pytest/config/__init__.py", line 532, in _set_initial_conftests
    self._try_load_conftest(current, namespace.importmode, rootpath)
  File "/usr/local/lib/python3.8/dist-packages/_pytest/config/__init__.py", line 551, in _try_load_conftest
    self._getconftestmodules(x, importmode, rootpath)
  File "/usr/local/lib/python3.8/dist-packages/_pytest/config/__init__.py", line 575, in _getconftestmodules
    mod = self._importconftest(conftestpath, importmode, rootpath)
  File "/usr/local/lib/python3.8/dist-packages/_pytest/config/__init__.py", line 620, in _importconftest
    assert mod not in mods
AssertionError

bnavigator commented 2 years ago

FYI: I also get this in an rpmbuild of asdf for openSUSE Linux. I attribute it to some import mismatch for the conftest.py installed into the sitelib.

Fails:

Have asdf installed in sitelib but collect from asdf in source directory. This worked with pytest < 7

[    3s] Executing(%check): /usr/bin/bash -e /var/tmp/rpm-tmp.cC3tTE
[    3s] + umask 022
[    3s] + cd /home/abuild/rpmbuild/BUILD
[    3s] + cd asdf-2.11.1
[    3s] + export LANG=en_US.UTF-8
[    3s] + LANG=en_US.UTF-8
[    3s] + export PY_IGNORE_IMPORTMISMATCH=1
[    3s] + PY_IGNORE_IMPORTMISMATCH=1
[    3s] ++ '[' -f _current_flavor ']'
[    3s] ++ cat _current_flavor
[    3s] + last_flavor=python38
[    3s] + '[' -z python38 ']'
[    3s] + '[' python38 '!=' python39 ']'
[    3s] + '[' -d build ']'
[    3s] + mv build _build.python38
[    3s] + '[' -d _build.python39 ']'
[    3s] + mv _build.python39 build
[    3s] + echo python39
[    3s] + python_flavor=python39
[    3s] + PYTHONDONTWRITEBYTECODE=1
[    3s] + pytest-3.9 --ignore=_build.python39 --ignore=_build.python310 --ignore=_build.python38 -v --remote-data=none
[    3s] Internet access disabled
[    3s] ============================= test session starts ==============================
[    3s] platform linux -- Python 3.9.12, pytest-7.1.1, pluggy-1.0.0 -- /usr/bin/python3.9
[    3s] cachedir: .pytest_cache
[    3s] rootdir: /home/abuild/rpmbuild/BUILD/asdf-2.11.1, configfile: setup.cfg, testpaths: asdf, docs
[    3s] plugins: asdf-2.11.1, sugar-0.9.4, openfiles-0.5.0, doctestplus-0.12.0, remotedata-0.3.2
[    4s] collecting ... collected 0 items / 1 error
[    4s] 
[    4s] ==================================== ERRORS ====================================
[    4s] ________________________ ERROR collecting test session _________________________
[    4s] /usr/lib/python3.9/site-packages/_pytest/runner.py:338: in from_call
[    4s]     result: Optional[TResult] = func()
[    4s] /usr/lib/python3.9/site-packages/_pytest/runner.py:369: in <lambda>
[    4s]     call = CallInfo.from_call(lambda: list(collector.collect()), "collect")
[    4s] /usr/lib/python3.9/site-packages/_pytest/main.py:723: in collect
[    4s]     for x in self._collectfile(pkginit):
[    4s] /usr/lib/python3.9/site-packages/_pytest/main.py:575: in _collectfile
[    4s]     ihook = self.gethookproxy(fspath)
[    4s] /usr/lib/python3.9/site-packages/_pytest/main.py:539: in gethookproxy
[    4s]     my_conftestmodules = pm._getconftestmodules(
[    4s] /usr/lib/python3.9/site-packages/_pytest/config/__init__.py:575: in _getconftestmodules
[    4s]     mod = self._importconftest(conftestpath, importmode, rootpath)
[    4s] /usr/lib/python3.9/site-packages/_pytest/config/__init__.py:620: in _importconftest
[    4s]     assert mod not in mods
[    4s] E   AssertionError
[    4s] =========================== short test summary info ============================
[    4s] ERROR  - AssertionError
[    4s] !!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
[    4s] =============================== 1 error in 0.29s ===============================
[    4s] error: Bad exit status from /var/tmp/rpm-tmp.cC3tTE (%check)

Works:

move the asdf source directory away and directly test with --pyargs asdf

[    3s] Executing(%check): /usr/bin/bash -e /var/tmp/rpm-tmp.wVMB1t
[    3s] + umask 022
[    3s] + cd /home/abuild/rpmbuild/BUILD
[    3s] + cd asdf-2.11.1
[    3s] + export LANG=en_US.UTF-8
[    3s] + LANG=en_US.UTF-8
[    3s] + mv asdf asdf.moved
[    3s] ++ '[' -f _current_flavor ']'
[    3s] ++ cat _current_flavor
[    3s] + last_flavor=python38
[    3s] + '[' -z python38 ']'
[    3s] + '[' python38 '!=' python39 ']'
[    3s] + '[' -d build ']'
[    3s] + mv build _build.python38
[    3s] + '[' -d _build.python39 ']'
[    3s] + mv _build.python39 build
[    3s] + echo python39
[    3s] + python_flavor=python39
[    3s] + PYTHONDONTWRITEBYTECODE=1
[    3s] + pytest-3.9 --ignore=_build.python39 --ignore=_build.python310 --ignore=_build.python38 -v --pyargs asdf --remote-data=none
[    3s] Internet access disabled
[    3s] ============================= test session starts ==============================
[    3s] platform linux -- Python 3.9.12, pytest-7.1.1, pluggy-1.0.0 -- /usr/bin/python3.9
[    3s] cachedir: .pytest_cache
[    3s] rootdir: /home/abuild/rpmbuild/BUILD/asdf-2.11.1, configfile: setup.cfg
[    3s] plugins: asdf-2.11.1, sugar-0.9.4, openfiles-0.5.0, doctestplus-0.12.0, remotedata-0.3.2
[    5s] collecting ... collected 721 items
[    5s] 
[    5s] asdf.py::asdf.asdf.AsdfFile.make_reference PASSED                        [  0%]
...
[  200s] ============ 691 passed, 8 skipped, 22 xfailed in 196.36s (0:03:16) ==========

chebee7i commented 2 years ago

Any chance this can be related to finding multiple conftest.py? This was working for us in linux on 7.0.1 and no longer working in 7.1.1. When we hackily removed conf files so that only one could be found, the issue went away.

So we've reverted and are pinning pytest to 7.0.1 for now.

Would be great to see this fixed, or provide a migration path for how to deal with it in 7.1.0. We are in a monorepo, and use the pytest runner for the entire repo (at the root). Has never been an issue in the past.

melund commented 2 years ago

Any chance this can be related to finding multiple conftest.py?

This could very well be the case. I had a colleague who had the same issue on Windows. It turned out to be triggered by him creating his the conda test environment as a subfolder inside the test folder (Not recommended)

So pytest was picking up an extra conftest.py file from the numpy folder in site-packages/ (numpy ships it tests files with the module)

jacksund commented 2 years ago

Just to add to this, the error occurs in my repo (issue 162) even when conda is set up properly (not as a subfolder like @melund describes). My repo only has a single conftest.py too. Leads me to believe this issue is happening in conda environments where installed dependencies also have conftest.py files. Maybe pytest is incorrectly grabbing conftest files from package dependencies. Is it that a possibility?

RonnyPfannschmidt commented 2 years ago

conftests are only ever searched locally, so unless you put evil magic in your tests folders you shouldnt see this

jacksund commented 2 years ago

lol good to know. I must have done some evil magic without realizing it :sweat_smile:

Kojoley commented 2 years ago

The issue is reproducible only when the IPython installed via pip<21.3 (first good commit https://github.com/pypa/pip/commit/e5be3f796e710301052afb7839acb6c7b471e3dd), so it might be just because of IPython's pyproject.toml

jgb commented 2 years ago

Hello,

just an update to say that this issue still exists, on linux, with pytest 7.2.0. Can anything be done about this? I have a single conftest.py in my root folder, and then several smaller conftest.py in subfolders for specific modules. This has always worked in pytest < 7.1.0. Am I doing something wrong? Or is this a bug that could be fixed?

Greetings,

jgb

dezhin commented 2 years ago

For our case, the issue was in symlinks which we created inside site-packages directory. We switched to --pyargs (python -m pytest --pyargs nextgisweb instead of python -m pytest package/nextgisweb/nextgisweb) and now everything works well.

jgb commented 1 year ago

@bluetech any news on this? Anything newer than pytest 7.0.1 is still broken for me on linux, including the latest 7.2.1. I've got one conftest.py in my root, and several smaller conftest.py in subfolders for different django apps. This works fine in 7.0.1 and before, but fails to run at all in any newer version, with the following output:

Traceback (most recent call last):
  File "/usr/local/bin/pytest", line 8, in <module>
    sys.exit(console_main())
  File "/usr/local/lib/python3.10/dist-packages/_pytest/config/__init__.py", line 190, in console_main
    code = main()
  File "/usr/local/lib/python3.10/dist-packages/_pytest/config/__init__.py", line 148, in main
    config = _prepareconfig(args, plugins)
  File "/usr/local/lib/python3.10/dist-packages/_pytest/config/__init__.py", line 329, in _prepareconfig
    config = pluginmanager.hook.pytest_cmdline_parse(
  File "/usr/local/lib/python3.10/dist-packages/pluggy/_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  File "/usr/local/lib/python3.10/dist-packages/pluggy/_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/usr/local/lib/python3.10/dist-packages/pluggy/_callers.py", line 55, in _multicall
    gen.send(outcome)
  File "/usr/local/lib/python3.10/dist-packages/_pytest/helpconfig.py", line 103, in pytest_cmdline_parse
    config: Config = outcome.get_result()
  File "/usr/local/lib/python3.10/dist-packages/pluggy/_result.py", line 60, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/usr/local/lib/python3.10/dist-packages/pluggy/_callers.py", line 39, in _multicall
    res = hook_impl.function(*args)
  File "/usr/local/lib/python3.10/dist-packages/_pytest/config/__init__.py", line 1058, in pytest_cmdline_parse
    self.parse(args)
  File "/usr/local/lib/python3.10/dist-packages/_pytest/config/__init__.py", line 1346, in parse
    self._preparse(args, addopts=addopts)
  File "/usr/local/lib/python3.10/dist-packages/_pytest/config/__init__.py", line 1248, in _preparse
    self.hook.pytest_load_initial_conftests(
  File "/usr/local/lib/python3.10/dist-packages/pluggy/_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  File "/usr/local/lib/python3.10/dist-packages/pluggy/_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/usr/local/lib/python3.10/dist-packages/pluggy/_callers.py", line 60, in _multicall
    return outcome.get_result()
  File "/usr/local/lib/python3.10/dist-packages/pluggy/_result.py", line 60, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/usr/local/lib/python3.10/dist-packages/pluggy/_callers.py", line 39, in _multicall
    res = hook_impl.function(*args)
  File "/usr/local/lib/python3.10/dist-packages/_pytest/config/__init__.py", line 1125, in pytest_load_initial_conftests
    self.pluginmanager._set_initial_conftests(
  File "/usr/local/lib/python3.10/dist-packages/_pytest/config/__init__.py", line 560, in _set_initial_conftests
    self._try_load_conftest(current, namespace.importmode, rootpath)
  File "/usr/local/lib/python3.10/dist-packages/_pytest/config/__init__.py", line 579, in _try_load_conftest
    self._getconftestmodules(x, importmode, rootpath)
  File "/usr/local/lib/python3.10/dist-packages/_pytest/config/__init__.py", line 603, in _getconftestmodules
    mod = self._importconftest(conftestpath, importmode, rootpath)
  File "/usr/local/lib/python3.10/dist-packages/_pytest/config/__init__.py", line 648, in _importconftest
    assert mod not in mods
AssertionError

kristjanvalur commented 1 year ago

I am consistently seeing this on windows with https://github.com/strawberry-graphql/strawberry, python 3.10, pytest 7.3. Must be something with my machine, since CI passes for that project, but I have to pin pytest to 7.0 for local development.

chebee7i commented 1 year ago

Is the issue here that we haven't provided enough information to reproduce the issue? Or that this isn't deemed a serious enough issue? Or something else?

The title of this issue should be amended so that it is not indicating this is a Windows only issue. This issue also impacts linux.

RonnyPfannschmidt commented 1 year ago

The assertion needs to get a message, lets add a Bugfix that shows The Module name

kristjanvalur commented 1 year ago

What would the required change be to get more information? I can always do a local change.

Carreau commented 1 year ago

You can at least change assert mod not in mods to assert mod not in mods, f"{mod} seem to be loaded twice.".

kristjanvalur commented 1 year ago

, f"{mod} seem to be loaded twice."

In my case, I get this: AssertionError: <module 'tests.conftest' from 'E:\\git\\strawberry\\tests\\conftest.py'> seem to be loaded twice.

Note, this happens only when using the "tests" tab of VS code, and refreshing tests. running pytest from the terminal works fine.

Here's the full stack trace:

File "C:\Users\kristjan\AppData\Local\pypoetry\Cache\virtualenvs\strawberry-graphql-TghffK6u-py3.7\lib\site-packages\_pytest\config\__init__.py", line 1067, in pytest_cmdline_parse
    self.parse(args)
  File "C:\Users\kristjan\AppData\Local\pypoetry\Cache\virtualenvs\strawberry-graphql-TghffK6u-py3.7\lib\site-packages\_pytest\config\__init__.py", line 1354, in parse
    self._preparse(args, addopts=addopts)
  File "C:\Users\kristjan\AppData\Local\pypoetry\Cache\virtualenvs\strawberry-graphql-TghffK6u-py3.7\lib\site-packages\_pytest\config\__init__.py", line 1257, in _preparse
    early_config=self, args=args, parser=self._parser
  File "C:\Users\kristjan\AppData\Local\pypoetry\Cache\virtualenvs\strawberry-graphql-TghffK6u-py3.7\lib\site-packages\pluggy\_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  File "C:\Users\kristjan\AppData\Local\pypoetry\Cache\virtualenvs\strawberry-graphql-TghffK6u-py3.7\lib\site-packages\pluggy\_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "C:\Users\kristjan\AppData\Local\pypoetry\Cache\virtualenvs\strawberry-graphql-TghffK6u-py3.7\lib\site-packages\pluggy\_callers.py", line 60, in _multicall
    return outcome.get_result()
  File "C:\Users\kristjan\AppData\Local\pypoetry\Cache\virtualenvs\strawberry-graphql-TghffK6u-py3.7\lib\site-packages\pluggy\_result.py", line 60, in get_result
    raise ex[1].with_traceback(ex[2])
  File "C:\Users\kristjan\AppData\Local\pypoetry\Cache\virtualenvs\strawberry-graphql-TghffK6u-py3.7\lib\site-packages\pluggy\_callers.py", line 39, in _multicall
    res = hook_impl.function(*args)
  File "C:\Users\kristjan\AppData\Local\pypoetry\Cache\virtualenvs\strawberry-graphql-TghffK6u-py3.7\lib\site-packages\_pytest\config\__init__.py", line 1134, in pytest_load_initial_conftests
    early_config.known_args_namespace, rootpath=early_config.rootpath
  File "C:\Users\kristjan\AppData\Local\pypoetry\Cache\virtualenvs\strawberry-graphql-TghffK6u-py3.7\lib\site-packages\_pytest\config\__init__.py", line 556, in _set_initial_conftests
    self._try_load_conftest(anchor, namespace.importmode, rootpath)
  File "C:\Users\kristjan\AppData\Local\pypoetry\Cache\virtualenvs\strawberry-graphql-TghffK6u-py3.7\lib\site-packages\_pytest\config\__init__.py", line 578, in _try_load_conftest
    self._getconftestmodules(x, importmode, rootpath)
  File "C:\Users\kristjan\AppData\Local\pypoetry\Cache\virtualenvs\strawberry-graphql-TghffK6u-py3.7\lib\site-packages\_pytest\config\__init__.py", line 602, in _getconftestmodules
    mod = self._importconftest(conftestpath, importmode, rootpath)
  File "C:\Users\kristjan\AppData\Local\pypoetry\Cache\virtualenvs\strawberry-graphql-TghffK6u-py3.7\lib\site-packages\_pytest\config\__init__.py", line 647, in _importconftest
    assert mod not in mods, f"{mod} seem to be  loaded twice."
AssertionError: <module 'tests.conftest' from 'E:\\git\\strawberry\\tests\\conftest.py'> seem to be  loaded twice.

jgb commented 1 year ago

You can at least change assert mod not in mods to assert mod not in mods, f"{mod} seem to be loaded twice.".

For me it simply reports that the conftest.py in my project root is loaded twice. I'm on Ubuntu 22.04 LTS and python 3.10. pytest 7.0.1 works fine, but any version beyond that, including the latest 7.3.1, is completely broken.

See below, /srv/vidicenter is my project root:

AssertionError: <module 'vidicenter.conftest' from '/srv/vidicenter/../vidicenter/conftest.py'> seem to be  loaded twice.

Carreau commented 1 year ago

I guess now the question is where is it loaded the first time ? And which one if is "wrong" if any are wrong.

kristjanvalur commented 1 year ago

I guess that if this is the only place where loading is happening, I can add a traceback for my case when it successfully loads it the first time.

kristjanvalur commented 1 year ago

Yes, well, adding an assert here, where it is initalizing self._dirpath2confmods:

        for parent in reversed((directory, *directory.parents)):
            if self._is_in_confcutdir(parent):
                conftestpath = parent / "conftest.py"
                if conftestpath.is_file():
                    mod = self._importconftest(conftestpath, importmode, rootpath)
                    clist.append(mod)
                    if "'tests.conftest'" in str(mod):
                        assert False, (mod, conftestpath, rootpath)

File "C:\Users\kristjan\AppData\Local\pypoetry\Cache\virtualenvs\strawberry-graphql-TghffK6u-py3.7\lib\site-packages\_pytest\config\__init__.py", line 1141, in pytest_load_initial_conftests
    early_config.known_args_namespace, rootpath=early_config.rootpath
  File "C:\Users\kristjan\AppData\Local\pypoetry\Cache\virtualenvs\strawberry-graphql-TghffK6u-py3.7\lib\site-packages\_pytest\config\__init__.py", line 556, in _set_initial_conftests
    self._try_load_conftest(anchor, namespace.importmode, rootpath)
  File "C:\Users\kristjan\AppData\Local\pypoetry\Cache\virtualenvs\strawberry-graphql-TghffK6u-py3.7\lib\site-packages\_pytest\config\__init__.py", line 573, in _try_load_conftest
    self._getconftestmodules(anchor, importmode, rootpath)
  File "C:\Users\kristjan\AppData\Local\pypoetry\Cache\virtualenvs\strawberry-graphql-TghffK6u-py3.7\lib\site-packages\_pytest\config\__init__.py", line 605, in _getconftestmodules
    assert False, (mod, conftestpath, rootpath)
AssertionError: (<module 'tests.conftest' from 'E:\\git\\strawberry\\tests\\conftest.py'>, WindowsPath('e:/git/strawberry/tests/conftest.py'), WindowsPath('e:/git/strawberry'))

This is where it is added to the _dirpath2confmods thing first.

petebman commented 1 year ago

I've encountered what I think is the same problem without any plugins being involved. I'm struggling to make a repro that doesn't involve the rest of our build system but it seems like the problem can happen when the test depends on a module that's farther down in the source tree. For example:

repro
  +- conftest.py
  +- problematic_test.py
  +- peer_dep.py
  +- nested_dep_folder
    +- __init__.py
    +- nested_dep.py

If problematic_test.py does something like

from path.to.repro.nested_dep_folder import nested_dep

def test_to_repro_problem():
  assert nested_dep.add_two(2) == 4

Then it seems like conftest.py gets loaded twice and blows up the second time. I put a breakpoint near assert mod not in mods and see that the _importconftest gets called twice. The first time it gets called I see pytest_load_initial_conftests in the backtrace (just like @kristjanvalur posted above) The second time it gets called I see pytest_make_collect_report in the backtrace and then it blows up with assert mod not in mods.

If I use peer_dep.py instead of the dependency in the nested folder then I still see the call _importconftest with pytest_load_initial_conftests in the backtrace but I don't see the second call to _importconftest that blows up.

This problem goes away if we roll back to an earlier (pre 7) version of pytest.

followben commented 1 year ago

I too confirm on Windows 10 that pytest 7.0.1 collection works fine but upgrading to 7.1.0 or later breaks collection, falling over due to loading our single conftest.py twice. Given the same codebase and dependencies, this doesn't occur on linux or mac for pytest 7.1.0 or later.

jgb commented 11 months ago

Hello,

any news on this topic? Pytest beyond version 7.0.1 is still broken for us on linux.

Greetings,

jgb

fcharras commented 11 months ago

Also see this problem on Windows 11 in the miniforge prompt, with up-to-date scikit-learn installed from source in editable mode, in a conda environment. After some research I found that the issue arise if I install scikit-learn with

pip install scikit-learn --no-build-isolation --no-use-pep517 -e .

but if just installing with

pip install scikit-learn --no-build-isolation -e .

it will not showcase the issue.

The difference between the two installs seems to be that on the first case the site-packages directory gets only an egg-link file and the root of the package gets a egg-info folder, the second one gets only a dist-info folder in site-packages. That somewhat echoes the feedback of another user that hints at an issue with symlinks in site-packages.

I tried to inspect the sequence of instructions that leads to the double load but without luck so far (with the same conclusions that the other users here).

fcharras commented 11 months ago

After thorough investigation with @lesteve slowly comparing the loading steps from both two environments on windows, one of those showcasing the issue, the other being sane, we found that for some reason the paths of the conftest files are sometimes small cased when used as keys in the name2plugin dictionary, but then the same small-casing is not applied at inspection time. This leads to the double loading of plugins, and the subsequent crash.

https://github.com/pytest-dev/pytest/pull/11708 fixes that.

Could other users confirm that the fix work for them too ?

jgb commented 11 months ago

@fcharras @bluetech I can confirm that this solves the problem for me on Linux!

fcharras commented 11 months ago

Great news! by any chance, are you using linux with a case-insensitive file system (ntfs or vfat partition maybe ?)

If not, then maybe your example on linux could help craft a better non-regression test, it would be interesting to know what you see when inspecting the variables str(conftestpath) and mod.__file__ the last time it's called before the crash. (maybe by inserting a print there and reporting the last two occurences)

RonnyPfannschmidt commented 11 months ago

@bluetech i just did a little crosschecking - it seems that py.path.local had an awful lot of norm-path calls all over the place + extra .lower calls for eq/hash on win32, so i suspect this issue comes about from replacing accidental normalize with missed normalize

jgb commented 11 months ago

@fcharras I'm using a regular ext4 case-sensitive FS on linux. These are the variables you asked for:

conftestpath=PosixPath('/srv/vidicenter/conftest.py') mod.__file__='/srv/vidicenter/../vidicenter/conftest.py'

bluetech commented 10 months ago

@fcharras I looked at your PR but I am not fully confident in merging it; it definitely fixes the issue with the assert but might lead to other subtle issues in places which consider the paths (the collection tree & conftests are very dependent on the filesystem).

@jgb Very interesting that you get it in linux, that means I could possibly reproduce and understand where the issue comes from. What would be very helpful is to get a reproducible example. I'm interested to know the origin of the two conflicting paths -- user input, filesystem API, Python's import system, etc. Basically I need:

OS (you said linux)
Filesystem type (you said ext4)
Filesystem layout
Which directory pytest is invoked from
pytest flags if any
pytest config file if any
Relevant environment variables if any

I tried myself to get to a mod.__file__='/srv/vidicenter/../vidicenter/conftest.py' situation with the details you provided in various ways but couldn't.

Another angle I was thinking about: pytest gets two paths to the same conftest, which it considers different. I'd like to understand why (see above), but let's take it as a fact for the sake of discussion. Meanwhile, the Python import system considers the paths to resolve to the same module. This means pytest and Python are not consistent in how they resolve paths to modules. I tried to understand how the Python import system normalizes paths, I read the docs and read some of the code, but man is it complex, I gave up for now.

Carreau commented 10 months ago

For what it is worth, I've unpinned on IPython and the error does not shows up anymore, though there might have been other changes in what IPython since the issue was opened, so there might (is?) likely multiple things that affects this bug.

bluetech commented 10 months ago

Thanks to the reproduction provided by @lesteve in https://github.com/pytest-dev/pytest/pull/11708#issuecomment-1868384602, I was able to reproduce the issue by running it in Wine.

Problem

The problem is this:

First _importconftest call:
- The input path is Z:\pytest-issue-9765-reproducer\project\conftest.py.
- pytest puts Z:\pytest-issue-9765-reproducer\project\ in sys.path, calculates the python import path for it project.conftest and does mod = importlib.import_module("project.conftest"). See function import_path.
- pytest registers plugin with name mod.__file__, which is z:\pytest-issue-9765-reproducer\project\conftest.py. Note the lower-case z:\. The lower-case drive name only happens with --no-use-pep517 --no-build-isolation, I haven't tried to understand why.
Second _importconftest call:
- The input path is Z:\pytest-issue-9765-reproducer\project\conftest.py (same).
- pytest checks if a plugin is already registered with name str(conftestpath), which is Z:\pytest-issue-9765-reproducer\project\conftest.py (upper case Z). Nope.
- pytest loads the conftest file again -> assert.

Interestingly, the problem does not occur when using pytest --import-mode=importlib because this mode is less lossy and preserves the path better.

Solution 1

Part of the solution proposed by @fcharras in #11708.

The inconsistency is that we check if the plugin exists using name str(conftestpath) but register it using mod.__file__. So instead, register the plugin using str(conftestpath). This fixes the issue in the reproduction repo for me.

Problem with this approach: the mod.__file__ still remains "broken"; I'm not sure if it will actually cause problems but I feel uneasy with it. (Just from a quick look, _getconftest_pathlist seems to use it...).

Solution 2

Full solution proposed by @fcharras.

In addition to 1, also fix inconsistencies in the inputs - that is, if the conftestpaths themselves are e.g. one time Z:/ and one time z:/. The inconsistencies here from case-insensitive filesystems and unnormalized paths (/foo/../foo/conftest.py). So instead of str(conftestpath) use normcase(normpath(conftestpath)).

The reproduction doesn't display this and I'm not sure how this one happens, but let's assume that it does...

If we assume the python import system keeps normalized paths in tact, this should ~fix the problem with just Solution 1.

Problem with this approach: the conftestpath will get out of sync with _dirpath2confmods which I think will cause issues; the code assumes they're consistent.

Solution 3

Normalize conftest paths as early as possible. Technically: change _get_directory function to _get_conftest_directory and make it normalize.

I think this should work pretty well, and I think will be less error prone, even if we keep using __file__ for registering.

Solution 4

4 = 3 + 1

Just to be safe against lossy __file__ issue, we can do 3 and also 1.

@fcharras I think we should go for Solution 4; WDYT?

jgb commented 10 months ago

@jgb Very interesting that you get it in linux, that means I could possibly reproduce and understand where the issue comes from. What would be very helpful is to get a reproducible example. I'm interested to know the origin of the two conflicting paths -- user input, filesystem API, Python's import system, etc. Basically I need:

OS (you said linux)

Filesystem type (you said ext4)

Filesystem layout

Which directory pytest is invoked from

pytest flags if any

pytest config file if any

Relevant environment variables if any

I tried myself to get to a mod.__file__='/srv/vidicenter/../vidicenter/conftest.py' situation with the details you provided in various ways but couldn't.

@bluetech Filesystem layout: all code resides in /srv/vidicenter, we've got all of these conftest.py files:

conftest.py newvidicenter/analytics/tests/conftest.py newvidicenter/api/tests/conftest.py newvidicenter/campaigns/tests/conftest.py newvidicenter/downloads/tests/conftest.py newvidicenter/remoteconfig/tests/conftest.py newvidicenter/syncers/django2es/tests/conftest.py newvidicenter/terms/tests/conftest.py newvidicenter/webservices/tests/conftest.py

This is my pytest.ini:

[pytest]
DJANGO_SETTINGS_MODULE=vidicenter.test_settings
addopts = -v --reuse-db --ignore=test-requirements.txt --ignore=tests --ignore=test_api_settings.py -m "not performance and not wsgi" --driver=Chrome --driver-path=binaries/chromedriver
norecursedirs = servers/soap .git newvidicenter/core/static static_root media tmp
pythonpath = ..
python_files = test_*.py asserts.py mixins.py

Pytest is invoked from the git root (/srv/vidicenter), usually as pytest -n4.

Hope this helps!

bluetech commented 10 months ago

Thanks @jgb. Am I guessing correctly that you added pythonpath = .., where .. is /srv, so that you could do import vidicenter.foo and have it resolve to /srv/vidicenter/foo.py?

jgb commented 10 months ago

Thanks @jgb. Am I guessing correctly that you added pythonpath = .., where .. is /srv, so that you could do import vidicenter.foo and have it resolve to /srv/vidicenter/foo.py?

Yes, correct! If there's anything else I can do to help out, don't hesitate to ask!

bluetech commented 10 months ago

@jgb A few more questions:

Is there a /srv/vidicenter/__init__.py file?
Which python version?
Which pytest version?
Is there a chance something is importing /srv/vidicenter/conftest.py before pytest does it?

jgb commented 10 months ago

@jgb A few more questions:

Is there a /srv/vidicenter/__init__.py file?

Which python version?

Which pytest version?

Is there a chance something is importing /srv/vidicenter/conftest.py before pytest does it?

@bluetech

yes there's a /srv/vidicenter/__init.py__ file
python 3.11
pytest 7.0.1 (the last one that works, everything newer breaks for us)
I did a quick grep in the code, the only thing I see is that in 3 different tests, we import a constant from our root conftest.py

bluetech commented 10 months ago

OK so I'm getting close to reproducing your case, but still can't. pytest imports the conftest like this (ImportMode.prepend is the default):

https://github.com/pytest-dev/pytest/blob/52db918a27b2eb5043de6e80215076a98b0b9fff/src/_pytest/pathlib.py#L555-L567

Before we enter this code the sys.path is ["/srv/vidicenter/..", ...] because of the pythonpath = .. in pytest.ini. The pkg_root is /srv, so right before the import_module call the sys.path becomes ["/srv", "/srv/vidicenter/..", ..]. Then the import_module("vidicenter.conftest") call resolves using the "/srv" entry, not the "/srv/vidicenter/.." entry which comes later, so the mod.__file__ is good.

However, if the vidicenter.conftest module is already somehow imported from python code, then it was resolved using "/srv/vidicenter/.." and its mod.__file__ will have the ... But the /srv/vidicenter/conftest.py is an "initial conftest", which is loaded very early, so I'm not sure how something could have imported the conftest before it.

If you can try to put a breakpoint() at the top of your conftest.py file and see what imports it that would be interesting. Maybe it's pytest-django that's doing it.

jgb commented 10 months ago

@bluetech this is what I get with a breakpoint at the top of my root conftest.py:

(vidicenter) root@14625dcd911b:/srv/vidicenter# pytest
--Return--
None
> <frozen importlib._bootstrap>(241)_call_with_frames_removed()

ipdb> WARNING No rollbar access token provided
ImportError while loading conftest '/srv/vidicenter/conftest.py'.
/root/.venv/vidicenter/lib/python3.11/site-packages/ddtrace/internal/module.py:225: in _exec_module
    self.loader.exec_module(module)
conftest.py:1: in <module>
    import debug
/root/.venv/vidicenter/lib/python3.11/site-packages/ddtrace/internal/module.py:225: in _exec_module
    self.loader.exec_module(module)
/usr/lib/python3.11/bdb.py:94: in trace_dispatch
    return self.dispatch_return(frame, arg)
/usr/lib/python3.11/bdb.py:153: in dispatch_return
    self.user_return(frame, arg)
/usr/lib/python3.11/pdb.py:369: in user_return
    self.interaction(frame, None)
/root/.venv/vidicenter/lib/python3.11/site-packages/IPython/core/debugger.py:455: in interaction
    OldPdb.interaction(self, frame, tb)
/usr/lib/python3.11/pdb.py:433: in interaction
    self._cmdloop()
/usr/lib/python3.11/pdb.py:397: in _cmdloop
    self.cmdloop()
/usr/lib/python3.11/cmd.py:126: in cmdloop
    line = input(self.prompt)
E   OSError: pytest: reading from stdin while output is captured!  Consider using `-s`.

RonnyPfannschmidt commented 10 months ago

The exception tells what to do, the debugger you use is not pytest aware so you need to disable io capture

pytest-dev / pytest