search

microsoft / vscode-python

Python extension for Visual Studio Code
https://aka.ms/pvsc-marketplace
MIT License
4.31k stars 1.18k forks source link

[Pytest] Tests not being discovered due to pytest_report hook #22650

Closed lubaroli closed 5 months ago

lubaroli commented 9 months ago

Type: Bug

Behaviour

After updating VS Code, VS Code Pytest is no longer discovering tests. Running Pytest on the terminal works as expected and tests are found correctly.

Expected vs. Actual

When running on ssh terminal, within Conda environment:

(kedro_dataiku) xxxxxx@xxxxxx:/ace/src/project_ace$ pytest --disable-warnings --collect-only
================================================= test session starts ==================================================
platform linux -- Python 3.7.16, pytest-3.10.1, py-1.10.0, pluggy-0.13.1
rootdir: /ace/src, inifile:
plugins: dash-1.19.0, anyio-2.2.0, typeguard-2.13.3, mock-1.13.0, cov-2.9.0
collected 1071 items

Output of VS Code after failing to discover tests:

2023-12-13 23:32:55.024 [info] > conda run -n kedro_dataiku --no-capture-output python ~/.vscode-server/extensions/ms-python.python-2023.22.0/pythonFiles/get_output_via_markers.py -m pytest -p vscode_pytest --collect-only
2023-12-13 23:32:55.024 [info] cwd: ./src/project_ace
2023-12-13 23:32:55.907 [info] >>>PYTHON-EXEC-OUTPUT

2023-12-13 23:32:56.172 [info] <<<PYTHON-EXEC-OUTPUT

2023-12-13 23:32:56.172 [error] Traceback (most recent call last):
  File "/home/xxxxx/.vscode-server/extensions/ms-python.python-2023.22.0/pythonFiles/get_output_via_markers.py", line 23, in <module>
    runpy.run_module(moduleName, run_name="__main__", alter_sys=True)
  File "/home/xxxxx/.conda/envs/kedro_dataiku/lib/python3.7/runpy.py", line 205, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/home/xxxxx/.conda/envs/kedro_dataiku/lib/python3.7/runpy.py", line 96, in _run_module_code

2023-12-13 23:32:56.172 [error]     mod_name, mod_spec, pkg_name, script_name)
  File "/home/xxxxx/.conda/envs/kedro_dataiku/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/xxxxx/.conda/envs/kedro_dataiku/lib/python3.7/site-packages/pytest.py", line 91, in <module>
    raise SystemExit(pytest.main())
  File "/home/xxxxx/.conda/envs/kedro_dataiku/lib/python3.7/site-packages/_pytest/config/__init__.py", line 58, in main

2023-12-13 23:32:56.172 [error]     config = _prepareconfig(args, plugins)
  File "/home/xxxxx/.conda/envs/kedro_dataiku/lib/python3.7/site-packages/_pytest/config/__init__.py", line 196, in _prepareconfig

2023-12-13 23:32:56.172 [error]     pluginmanager=pluginmanager, args=args
  File "/home/xxxxx/.conda/envs/kedro_dataiku/lib/python3.7/site-packages/pluggy/hooks.py", line 286, in __call__

2023-12-13 23:32:56.172 [error]     return self._hookexec(self, self.get_hookimpls(), kwargs)

2023-12-13 23:32:56.172 [error]   File "/home/xxxxx/.conda/envs/kedro_dataiku/lib/python3.7/site-packages/pluggy/manager.py", line 93, in _hookexec

2023-12-13 23:32:56.172 [error]     return self._inner_hookexec(hook, methods, kwargs)
  File "/home/xxxxx/.conda/envs/kedro_dataiku/lib/python3.7/site-packages/pluggy/manager.py", line 87, in <lambda>

2023-12-13 23:32:56.172 [error]     firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
  File "/home/xxxxx/.conda/envs/kedro_dataiku/lib/python3.7/site-packages/pluggy/callers.py", line 203, in _multicall

2023-12-13 23:32:56.172 [error]     gen.send(outcome)
  File "/home/xxxxx/.conda/envs/kedro_dataiku/lib/python3.7/site-packages/_pytest/helpconfig.py", line 93, in pytest_cmdline_parse

2023-12-13 23:32:56.172 [error]     config = outcome.get_result()
  File "/home/xxxxx/.conda/envs/kedro_dataiku/lib/python3.7/site-packages/pluggy/callers.py", line 80, in get_result

2023-12-13 23:32:56.172 [error]     raise ex[1].with_traceback(ex[2])
  File "/home/xxxxx/.conda/envs/kedro_dataiku/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall

2023-12-13 23:32:56.172 [error]     res = hook_impl.function(*args)
  File "/home/xxxxx/.conda/envs/kedro_dataiku/lib/python3.7/site-packages/_pytest/config/__init__.py", line 675, in pytest_cmdline_parse

2023-12-13 23:32:56.173 [error]     self.parse(args)
  File "/home/xxxxx/.conda/envs/kedro_dataiku/lib/python3.7/site-packages/_pytest/config/__init__.py", line 845, in parse

2023-12-13 23:32:56.173 [error]     self._preparse(args, addopts=addopts)
  File "/home/xxxxx/.conda/envs/kedro_dataiku/lib/python3.7/site-packages/_pytest/config/__init__.py", line 795, in _preparse

2023-12-13 23:32:56.173 [error]     self.pluginmanager.consider_preparse(args)
  File "/home/xxxxx/.conda/envs/kedro_dataiku/lib/python3.7/site-packages/_pytest/config/__init__.py", line 475, in consider_preparse

2023-12-13 23:32:56.173 [error]     self.consider_pluginarg(opt2)
  File "/home/xxxxx/.conda/envs/kedro_dataiku/lib/python3.7/site-packages/_pytest/config/__init__.py", line 489, in consider_pluginarg

2023-12-13 23:32:56.173 [error]     self.import_plugin(arg)
  File "/home/xxxxx/.conda/envs/kedro_dataiku/lib/python3.7/site-packages/_pytest/config/__init__.py", line 537, in import_plugin

2023-12-13 23:32:56.173 [error]     self.register(mod, modname)
  File "/home/xxxxx/.conda/envs/kedro_dataiku/lib/python3.7/site-packages/_pytest/config/__init__.py", line 306, in register
    ret = super(PytestPluginManager, self).register(plugin, name)
  File "/home/xxxxx/.conda/envs/kedro_dataiku/lib/python3.7/site-packages/pluggy/manager.py", line 126, in register

2023-12-13 23:32:56.173 [error]     self._verify_hook(hook, hookimpl)
  File "/home/xxxxx/.conda/envs/kedro_dataiku/lib/python3.7/site-packages/pluggy/manager.py", line 261, in _verify_hook

2023-12-13 23:32:56.173 [error]     notinspec,
pluggy.manager.PluginValidationError: Plugin 'vscode_pytest' for hook 'pytest_report_teststatus'
hookimpl definition: pytest_report_teststatus(report, config)
Argument(s) {'config'} are declared in the hookimpl but can not be found in the hookspec

Steps to reproduce:

  1. Configure tests with Pytest
  2. Refresh test discovery

Diagnostic data

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

``` XXX ```

User Settings

``` condaPath: "" languageServer: "Pylance" testing • cwd: "" • pytestEnabled: true ```

Extension version: 2023.22.1 VS Code version: Code 1.85.1 (0ee08df0cf4527e40edc9aa28f4b5bd38bbff2b2, 2023-12-13T09:49:37.021Z) OS version: Windows_NT x64 10.0.19045 Modes: Remote OS version: Linux x64 3.10.0-1160.99.1.el7.x86_64 Remote OS version: Linux x64 3.10.0-1160.99.1.el7.x86_64

System Info |Item|Value| |---|---| |CPUs|12th Gen Intel(R) Core(TM) i7-12850HX (24 x 2419)| |GPU Status|2d_canvas: enabled
canvas_oop_rasterization: enabled_on
direct_rendering_display_compositor: disabled_off_ok
gpu_compositing: enabled
multiple_raster_threads: enabled_on
opengl: enabled_on
rasterization: enabled
raw_draw: disabled_off_ok
video_decode: enabled
video_encode: enabled
vulkan: disabled_off
webgl: enabled
webgl2: enabled
webgpu: enabled| |Load (avg)|undefined| |Memory (System)|63.67GB (40.83GB free)| |Process Argv|--crash-reporter-id d642995a-15db-49b9-871b-7574d0bace4e| |Screen Reader|no| |VM|0%| |Item|Value| |---|---| |Remote|SSH: ace| |OS|Linux x64 3.10.0-1160.99.1.el7.x86_64| |CPUs|Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz (48 x 2899)| |Memory (System)|371.81GB (297.91GB free)| |VM|0%| |Item|Value| |---|---| |Remote|Container dataiku-base-ace-lbarcelo (reverent_saha) @ ace| |OS|Linux x64 3.10.0-1160.99.1.el7.x86_64| |CPUs|Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz (48 x 2899)| |Memory (System)|371.81GB (297.91GB free)| |VM|0%|
A/B Experiments ``` vsliv368:30146709 vsreu685:30147344 python383:30185418 vspor879:30202332 vspor708:30202333 vspor363:30204092 vslsvsres303:30308271 vserr242:30382549 pythontb:30283811 vsjup518:30340749 pythonptprofiler:30281270 vsdfh931:30280409 vshan820:30294714 vstes263:30335439 vscod805cf:30301675 binariesv615:30325510 bridge0708:30335490 bridge0723:30353136 vsaa593cf:30376535 pythonvs932:30410667 py29gd2263:30899288 vsclangdf:30486550 c4g48928:30535728 dsvsc012:30540252 azure-dev_surveyone:30548225 282f8724:30602487 89544117:30613380 vscrp:30673768 showlangstatbar:30737416 0bi6i642:30917235 fixshowwlkth:30771522 showindicator:30805244 pythongtdpath:30769146 i26e3531:30792625 welcomedialog:30910333 pythonnosmt12:30797651 pythonidxpt:30866567 pythonnoceb:30805159 asynctok:30898717 dsvsc013:30795093 dsvsc014:30804076 dsvsc015:30845448 pythontestfixt:30902429 pyreplss1:30897532 pythonmypyd1:30879173 pythoncet0:30885854 pythontbext0:30879054 dsvsc016:30899300 dsvsc017:30899301 dsvsc018:30899302 aa_t_chat:30882232 ```
eleanorjboyd commented 9 months ago

Hi! From my understanding what this error message {'config'} are declared in the hookimpl but can not be found in the hookspec is saying is that the config argument is in misalignment. Specifically that the vscode-pytest plugin implementation of pytest_report_teststatus has argument {'config'} but not in the hookspec or the definition outlines what parameters this function should accept.

Referring to pytest documentation here, it shows that config is defined as an argument.

I am seeing from your stdout that your version are Python 3.7.16, pytest-3.10.1, py-1.10.0, pluggy-0.13.1. Pytest is not on version 7.4.3 and Python 3.7 is at end of life so could you try updating both of those because this is likely the problem. Thanks

vanackere commented 9 months ago

Hello,

I'm encountering the same bug (also python 3.7 but cannot upgrade the version since this is for a maintenance branch of a product and that's where dev-containers are useful !)

2023-12-15 09:04:21.566 [info] Experiment 'pythonRecommendTensorboardExtcf' is active
2023-12-15 09:04:21.566 [info] Experiment 'pythonREPLSmartSend' is active
2023-12-15 09:04:21.566 [info] Experiment 'pythonTestAdapter' is active
2023-12-15 09:04:21.566 [info] Default formatter is set to null for workspace /workspaces/wab
2023-12-15 09:04:21.566 [info] Test server listening.
2023-12-15 09:04:21.566 [info] Python interpreter path: /usr/bin/python3
2023-12-15 09:04:32.633 [info] Starting Pylance language server.
2023-12-15 09:04:50.285 [info] Discover tests for workspace name: wab - uri: /workspaces/wab
2023-12-15 09:04:50.285 [info] Running discovery for pytest using the new test adapter.
2023-12-15 09:04:50.286 [info] All environment variables set for pytest discovery for workspace /workspaces/wab: {"PYTHONPATH":"/home/dev/.vscode-server/extensions/ms-python.python-2023.22.1/pythonFiles:srcbuilt:src[...],"VSCODE_AGENT_FOLDER":"/home/dev/.vscode-server","SHELL":"/bin/bash","PWD":"/vscode/vscode-server/bin/linux-x64/0ee08df0cf4527e40edc9aa28f4b5bd38bbff2b2","VSCODE_HANDLES_SIGPIPE":"true","VSCODE_AMD_ENTRYPOINT":"vs/workbench/api/node/extensionHostProcess","VSCODE_HANDLES_UNCAUGHT_ERRORS":"true","VSCODE_NLS_CONFIG":"{\"locale\":\"en\",\"osLocale\":\"en\",\"availableLanguages\":{}}","REMOTE_CONTAINERS_IPC":"/tmp/vscode-remote-containers-ipc-6dc7686f-cb87-4d92-b5fc-e93baef610d2.sock",[...],"REMOTE_CONTAINERS_SOCKETS":"[\"/tmp/vscode-ssh-auth-6dc7686f-cb87-4d92-b5fc-e93baef610d2.sock\",\"/tmp/.X11-unix/X1\"]","TEST_UUID":"2089bcb5-b3e5-433d-a2ba-c27f3960589f","TEST_PORT":"36053"} 

2023-12-15 09:04:50.416 [info] > /usr/bin/python3 -m pytest -p vscode_pytest --collect-only -s --import-mode=append --disable-warnings
2023-12-15 09:04:50.417 [info] cwd: .
2023-12-15 09:04:51.022 [error] Traceback (most recent call last):

2023-12-15 09:04:51.022 [error]   File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main

2023-12-15 09:04:51.022 [error]     
2023-12-15 09:04:51.022 [error] "__main__", mod_spec)
2023-12-15 09:04:51.022 [error] 

2023-12-15 09:04:51.023 [error]   File "/usr/lib/python3.7/runpy.py", line 85, in _run_code

2023-12-15 09:04:51.023 [error]     
2023-12-15 09:04:51.030 [error] exec(code, run_globals)
2023-12-15 09:04:51.030 [error] 

2023-12-15 09:04:51.030 [error]   File "/usr/lib/python3/dist-packages/pytest.py", line 91, in <module>

2023-12-15 09:04:51.031 [error]     
2023-12-15 09:04:51.039 [error] raise SystemExit(pytest.main())

This is probably related to "Experiment 'pythonTestAdapter' is active" ? Anyway it would be nice to have a fix other than "upgrade the python version" if possible ;-)

vanackere commented 9 months ago

I can also confirm that setting "python.experiments.optOutFrom": [ "pythonTestAdapter" ] fixes the issue

peacefulseeker commented 9 months ago

I can also confirm that setting "python.experiments.optOutFrom": [ "pythonTestAdapter" ] fixes the issue

same here(with no obvious error in python output, though). for platform darwin -- Python 3.7.17, pytest-7.4.3, pluggy-1.2.0 environment opting out from pythonTestAdapter helped to discover tests in Vscode Testing section. (make sure to put that in user settings, not workspace settings and reload the VSCode to respect new settings)

Before (and after) that running pytest --collect-only in the terminal had the expected output, so it was the testing UI only that did not pick up discovered tests.

eleanorjboyd commented 9 months ago

@peacefulseeker, are you no longer seeing the same error regarding using python 3.7? If you are not please create a new issue with your project structure and logs so I can better assist, thanks!

eleanorjboyd commented 9 months ago

@vanackere in terms of a fix you can also pin the python extension version which is compatible with 3.7:

Hello! For the python extension we follow python.org and their schedule on which version we support. Here is their supported versions, https://devguide.python.org/versions/, and 3.7 has reached end of life for support. This means new releases of our extension will be compatible back to 3.8 as of now. As a work around, you can pin the version of the python extension which still worked for you with 3.7. You can see in the release notes of v2023.18.0 we dropped Python 3.7 support https://github.com/microsoft/vscode-python/releases so the previous version might be the best version to pin.

peacefulseeker commented 9 months ago

@peacefulseeker, are you no longer seeing the same error regarding using python 3.7? If you are not please create a new issue with your project structure and logs so I can better assist, thanks!

Yes, the issue is gone(for pytest) after I've opted out from pythonTestAdapter experiment, in particular, put the "python.experiments.optOutFrom": [ "pythonTestAdapter" ] in the settings.json

eleanorjboyd commented 9 months ago

Are you able to get it working still in the experiment? We will soon switch to fully using the rewrite as our testing infrastructure which will disable the use of the experiment opt out as a work around.

vanackere commented 9 months ago

@eleanorjboyd : thanks for the hint, we'll try to pin the version in devcontainer.json

My only fear with that approach is that the pinned extension might stop working with future version of VSCode, which will also be a problem for lot of other users working on legacy code base. For exemple Python 3.7 was the default python version on Debian 10 (end-of-life too... but on which one of our products has still a stable supported branch that we must maintain for several more years). How long can we be certain that the pinned extension will still be functional ? If there is a risk that the older extension would stop working, perhaps Microsoft could consider publishing a "Legacy" python extension that would still allow to work with older unsupported versions of python ?

eleanorjboyd commented 9 months ago

Looping in @brettcannon here as he can speak better to the future of our extension and VS Code as it relates to legacy python versions.

brettcannon commented 9 months ago

How long can we be certain that the pinned extension will still be functional ?

No explicit guarantees, but VS Code itself takes API stability seriously, so there's no specific expectation an older version would stop working.

perhaps Microsoft could consider publishing a "Legacy" python extension that would still allow to work with older unsupported versions of python ?

We implicitly already do that via the ability to install old versions of the extension. Otherwise maintaining support w/ older versions of Python becomes too costly time-wise based on the usage we see of old Python versions.

vanackere commented 9 months ago

What is the proper way to pin the version of the extension ?

I've tried putting 

"extensions": [
                "ms-python.python@2023.16.0",

In devcontainer.json but still end up with the latest version.

Also should I also pin ms-python.vscode-pylance to a specific version ?

peacefulseeker commented 9 months ago

Are you able to get it working still in the experiment? We will soon switch to fully using the rewrite as our testing infrastructure which will disable the use of the experiment opt-out as a workaround.

No, I was not able. Makes sense, that at a certain point, the test discovery is expected to be broken and py3.7 support will be completely dropped, although will still be possible to install previous versions of extensions 👍

peacefulseeker commented 9 months ago

@vanackere , you should be able to use Install another version... in a settings dropdown when you click the cogwheel icon in the extensions search view. Then pick the desired version and reload the editor.

vanackere commented 9 months ago

@peacefulseeker : manual installation works... but that's not really helpful since we need to ensure in the devcontainer.json file that the correct extension version is actually installed and don't want to require that every engineer in our company has to perform a manual setup each time they have to work on the legacy branch of our product...

brettcannon commented 9 months ago

What is the proper way to pin the version of the extension ?

I've tried putting

"extensions": [
              "ms-python.python@2023.16.0",

In devcontainer.json but still end up with the latest version.

If your version of VS Code is too old then you may be hitting https://github.com/microsoft/vscode/issues/63903 , otherwise it should work and you're hitting a bug in either VS Code or the devcontainer extension.

Also should I also pin ms-python.vscode-pylance to a specific version ?

I honestly don't know for your specific case; Pylance has its own compatibility policy. But since they don't execute Python code to function, I suspect they can be more conservative than us.

vanackere commented 9 months ago

@brettcannon version of vscode is quite recent (Version: 1.85.1 Commit: 0ee08df0cf4527e40edc9aa28f4b5bd38bbff2b2)

This must be a bug in devcontainer extension then, i'll investigate and report

eleanorjboyd commented 9 months ago

@lubaroli, please let me know if your issue has been resolved in this discussion and if we can close the issue. thanks

lubaroli commented 9 months ago

@lubaroli, please let me know if your issue has been resolved in this discussion and if we can close the issue. thanks

Thanks for the quick support @eleanorjboyd, I'm in exactly the same boat as @vanackere, working on legacy systems, but since python 3.7 is reaching EOL, I understand this is not a priority requirement and the issue can be closed. For now I'll pin the devcontainer extension as suggested and hope it is a feasible alternative for as long as it takes us to migrate our system.

vanackere commented 9 months ago

@brettcannon as a follow-up I can confirm that there is a bug that makes extension pinning not work in our case.

I've reduced it to a very small reproducible case and opened an issue at https://github.com/microsoft/vscode/issues/203457, I hope this can be prioritized and fixed soon (hopefully before the testing rewrite lands and fully breaks python 3.7 without any possible workaround for us :cold_sweat: )

eleanorjboyd commented 9 months ago

@karthiknadig looping you in on this as you might have more understanding around the extension pinning

brettcannon commented 9 months ago

I've reduced it to a very small reproducible case and opened an issue

Thanks for finding that bug. I can't make any promises around prioritizing that issue, but I will see what I can do.

eleanorjboyd commented 5 months ago

closing as the issue opened on core in regards to extension pinning has been resolved. Thanks!