search

microsoft / vscode-python

Python extension for Visual Studio Code
https://aka.ms/pvsc-marketplace
MIT License
4.29k stars 1.18k forks source link

Rewrite testing adapter #17242

Open kimadeline opened 3 years ago

kimadeline commented 3 years ago

Meta-issue: https://github.com/microsoft/vscode-python/issues/16936

Unittest

Pytest

Cleanup

Notes

Public issues

  1. https://github.com/microsoft/vscode-python/issues/8678
  2. https://github.com/microsoft/vscode-python/discussions/16565
  3. https://github.com/microsoft/vscode-python/issues/16929
  4. https://github.com/microsoft/vscode-python/issues/17107
  5. https://github.com/microsoft/vscode-python/issues/17903
  6. https://github.com/microsoft/vscode-python/issues/17957

Issues we expect this rewrite will fix

pytest feedback on how to things the "right way"

unittest missing features

Bilalh commented 2 years ago

It would be nice if other python testing frameworks such as behave (a BDD framework) https://behave.readthedocs.io/en/stable/index.html were supported by the testing adapter.

kimadeline commented 2 years ago

Hi @Bilalh, support for behave is not within scope of this issue. Feel free to open a feature request on the repo and we'll take a look at it. Thanks!

1st commented 2 years ago

https://github.com/microsoft/vscode-python/issues/18431

kimadeline commented 2 years ago

Hi @1st, please open a separate issue with your specific details, this issue is here to track work in progress. Thanks!

1st commented 2 years ago

Can you please release this change? It's better to split work and release incrementally, because over the time it becomes really hard to release a huge change.

I expect to see that when message is written to stderr the test discovery will not fail, like it does now. I want to test this fix. But because this ticket is still open, I can't test your fix, because it's not released yet.

kimadeline commented 2 years ago

As you can see in the initial comment, this is a work in progress, with one task per checkbox. We can let you know once the fix for your issue is released.

To clarify, this is a meta-issue, tracking the refactoring of our testing features. The work done here is already going to be released incrementally.

kr-hansen commented 1 year ago

Is there a way to track which of these issues get addressed in which releases? I'm particularly looking to see which version of vscode-python it was believed that this issue was solved in. I'd like to double check I have the correct version that is believed to have fixed this before opening another issue.

I'm still getting the Unexpected pytest node for pytest discovery with VSCode but it works fine in the terminal. Thought that above issue was closed it May it seems it wasn't solved? Is this likely to be connected to #19809 linked above?

eleanorjboyd commented 1 year ago

Is there a way to track which of these issues get addressed in which releases? I'm particularly looking to see which version of vscode-python it was believed that this issue was solved in. I'd like to double check I have the correct version that is believed to have fixed this before opening another issue.

I'm still getting the Unexpected pytest node for pytest discovery with VSCode but it works fine in the terminal. Thought that above issue was closed it May it seems it wasn't solved? Is this likely to be connected to #19809 linked above?

All our releases should include the issues that are resolved in the given release. You can review the releases here: https://github.com/microsoft/vscode-python/releases and this should have the required information. Additionally this is a central ticket for the rewrite and will be updated when fixes are made.

kr-hansen commented 1 year ago

All our releases should include the issues that are resolved in the given release.

So if there is an issue listed as closed that linked to this central ticket, but it doesn't show up in a release then it is assumed it won't be solved until this rewrite is completed? Just making sure I understand so I don't open a new issue that is the same as ones listed as closed and linked to this ticket but still not fixed.

eleanorjboyd commented 1 year ago

Yes sorry for the confusion- all issues closed against this issue should be solved by the pytest rewrite. If we closed an issue and referenced this one then we are predicting that issue should be resolved by the rewrite. After we finish and ship the rewrite if that is not the case then that would be the time to file additional bugs. I hope this helps!

niander commented 1 year ago

Will this work cover the ability of selecting which folders you want tests to be discovered from? The issue we have right now is that vscode is using pytest to discover all tests including tests that won't work and we are unable to run tests for other projects.

karthiknadig commented 1 year ago

@niander The intent with this re-write is to get out of pytests way when discovering and running tests. After the re-write, you should be able to provide configuration to pytest (using pytest.ini) to restrict where to looks for tests.

niander commented 1 year ago

@karthiknadig thanks for the reply. The issue is that we have several different Python projects in our repository and some of them have tests that won't work unless they are installed in a venv. But, sometimes, we don't want to install them because we are actually working on a different Python project that doesn't depend on them. Since test discover fail for all tests we cant focus on just the tests we are interested in unless we change the vscode settings in the workspace.

eleanorjboyd commented 1 year ago

Could you attach a picture of your file structure with which tests to run and which to not run? Additionally what have you set as your root directory for pytest to find tests? This setting should exist in your settings.json. Thanks!

karrtikr commented 1 year ago

We should make sure we handle https://github.com/microsoft/vscode-python/issues/17810 with this.

eleanorjboyd commented 1 year ago

Hello everyone! We are wrapping up the rewrite of the testing infrastructure and heading into the stage of testing issues and seeing which ones remain. Currently, we are rolling out this new infrastructure in parallel and users can try out the new changes by setting "python.experiments.optInto": ["pythonTestAdapter"] in their user settings.json. Right now you must be on vscode insiders to test but we are in the process of switching all users to the rewrite incrementally so if you do not have insiders, watch our release notes to get updated on when it will begin to hit stable.

I will be triaging our backlog of testing issues and determining which issues still exist. In the meantime, if you have an issue that was waiting on the rewrite to potentially be fixed now is a great time to test it! If you do give it a try, please update the specific issue with the outcome (success or how it is still failing) so I can more quickly determine the ongoing issues following the rewrite. Thanks everyone for your interest and patience as we get this rolled out!

JensMadsen commented 4 weeks ago

"python.experiments.optInto": ["pythonTestAdapter"]

@eleanorjboyd I have enabled the pythonTestAdapter. I still see that the startup time is very long. I use unittest. When I run a single unit test stuff related to tests elsewhere (e.g. other folders) is initialized. Example: I run a straightforward test that tests some simple string gymnastics. Before the test starts I see in the output terminal that e.g. Pytorch is imported which takes a long time.

Please let me know if I can help in any way debugging this :-)

eleanorjboyd commented 3 weeks ago

Hi @JensMadsen! Could you open a new issue and include some of these logs? From there I can help you out a bit more- thanks!

JensMadsen commented 3 weeks ago

Hi @JensMadsen! Could you open a new issue and include some of these logs? From there I can help you out a bit more- thanks!

@eleanorjboyd I will do that but I am unsure what logs you refer to.

eleanorjboyd commented 2 weeks ago

If you go to the command palette and look for the command- python: report issue... that is the best way submit all the logs and info to us as a new issue so I can take a look. Hopefully that would work- thanks!

JensMadsen commented 2 weeks ago

@eleanorjboyd done https://github.com/microsoft/vscode-python/issues/24022