Tests not Reporting Status if One of the Test Folder are Outside of Workspace Folder

AngellusMortis commented 5 years ago

Environment data

VS Code version: 1.35 (Insiders)
Extension version (available under the Extensions sidebar): 2019.4.12954
OS and version: Ubuntu 18.04 LTS (interrupter is running inside of Debian 9 container via Remote Container feature)
Python version (& distribution if applicable, e.g. Anaconda): 3.7
Type of virtual environment used (N/A | venv | virtualenv | conda | ...): Remote Container, system Python
Relevant/affected Python packages and their versions: pytest?

Expected behavior

After running tests via PyTest, the status symbol next to them should change from a ? to either based on test success or failure.

Actual behavior

After running tests via PyTest, the status symbol never stop showing as ?.

Steps to reproduce:

Create Python project with tests
Create a second Python project with tests next to the first one
Open the second Python project as project folder in VS Code
Configure PyTest to run tests for both projects

The use case for this is that we have shared "framework" of code that all of our projects are built that has a set of test that get run to make sure our code does not break the framework. I do not want VS Code running full linting/code checks on the framework because that is not the code I am working on. Just need to make sure tests pass.

Ex. (assuming 1. is at /project1 and 2. is at /project2, and /project2 is the folder you have open in VS Code)

{
    "python.testing.pyTestEnabled": true,
    "python.testing.pyTestArgs": [
       "/project1",
       "/project2",
    ],
}

Also, if project1 is not pip install -e in your environment, you will need to set a $PYTHONPATH environment variable.

Logs

Nothing useful. Just a lot of "Reloading modules... done. Analysis restarted." Python Test Log has nothing but standard PyTest output, no types of errors. Console under Developer tools has nothing useful enough, just "[Extension Host] Python Extension: Cached data exists getEnvironmentVariables, "

DonJayamanne commented 5 years ago

Please could you provide a sample repo so we can replicate this issue.

AngellusMortis commented 5 years ago

I figured out this is two pronged. If you have pytest.ini in a parent directory (outside of workspace) or you have tests that runs outside of workspace, it causes test results to not show up correctly.

I will try to slap together a sample repo this weekend if I can.

AngellusMortis commented 5 years ago

Sorry it took so long for me to make this, but here is a repo you should be able to reproduce the issue with.

https://github.com/AngellusMortis/pytest-vscode

I have a Windows 10 machine, so I generally only use the Remote Development extensions now. I cannot promise it is still reproducible without them, but I imagine it should be.

DonJayamanne commented 5 years ago

Can also replicate with the latest python code used for discovering pytest tests.

karrtikr commented 2 years ago

The use case for this is that we have shared "framework" of code that all of our projects are built that has a set of test that get run to make sure our code does not break the framework. I do not want VS Code running full linting/code checks on the framework because that is not the code I am working on. Just need to make sure tests pass. https://github.com/AngellusMortis/pytest-vscode

In this case your project is actually pytest-vscode which contains the code you want to test, so I think the correct thing should be opening pytest-vscode as a workspace instead. We no longer run any linting checks/code checks unless you open the file so it shouldn't be a problem. Let me know if you disagree and we'll re-evaluate opening the issue.

cc @karthiknadig

microsoft / vscode-python