Unittest discovery silently fails when Pytest is not installed

[joyceerhl]

Testing https://github.com/microsoft/vscode-python/issues/13301

1. Run Python: Configure Tests
2. Choose unittest
3. :bug: nothing happens. When I open the test view I see that there was a discovery error and that I should see Output > Python. I would expect an error message with a link or button to open the output channel
4. :bug: the output channel says test discovery failed because pytest wasn't installed. Should we help the user install pytest in this flow? Also, is it expected that pytest is required even if I'm trying to use unittest?

[eleanorjboyd]

realized this is likely the fix: https://github.com/microsoft/vscode-python/pull/21726

[roblourens]

I ran into this as well, so if you were expecting it to be fixed, I don't think it is.

I was also curious why pytest is used to do discovery for unittest? It just knows about everything?

[eleanorjboyd]

pytest should not be needed if you are just using unittest, will investigate and put in a fix!

[eleanorjboyd]

@roblourens, testing this and unable to reproduce the behavior. Could you send over the test file you had loaded and the settings you picked (when configuring testing so it might look something like below)

{
  "python.testing.unittestArgs": ["-v", "-s", ".", "-p", "*test*.py"],
  "python.testing.pytestEnabled": false,
  "python.testing.unittestEnabled": true
}

[github-actions[bot]]

Because we have not heard back with the information we requested, we are closing this issue for now. If you are able to provide the info later on, then we will be happy to re-open this issue to pick up where we left off.

Happy Coding!

[rzhao271]

Repro steps

1. My potentially irrelevant user settings:

   "python.experiments.optInto": [
   "pythonTestAdapter"
   ],
   "python.terminal.executeInFileDir": true,
   "python.createEnvironment.trigger": "prompt",

2. Navigate to a directory on Windows Terminal (pwsh profile)
3. Run the following lines

mkdir example-python-tests
cd example-python-tests
ni sample.py
ni test_sample.py
mkdir .vscode
ni .vscode/settings.json
code-insiders .

3. VS Code Insiders should be open now. Create a new venv Python environment and confirm that pytest isn't installed in the venv

4. Copy the following to sample.py

   def xyz():
   return 0

5. Copy the following to test_sample.py

   
   import unittest

class TestScratch(unittest.TestCase): def test_add(self): pass

6. Copy the following to .vscode/settings.json
```json
{
    "python.testing.unittestArgs": [
    "-v",
    "-s",
        ".",
        "-p",
        "test_*.py"
    ],
    "python.testing.pytestEnabled": false,
    "python.testing.unittestEnabled": true
}

7. Click on the beaker icon in the activity bar to go to the tests view
8. :bug: pytest Discovery error is shown

[eleanorjboyd]

hm tried these steps and still not seeing it, will keep my eye out as I have seen this before but also couldn't repro it a second time.

[eleanorjboyd]

From: https://github.com/microsoft/vscode-python/issues/22833 filed by @luabud with repo steps

Type: Bug

Behaviour

When trying to set up test discovery with unittest, I can only get pytest to trigger discovery to trigger. Even though I have the following settings in my workspace:

    "python.testing.unittestArgs": [
        "-v",
        "-s",
        "./tests",
        "-p",
        "test_*.py"
    ],
    "python.testing.pytestEnabled": false,
    "python.testing.unittestEnabled": true

Steps to reproduce:

1. Create a new empty folder
2. Add a file called e.g. test_1.py with the following content:

   
   import unittest

class TestStringMethods(unittest.TestCase): def test_upper(self): self.assertEqual("hello".upper(), "HELLO")

def test_isupper(self):
    self.assertTrue("HELLO".isupper())
    self.assertFalse("Hello".isupper())

def test_split(self):
    s = "hello world"
    self.assertEqual(s.split(), ["hello", "world"])
    # check that s.split fails when the separator is not a string
    with self.assertRaises(TypeError):
        s.split(2)

3. Click on the test explorer, select configure tests, select "unittest" and then "." and then "test_*.py"
4. See that it will fail pytest discovery
![image](https://github.com/microsoft/vscode-python/assets/45497113/9b64a2ef-c7dc-48ff-8028-1f8da9ff2d56)

<!--
**After** creating the issue on GitHub, you can add screenshots and GIFs of what is happening. Consider tools like https://www.cockos.com/licecap/, https://github.com/phw/peek or https://www.screentogif.com/ for GIF creation.
-->

<!-- **NOTE**: Everything below except Python output panel is auto-generated; no editing required. Please do provide Python output panel. -->
# Diagnostic data

-   Python version (& distribution if applicable, e.g. Anaconda): 3.12.0
-   Type of virtual environment used (e.g. conda, venv, virtualenv, etc.): Global
-   Value of the `python.languageServer` setting: Pylance

<details>

<summary>Output for <code>Python</code> in the <code>Output</code> panel (<code>View</code>→<code>Output</code>, change the drop-down the upper-right of the <code>Output</code> panel to <code>Python</code>)
</summary>

<p>

024-02-01 18:09:54.242 [debug] Testing: Manually triggered test refresh 2024-02-01 18:09:54.242 [debug] Testing: Clearing all discovered tests 2024-02-01 18:09:54.242 [debug] Testing: Forcing test data refresh 2024-02-01 18:09:54.242 [debug] Testing: Refreshing all test data 2024-02-01 18:09:54.242 [debug] Found cached env for REDACTED\AppData\Local\Programs\Python\Python312\python.exe 2024-02-01 18:09:54.244 [info] Discover tests for workspace name: sample - uri: REDACTED\Apps\sample 2024-02-01 18:09:54.244 [info] Running discovery for unittest using the new test adapter. 2024-02-01 18:09:54.245 [info] All environment variables set for pytest discovery for workspace REDACTED\Apps\sample: REDACTED 2024-02-01 18:09:54.245 [debug] Found cached env for REDACTED\AppData\Local\Programs\Python\Python312\python.exe 2024-02-01 18:09:54.263 [debug] Activation Commands received undefined for shell cmd, resource REDACTED\Apps\sample and interpreter REDACTED\AppData\Local\Programs\Python\Python312\python.exe 2024-02-01 18:09:54.279 [debug] Running pytest discovery with command: -m pytest -p vscode_pytest --collect-only for workspace REDACTED\Apps\sample. 2024-02-01 18:09:54.280 [info] > ~\AppData\Local\Programs\Python\Python312\python.exe -m pytest -p vscode_pytest --collect-only 2024-02-01 18:09:54.280 [info] cwd: . 2024-02-01 18:09:54.353 [error] REDACTED\AppData\Local\Programs\Python\Python312\python.exe: No module named pytest

2024-02-01 18:09:54.363 [error] Subprocess exited unsuccessfully with exit code 1 and signal null on workspace REDACTED\Apps\sample. 2024-02-01 18:09:54.363 [error] Subprocess exited unsuccessfully with exit code 1 and signal null on workspace REDACTED\Apps\sample. Creating and sending error discovery payload 2024-02-01 18:09:54.363 [error] pytest test discovery error for workspace: REDACTED\Apps\sample

The python test process was terminated before it could exit on its own, the process errored with: Code: 1, Signal: null for workspace REDACTED\Apps\sample 2024-02-01 18:09:54.363 [info] ResultResolver EOT received for discovery. 2024-02-01 18:09:54.363 [debug] deferredTill EOT resolved for REDACTED\Apps\sample 2024-02-01 18:09:54.363 [info] Disposing data receiver for REDACTED\Apps\sample and deleting UUID; pytest discovery.

</p>
</details>

<details>

<summary>User Settings</summary>

<p>

venvFolders: ""

languageServer: "Pylance"

testing • unittestArgs: "" • unittestEnabled: true

experiments • optInto: ["pythonTestAdapter","regionCommentDiagnostics"]

</p>
</details>

Extension version: 2024.0.0
VS Code version: Code - Insiders 1.87.0-insider (442c133fe92b5e2606c8242caae4e68938faccc7, 2024-02-01T09:49:33.086Z)
OS version: Windows_NT x64 10.0.22621
Modes:

<details>
<summary>System Info</summary>

|Item|Value|
|---|---|
|CPUs|11th Gen Intel(R) Core(TM) i7-1185G7 @ 3.00GHz (8 x 2995)|
|GPU Status|2d_canvas: enabled<br>canvas_oop_rasterization: enabled_on<br>direct_rendering_display_compositor: disabled_off_ok<br>gpu_compositing: enabled<br>multiple_raster_threads: enabled_on<br>opengl: enabled_on<br>rasterization: enabled<br>raw_draw: disabled_off_ok<br>skia_graphite: disabled_off<br>video_decode: enabled<br>video_encode: enabled<br>vulkan: disabled_off<br>webgl: enabled<br>webgl2: enabled<br>webgpu: enabled|
|Load (avg)|undefined|
|Memory (System)|31.71GB (12.58GB free)|
|Process Argv|--log trace --log ms-python.autopep8=debug --crash-reporter-id 11494669-52ca-4f2c-aa0e-29172189cc8e|
|Screen Reader|no|
|VM|0%|
</details><details>
<summary>A/B Experiments</summary>

vsliv368cf:30146710 vspor879:30202332 vspor708:30202333 vspor363:30204092 vsc_aa:30263845 vscod805:30301674 vsaa593:30376534 py29gd2263:30784851 c4g48928:30535728 2i9eh265:30646982 962ge761:30841072 pythongtdpath:30726887 welcomedialog:30812478 pythonidxpt:30768918 pythonnoceb:30776497 asynctok:30898717 dsvsc013:30777762 dsvsc014:30777825 dsvsc015:30821418 pythontestfixt:30866404 pythonregdiag2:30926734 pyreplss1:30879911 pythonmypyd1:30859725 pythoncet0:30859736 pythontbext0:30879054 accentitlementst:30870582 dsvsc016:30879898 dsvsc017:30880771 dsvsc018:30880772 8082a590:30953407 edj9j583:30943796 fegfb526:30952798 7j2b6412:30951517 bg6jg535:30946824

</details>

<!-- generated by issue reporter -->

[eleanorjboyd]

@luabud tried the steps and got success with no reference to the pytest error. Are you able to repro this if you try multiple times or is it flaky for you too?

[luabud]

seems to be flaky too, I am unable to repro this today

[luabud]

@eleanorjboyd ah just reproduced it again!

funny enough by downgrading to the previous release version I can't repro (could be a coincidence I guess!)

same settings:

{
    "python.testing.unittestArgs": [
        "-v",
        "-s",
        "./tests",
        "-p",
        "*test*.py"
    ],
    "python.testing.pytestEnabled": false,
    "python.testing.unittestEnabled": true
}

one thing I noticed though is that whenever I can't reproduce it, it's because the tests had been discovered before successfully. So for example, I clone a repo for the first time, I configure the test features to use unittest all using v2024.0.0, I can repro the issue. But then when I downgrade to v2023.22,1, tests are discovered. Then I upgrade to 2024.0.0, retrigger test discovery, it continues to work. Could be a big coincidence, but this it was consistently the case for me. When I open repos that had been configured to use unittest before, it simply works. It can really just reproduce it when configuring repos that I had never configured before.

[luabud]

never mind, must be a coincidence cause I just repro'ed it using the old version (which makes sense, since all the reports here use older versions):

[eleanorjboyd]

found the issue and am working on a solution: issue: when you activate the python extension, it checks if unittest enabled, if false then just sets the adapter to pytest. This means then if someone sets their testing type to unittest after activation the adapter is still the pytest adapter and so pytest is run. solution: instead of creating adapters on activation, they should instead be created dynamically when the call for either run or discovery comes in. Then there is no chance of the wrong one being pre-selected.