Type: Bug

Behaviour

Expected vs. Actual

Extension should discover all unit tests but it does not. In terminal it finds a lot more test cases that in the extension.

Steps to reproduce:

I think I found the pattern for this issue. When you have several classes inside a unit test file, the extension only discovers the last class tests.

Diagnostic data

• Python version (& distribution if applicable, e.g. Anaconda): 3.10.12
• Type of virtual environment used (e.g. conda, venv, virtualenv, etc.): Venv
• Value of the python.languageServer setting: Default

Extension version: 2023.22.0 VS Code version: Code 1.85.0 (af28b32d7e553898b2a91af498b1fb666fdebe0c, 2023-12-06T20:48:09.019Z) OS version: Windows_NT x64 10.0.22621 Modes: Remote OS version: Linux x64 5.15.133.1-microsoft-standard-WSL2

As a side note, it also shows two test trees (as shown in the ss) when it should only show the last one. The first has nothing inside and if I run that one it runs indefinitely. It does not change the main behavior but may be related to the issue.

Hello! Could you include a layout of your project and your logs from the python output channel? For your logs, can you first set your log level to trace via theDeveloper: set log level command in the command palette? With these I can better help figure out what might be going wrong. Thanks

In Terminal it shows 2152 tests:

======================== 2152 tests collected in 5.90s =========================

2023-12-14 01:45:16.636 [debug] deferredTill EOT resolved for /home/rodzun/reviNEW/z-dash
2023-12-14 01:45:16.636 [info] Disposing data receiver for /home/rodzun/reviNEW/z-dash and deleting UUID; pytest discovery.

In the extension 1352 tests:

All apps have this folder structure:

Hello again, sorry for the back and forth. I looked through you logs and don't see anything which seems out of place. I explored your original comment where multiple test classes per file caused only the last test class to be found but was able to discover all 4 tests in this example. Does this example work for you? Is there a difference in setup between my example below and your multi-class setup?

# Example of a Python file with multiple classes and their corresponding tests using unittest

# Importing unittest module
import unittest

# First class to be tested
class Calculator:
    def add(self, a, b):
        return a + b

    def subtract(self, a, b):
        return a - b

# Second class to be tested
class StringManipulator:
    def capitalize(self, string):
        return string.capitalize()

    def reverse(self, string):
        return string[::-1]

# Test class for Calculator
class TestCalculator(unittest.TestCase):
    def test_add(self):
        calc = Calculator()
        self.assertEqual(calc.add(2, 3), 5)

    def test_subtract(self):
        calc = Calculator()
        self.assertEqual(calc.subtract(5, 3), 2)

# Test class for StringManipulator
class TestStringManipulator(unittest.TestCase):
    def test_capitalize(self):
        manipulator = StringManipulator()
        self.assertEqual(manipulator.capitalize("hello"), "Hello")

    def test_reverse(self):
        manipulator = StringManipulator()
        self.assertEqual(manipulator.reverse("hello"), "olleh")

No worries, thank you for the tech support. Testing a little bit, I found out that the issue arises when using classes with pytest. It works fine with Unittest, it does not when "converting" to pytest. When doing so, the extension only finds tests from the last class. The bug is consistent with other colleagues as well.

# First class to be tested
class Calculator:
    def add(self, a, b):
        return a + b

    def subtract(self, a, b):
        return a - b

# Second class to be tested
class StringManipulator:
    def capitalize(self, string):
        return string.capitalize()

    def reverse(self, string):
        return string[::-1]

# Test functions for Calculator
class TestCalculator:
    def test_add(self):
        calc = Calculator()
        assert calc.add(2, 3) == 5

    def test_subtract(self):
        calc = Calculator()
        assert calc.subtract(5, 3) == 2

# Test functions for StringManipulator
class TestStringManipulator:
    def test_capitalize(self):
        manipulator = StringManipulator()
        assert manipulator.capitalize("hello") == "Hello"

    def test_reverse():
        manipulator = StringManipulator()
        assert manipulator.reverse("hello") == "olleh"

In this image you can see that only the last two tests are shown:

hm I am still seeing all them still be discovered with the code snippet you have above. Are you settings the same as mine? What type of machine are you on? Also if you set your logs to trace you should see a payload with all the test data, can you send over that / just your whole logs? Thanks

These are my settings:

{
    "python.testing.pytestArgs": [
        "zdash"
    ],
    "python.testing.unittestEnabled": false,
    "python.testing.pytestEnabled": true,
    "python.analysis.logLevel": "Trace"
}

My log is huge so I send you the related parts, let me know if you still need all the logs:

2023-12-15 15:11:37.300 [info] <Package api>
  <Module test_test.py>
    <Class TestCalculator>
        <Function test_add>
        <Function test_subtract>
    <Class TestStringManipulator>
2023-12-15 15:11:37.300 [info] 
        <Function test_capitalize>
        <Function test_reverse>

{"name": "test_test.py", "path": "/home/rodzun/reviNEW/z-dash/zdash/revi/personnel/tests/api/test_test.py", "type_": "file", "id_": "/home/rodzun/reviNEW/z-dash/zdash/revi/personnel/tests/api/test_test.py", "children": [{"name": "test_add", "path": "/home/rodzun/reviNEW/z-dash/zdash/revi/personnel/tests/api/test_test.py", "lineno": "21", "type_": "test", "id_": "/home/rodzun/reviNEW/z-dash/zdash/revi/personnel/tests/api/test_test.py::TestCalculator::test_add", "runID": "/home/rodzun/reviNEW/z-dash/zdash/revi/personnel/tests/api/test_test.py::TestCalculator::test_add"}, {"name": "test_subtract", "path": "/home/rodzun/reviNEW/z-dash/zdash/revi/personnel/tests/api/test_test.py", "lineno": "25", "type_": "test", "id_": "/home/rodzun/reviNEW/z-dash/zdash/revi/personnel/tests/api/test_test.py::TestCalculator::test_subtract", "runID": "/home/rodzun/reviNEW/z-dash/zdash/revi/personnel/tests/api/test_test.py::TestCalculator::test_subtract"}]}, {"name": "test_test.py", "path": "/home/rodzun/reviNEW/z-dash/zdash/revi/personnel/tests/api/test_test.py", "type_": "file", "id_": "/home/rodzun/reviNEW/z-dash/zdash/revi/personnel/tests/api/test_test.py", "children": [{"name": "test_capitalize", "path": "/home/rodzun/reviNEW/z-dash/zdash/revi/personnel/tests/api/test_test.py", "lineno": "31", "type_": "test", "id_": "/home/rodzun/reviNEW/z-dash/zdash/revi/personnel/tests/api/test_test.py::TestStringManipulator::test_capitalize", "runID": "/home/rodzun/reviNEW/z-dash/zdash/revi/personnel/tests/api/test_test.py::TestStringManipulator::test_capitalize"}, {"name": "test_reverse", "path": "/home/rodzun/reviNEW/z-dash/zdash/revi/personnel/tests/api/test_test.py", "lineno": "35", "type_": "test", "id_": "/home/rodzun/reviNEW/z-dash/zdash/revi/personnel/tests/api/test_test.py::TestStringManipulator::test_reverse", "runID": "/home/rodzun/reviNEW/z-dash/zdash/revi/personnel/tests/api/test_test.

All tests seems to be found in the logs, but they are not shown in the left tree panel, and certainly are not counted or tested there.

I'm using Windows 11 with WSL2. VSCode 1.85.0

Could you try my example project and see if it works for you here? Are any of my settings / design wrong here for your setup?

test_classes22643.zip

Tested it and the issue persists. The logs seem to find all four tests but as you can see in the screenshot it only finds two in the panel in the left.

I'm also experiencing this same issue.

@rodzun, I am not familiar with the "zdash" arg, what is that for?

@pnelson-bto Did you try the example project I sent? If so what settings or config are different?

Are you both on Windows to Linux using WSL? This could be the problem that is causing the bug

No, because I only use pytest. I even tried it 2 bare-boned pytest classes, which just asserted that 1 == 1. I am on Ubuntu. This issue seems to be relatively new. I recently performed the latest update on VSCode.

Hi @eleanorjboyd , zdash is just the name of my project and a folder on top of it. I'm using WSL. Updated to last release and still having the issue.

No, because I only use pytest. I even tried it 2 bare-boned pytest classes, which just asserted that 1 == 1. I am on Ubuntu. This issue seems to be relatively new. I recently performed the latest update on VSCode.

The repo should still work as pytest also discovers and executes unittest test cases. You coiuld also send over a minimal repro in your case if that would be easier. Thanks

@rodzun, those logs were helpful, what I am seeing is that pytest sees 4 tests but the payload returned by the subprocess is only 2 (with the class TestCalculator also missing) so somewhere in building the test tree something is going wrong. I see from your logs you have a fair number of plugins, could you try without these and see if it makes a difference? plugins: forked-1.6.0, django-3.10.0, mock-3.11.1, xdist-2.5.0, factoryboy-2.1.0, Faker-19.3.1, requests-mock-1.8.0, datadir-1.3.1, lazy-fixture-0.6.3. If this doesn't work I can recreate your whole setup but the way your error is showing (with half the tests) it doesn't strike me as windows/wsl related. Thanks and let me know

This is your payload returned in a more understandable format

{
    "cwd": "/home/rodzun/test_classes22643",
    "status": "success",
    "tests": {
        "name": "test_classes22643",
        "path": "/home/rodzun/test_classes22643",
        "type_": "folder",
        "children": [
            {
                "name": "test_many_classes.py",
                "path": "/home/rodzun/test_classes22643/test_many_classes.py",
                "type_": "file",
                "id_": "/home/rodzun/test_classes22643/test_many_classes.py",
                "children": [
                    {
                        "name": "test_add",
                        "path": "/home/rodzun/test_classes22643/test_many_classes.py",
                        "lineno": "19",
                        "type_": "test",
                        "id_": "/home/rodzun/test_classes22643/test_many_classes.py::TestCalculator::test_add",
                        "runID": "/home/rodzun/test_classes22643/test_many_classes.py::TestCalculator::test_add"
                    },
                    {
                        "name": "test_subtract",
                        "path": "/home/rodzun/test_classes22643/test_many_classes.py",
                        "lineno": "23",
                        "type_": "test",
                        "id_": "/home/rodzun/test_classes22643/test_many_classes.py::TestCalculator::test_subtract",
                        "runID": "/home/rodzun/test_classes22643/test_many_classes.py::TestCalculator::test_subtract"
                    }
                ]
            }
        ],
        "id_": "/home/rodzun/test_classes22643"
    },
    "error": []
}

I'm facing the same issue. I'm using pytest and tests from only the last class are shown in sidebar. However, in the Output tab all of them are listed correctly.

This problem has been plaguing me for months. My temporary solution has been to edit the VSCode state file to disable the "pythonTestAdapter" experimental feature.

Now that I've looked through this page and the one for issue #22457, it seems the easiest solution is to just update my pytest plugin from 6.2.x to the latest 8.x.x version - it immediately fixed the issue.

My company updated pytest to 7.4.4 version and it's working correctly now

Glad that worked! Our goal is to support all versions of pytest which that teams supports. Could you provide your specific pytest version which was causing the failure? We have added an old 6.2.x version to our CI to run and are not seeing this error as of now.

Thanks for all the investigation, everyone. Upgrading to pytest@7.4.4 worked for me as well.

My issue might be related. I have a project that uses pytest-lazy-fixture, and the test discovery broke recently after working fine for months. I am using pytest 7.4.4. and the v2024.5.11141012 (pre-release) Python extension. I have this minimum reproducible example:

import pytest
from pytest_lazyfixture import lazy_fixture

@pytest.fixture
def fixture_a():
    return "a"

@pytest.fixture
def fixture_b():
    return "b"

@pytest.fixture
def fixture_c():
    return "c"

class TestA:
    @pytest.fixture(
        scope="class",
        params=[lazy_fixture("fixture_a"), lazy_fixture("fixture_b")],
    )
    def fixt(self, request):
        return request.param

    def test_a(self, fixt):
        assert fixt in ["a", "b"]

class TestB:
    @pytest.fixture(
        scope="class",
        params=[lazy_fixture("fixture_a"), lazy_fixture("fixture_c")],
    )
    def fixt(self, request):
        return request.param

    def test_a(self, fixt):
        assert fixt in ["a", "c"]

The tests tree is incorrect:

If I try to go to run/debug/go to test to the ones that are found twice, they all fall under the test that's declared first.

If I switch to the stable Python extension, the tests are discovered and run correctly:

Hi @Tinuxx, I am able to repro this issue. I will work on putting in a fix. Thanks!

Because we have not heard back with the information we requested, we are closing this issue for now. If you are able to provide the info later on, then we will be happy to re-open this issue to pick up where we left off.

Happy Coding!

I'm also seeing this problem over here. Downgrading to v2024.4.1 of the Python extension is my current workaround. When looking through the debug logs, the newer broken extensions are not getting the ResultResolver EOT log messages.

Working on v2024.4.1:

======================== 8431 tests collected in 24.23s ========================

2024-06-18 23:41:26.450 [info] ResultResolver EOT received for discovery.
2024-06-18 23:41:26.450 [info] ResultResolver EOT received for discovery.
...
======================== 8431 tests collected in 24.44s ========================

2024-06-18 23:41:27.776 [info] ResultResolver EOT received for discovery.
2024-06-18 23:41:27.776 [info] ResultResolver EOT received for discovery.
2024-06-18 23:41:28.114 [info] Disposing data receiver for /home/vscode/workspace and deleting UUID; pytest discovery.
2024-06-18 23:41:28.165 [info] Disposing data receiver for /home/vscode/workspace and deleting UUID; pytest discovery.

Broken on v2024.4.1:

2024-06-18 23:30:37.108 [info] ======================== 8431 tests collected in 10.11s ========================

2024-06-18 23:30:37.109 [debug] client emitted close event, connectionCount:  0
2024-06-18 23:30:37.109 [debug] connection count is <= 0, closing the server:  /tmp/python-test-discovery-36c77d1b7d832c04556f.sock
2024-06-18 23:30:37.109 [debug] Test Discovery named pipe /tmp/python-test-discovery-36c77d1b7d832c04556f.sock closed
2024-06-18 23:30:37.109 [debug] Test Discovery named pipe /tmp/python-test-discovery-36c77d1b7d832c04556f.sock disposed
2024-06-18 23:30:38.912 [debug] deferredTill EOT resolved
2024-06-18 23:30:39.156 [debug] deferredTill EOT resolved

Hi @mmaday, could you open a new issue with full repro steps? I am not sure if the problem you are facing is related

@Tinuxx, I am seeing the correct test explorer view now when I try to re-run the sample in the insiders extension- is it working for you?

@eleanorjboyd I can confirm that the test discovery works and is displayed correctly after switching to pre-release:

Thanks! 😄

Great! Thanks for verifying!

Because we have not heard back with the information we requested, we are closing this issue for now. If you are able to provide the info later on, then we will be happy to re-open this issue to pick up where we left off.

Happy Coding!

closing as complete- please comment if that is not the case