False positive tests when bulk run

[dpopchev]

Neovim-0.8 Python-3.8.10

PS: Note running individually has expected behavior -- inner test break outer ones PSS: Running outer creates false positive.

[rcarriga]

Can you provide some way to reproduce?

[dpopchev]

Sorry, cannot reproduce.

It was rather late when I observed the behavior; thus the issue may have been into the keyboard operator.

[dpopchev]

Just re-run some test, it is a strange mixture using fixtures. Yes, intentionally leaving out needing imports.

import pytest

@pytest.fixture
def nrpydir(tmp_path):
    return 10

# step 1, cursor here, require("neotest").run.run(vim.fn.expand("%")) produces Passes on all
def test_initializer_creates_nothing(nrpydir, tmp_path):
    assert True

# step 2, cursor here, require("neotest").run.run(vim.fn.expand("%")) produces Passes on all
# step 3, cursor here, require("neotest").run.run(), produces Passes on all Test group
# step 5, cursor here, require("neotest").run.run(), overwrites Fails with Passes
class TestCreateMthd:
    @pytest.fixture()
    def call_create(self, nrpydir):
        nrpydir.create()

    def test_a(self):
        assert True
    # step 4, cursor here, require("neotest").run.run(), produces Fail on TestRootAttr group
    class TestRootAttr:

        def test_root_dir_exists(self, nrpydir, call_create):
            assert False

        def test_root_attr(self, nrpydir, call_create, tmp_path):
            assert False

        class TestSecondCallEffect:

            class TestDefaultBehavior:

                def test_no_change(self, call_create_second_time, nrpydir, tmp_path):
                    assert False

[dpopchev]

PS: Ignore closign

[dpopchev]

Son got his hands on the PC, sorry.

[rcarriga]

No worries, thank you for the reproduction. I also see the same so I'll be able to take a look at some stage and figure this out :smile:

[christoph-blessing]

I think I have a simpler reproduction for the same issue:

def test_foo():
    assert True

def test_bar(fixture):  # Fixture does not exist
    assert False

Running all test in the file incorrectly reports that they all passed. Running them individually produces the expeced result. This seems to happen when the first test passes, a subsequent test encounters an error during setup and multiple tests are ran. Hope it helps.

[noamsto]

Any updates on this? I also see this issue when pytest-django setup fixtures throws IntegerityError