search

pytest-dev / pytest

The pytest framework makes it easy to write small tests, yet scales to support complex functional testing
https://pytest.org
MIT License
11.8k stars 2.62k forks source link

run class scoped parametrized fixture's dependency fixture's teardown after each test class iteration #9427

Open marshall7m opened 2 years ago

marshall7m commented 2 years ago

Problem:

The conftest.py fixture's setup/teardown is not executed after each test class iteration.

When running pytest test_foo.py --setup-plan, the SETUP C conf_fixt is only executed before the first parametrized class iteration, and TEARDOWN C conf_fixt is only executed after the last parameterized class iteration is complete.

Related Files:

conftest.py

import pytest
from random import randint

@pytest.fixture(scope='class')
def conf_fixt():
    y = randint(1, 100000)
    yield y
    print(f'teardown: {y}')

test_foo.py

from buildspecs.foo.conftest import conf_fixt
import pytest

@pytest.mark.parametrize("scenario", [
    ("scenario_1"),
    ("scenario_2")
], scope='class')

class TestZoo:
    @pytest.fixture(scope="class")
    def scenario_1(self):
        yield 'scenario_1'

    @pytest.fixture(scope="class")
    def scenario_2(self):
        yield 'scenario_2'

    def test_scenario(self, scenario, request, conf_fixt):
        print(f'scenario value: {request.getfixturevalue(scenario)}')
        print(f'conf_fixt value: {conf_fixt}')

    def test_other(self, scenario, request, conf_fixt):
        print(f'scenario value: {request.getfixturevalue(scenario)}')
        print(f'conf_fixt value: {conf_fixt}')

Actual:

pytest test_foo.py

platform linux -- Python 3.9.8, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /src
plugins: mock-3.6.1
collected 4 items                                                                                   

test_foo.py scenario value: scenario_1
conf_fixt value: 111111
.scenario value: scenario_1
conf_fixt value: 111111
.scenario value: scenario_2
conf_fixt value: 111111
.scenario value: scenario_2
conf_fixt value: 111111
.teardown: 111111

pytest test_foo.py --setup-plan

platform linux -- Python 3.9.8, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /src
plugins: mock-3.6.1
collected 2 items                                                                                   

test_foo.py 
      SETUP    C conf_fixt
      SETUP    C scenario['scenario_1']
        buildspecs/foo/test_foo.py::TestZoo::test_scenario[scenario_1] (fixtures used: conf_fixt, request, scenario)
      TEARDOWN C scenario['scenario_1']
      SETUP    C scenario['scenario_2']
        buildspecs/foo/test_foo.py::TestZoo::test_scenario[scenario_2] (fixtures used: conf_fixt, request, scenario)
      TEARDOWN C scenario['scenario_2']
      TEARDOWN C conf_fixt

Expected:

pytest test_foo.py

platform linux -- Python 3.9.8, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /src
plugins: mock-3.6.1
collected 4 items                                                                                   

test_foo.py scenario value: scenario_1
conf_fixt value: 111111
.scenario value: scenario_1
conf_fixt value: 111111
.teardown: 111111
.scenario value: scenario_2
conf_fixt value: 222222
.scenario value: scenario_2
conf_fixt value: 222222
.teardown: 222222

pytest test_foo.py --setup-plan

platform linux -- Python 3.9.8, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /src
plugins: mock-3.6.1
collected 2 items                                                                                   

test_foo.py 
      SETUP    C conf_fixt
      SETUP    C scenario['scenario_1']
        buildspecs/foo/test_foo.py::TestZoo::test_scenario[scenario_1] (fixtures used: conf_fixt, request, scenario)
      TEARDOWN C scenario['scenario_1']
      ###Expected teardown
      TEARDOWN C conf_fixt 
      ###
      SETUP    C scenario['scenario_2']
        buildspecs/foo/test_foo.py::TestZoo::test_scenario[scenario_2] (fixtures used: conf_fixt, request, scenario)
      TEARDOWN C scenario['scenario_2']
      TEARDOWN C conf_fixt

Attempts:

Add the conf_fixt fixture to the pytest.marks.parametrize() decorator like so:

import pytest

@pytest.mark.parametrize("scenario,conf_fixt_param", [
    ("scenario_1", "conf_fixt"),
    ("scenario_2", "conf_fixt")
], scope='class')

class TestZoo:
    @pytest.fixture(scope="class")
    def scenario_1(self):
        yield 'scenario_1'

    @pytest.fixture(scope="class")
    def scenario_2(self):
        yield 'scenario_2'

    def test_scenario(self, scenario, request, conf_fixt_param):
        print(f'scenario value: {request.getfixturevalue(scenario)}')
        print(f'conf_fixt value: {request.getfixturevalue(conf_fixt_param)}')

    def test_other(self, scenario, request, conf_fixt_param):
        print(f'scenario value: {request.getfixturevalue(scenario)}')
        print(f'conf_fixt value: {request.getfixturevalue(conf_fixt_param)}')

Same Actual and Expected output as the original problem.

joniemi commented 2 years ago

I'm running into the same issue. Pytest does not teardown parent fixtures to a parametrized fixture in class scope until all test class instances have run.

You can work around the bug by making the dependency fixture depend on the parametrized fixture. In the example by @marshall7m , you would add the parameter scenario to conf_fixt:

conftest.py

import pytest
from random import randint

@pytest.fixture(scope='class')
def conf_fixt(scenario):
    y = randint(1, 100000)
    yield y
    print(f'teardown: {y}')

Now when you run pytest --setup-plan it results in

issue9427/test_foo.py
      SETUP    C scenario['scenario_1']
      SETUP    C conf_fixt (fixtures used: scenario)
        issue9427/test_foo.py::TestZoo::test_scenario[scenario_1] (fixtures used: conf_fixt, request, scenario)
        issue9427/test_foo.py::TestZoo::test_other[scenario_1] (fixtures used: conf_fixt, request, scenario)
      TEARDOWN C conf_fixt
      TEARDOWN C scenario['scenario_1']
      SETUP    C scenario['scenario_2']
      SETUP    C conf_fixt (fixtures used: scenario)
        issue9427/test_foo.py::TestZoo::test_scenario[scenario_2] (fixtures used: conf_fixt, request, scenario)
        issue9427/test_foo.py::TestZoo::test_other[scenario_2] (fixtures used: conf_fixt, request, scenario)
      TEARDOWN C conf_fixt
      TEARDOWN C scenario['scenario_2']

It's awkward though and not possible if your parametrized fixture has a direct dependency.

I'm running pytest 6.2.5 & python 3.8.10