--benchmark-only but all fixtures are run

[y-ishida]

I ran pytest command with --benchmark-only option but all fixtures of skipped tests are run. I think that skipped tests' fixtures shouldn't be run as same as marking of 'skip'. Fixtures of a test marked @pytest.mark.skip are not be run. Or are there any workaround?

The short code is as follows.

import pytest
import time

@pytest.fixture()
def prep():
    print('run prep')

@pytest.mark.skip
def test_a(prep):
    pass

def test_b(prep):
    pass

def test_c(benchmark):
    benchmark(time.sleep, 0.1)

And then run pytest without --benchmark-only.

$ pytest -vs
...
test_foo.py::test_a SKIPPED
test_foo.py::test_b run prep
PASSED
test_foo.py::test_c PASSED
...

The fixture 'prep' isn't run at the test_a that marked 'skip'. Next, run pytest with --benchmark-only.

$ pytest -vs --benchmark-only
...
test_foo.py::test_a SKIPPED
test_foo.py::test_b run prep
SKIPPED
test_foo.py::test_c PASSED
...

The fixture 'prep' is run at skipped test_b.

Thanks!

[ionelmc]

Ooof.... well now I've looked at the code and this is sort of a chicken and egg situation: I could technically use pytest_collection_modifyitems instead of pytest_runtest_call (and then be able to avoid the situation you're describing) but that also means I cannot know if tests that require a benchmark "fixture" would actually get the fixture from my plugin. Thus I'd be applying benchmark skipping behavior on tests that have a benchmark fixture that isn't actually a pytest_benchmark.fixture.BenchmarkFixture.

I guess that would be some acceptable tradeoff?

[ionelmc]

Fixed in master.