Nested lazy fixtures not respecting scope

[mtraynham]

Hi, I have a fairly complex set of fixtures to help with parameterization of polymorphic requests. Unfortunately it seems that some of the fixtures stop respecting their intended scope and are called multiple times.

Theoretically, let's say we are building a graph which has vertices & edges. Well, the vertices themselves can be polymorphic fixtures, some of their properties can be mutated, and the test cases are building edges between them.

In the following example, I have a series of network primitives, Ports & Cloud Ports, which may have multiple locations associated with them. At the end, I take two vertex fixtures and test them.

The test list looks fine, but the count of how many times a session scoped fixture is invoked seems incorrect. In all cases, I would have expected the counts to match how many dependent parameters there are, but in some cases I see 8, others I see 16. I use a wrapper here to collect invocation counts and dump them at the end of the tests.

It's as if the cache key is invalidated or doesn't match because of where the fixture is referenced in the hierarchy of fixtures.

test_lazy_fixture[port-cloud_attachment-cloud_port_c-location_a]
test_lazy_fixture[port-cloud_attachment-cloud_port_c-location_b]
test_lazy_fixture[port-cloud_attachment-cloud_port_d-location_a]
test_lazy_fixture[port-cloud_attachment-cloud_port_d-location_b]
test_lazy_fixture[port-port_attachment-location_c-location_a]
test_lazy_fixture[port-port_attachment-location_c-location_b]
test_lazy_fixture[port-port_attachment-location_d-location_a]
test_lazy_fixture[port-port_attachment-location_d-location_b]
test_lazy_fixture[port_group-cloud_attachment-cloud_port_c-location_a]
test_lazy_fixture[port_group-cloud_attachment-cloud_port_c-location_b]
test_lazy_fixture[port_group-cloud_attachment-cloud_port_d-location_a]
test_lazy_fixture[port_group-cloud_attachment-cloud_port_d-location_b]
test_lazy_fixture[port_group-port_attachment-location_c-location_a]
test_lazy_fixture[port_group-port_attachment-location_c-location_b]
test_lazy_fixture[port_group-port_attachment-location_d-location_a]
test_lazy_fixture[port_group-port_attachment-location_d-location_b]

{
  "attachment": 1,
  "cloud_attachment": 8,
  "cloud_port": 8,
  "cloud_port_c": 1,
  "cloud_port_d": 1,
  "link_port": 1,
  "location_a": 1,
  "location_b": 1,
  "location_c": 1,
  "location_d": 1,
  "port": 8,
  "port_attachment": 8,
  "port_group": 8,
  "primary_location": 16,
  "secondary_location": 8,
  "secondary_port": 8
}

import collections
import functools
import json

import pytest
from pytest_lazyfixture import lazy_fixture

_COUNTERS = collections.defaultdict(int)

def teardown_module(module):
    print(json.dumps(_COUNTERS, sort_keys=True, indent=2))

def counter(fn):
    @functools.wraps(fn)
    def wrapper(*args, **kwargs):
        fn(*args, **kwargs)
        _COUNTERS[fn.__name__] += 1

    return wrapper

@pytest.fixture(scope='session')
@counter
def location_a() -> str:
    return 'location_a'

@pytest.fixture(scope='session')
@counter
def location_b() -> str:
    return 'location_b'

@pytest.fixture(params=lazy_fixture([
    'location_a',
    'location_b'
]), scope='session')
@counter
def primary_location(request: pytest.FixtureRequest) -> str:
    return request.param  # type: ignore

@pytest.fixture(scope='session')
@counter
def location_c() -> str:
    return 'location_c'

@pytest.fixture(scope='session')
@counter
def location_d() -> str:
    return 'location_d'

@pytest.fixture(params=lazy_fixture([
    'location_c',
    'location_d'
]), scope='session')
@counter
def secondary_location(request: pytest.FixtureRequest) -> str:
    return request.param  # type: ignore

@pytest.fixture(scope='session')
@counter
def cloud_port_c() -> str:
    return 'cloud_port_c'

@pytest.fixture(scope='session')
@counter
def cloud_port_d() -> str:
    return 'cloud_port_d'

@pytest.fixture(params=lazy_fixture([
    'cloud_port_c',
    'cloud_port_d'
]), scope='session')
@counter
def cloud_port(request: pytest.FixtureRequest) -> str:
    return request.param  # type: ignore

@pytest.fixture(scope='session')
@counter
def port(primary_location: str) -> str:
    return f'port-{primary_location}'

@pytest.fixture(scope='session')
@counter
def port_group(primary_location: str) -> str:
    return f'port_group-{primary_location}'

@pytest.fixture(params=lazy_fixture([
    'port',
    'port_group'
]), scope='session')
@counter
def link_port(request: pytest.FixtureRequest) -> str:
    return request.param  # type: ignore

@pytest.fixture(scope='session')
@counter
def secondary_port(secondary_location: str) -> str:
    return f'secondary_port-{secondary_location}'

@pytest.fixture(scope='session')
@counter
def port_attachment(secondary_port: str) -> str:
    return f'port_attachment-{secondary_port}'

@pytest.fixture(scope='session')
@counter
def cloud_attachment(cloud_port: str) -> str:
    return f'cloud_attachment-{cloud_port}'

@pytest.fixture(params=lazy_fixture([
    'port_attachment',
    'cloud_attachment'
]), scope='session')
@counter
def attachment(request: pytest.FixtureRequest) -> str:
    return request.param  # type: ignore

def test_lazy_fixture(link_port: str, attachment: str) -> None:
    print(f'{link_port}-{attachment}')

[mtraynham]

To give an idea of how this was before adopting lazy_fixture, we only had one level of parameterization, but this required multiple tests.

import pytest

@pytest.fixture(scope='session')
def location_a() -> str:
    return 'location_a'

@pytest.fixture(scope='session')
def location_b() -> str:
    return 'location_b'

@pytest.fixture(params=[
    'location_a',
    'location_b'
], scope='session')
def primary_location(request: pytest.FixtureRequest) -> str:
    return request.getfixturevalue(request.param)

@pytest.fixture(scope='session')
def location_c() -> str:
    return 'location_c'

@pytest.fixture(scope='session')
def location_d() -> str:
    return 'location_d'

@pytest.fixture(params=[
    'location_c',
    'location_d'
], scope='session')
def secondary_location(request: pytest.FixtureRequest) -> str:
    return request.getfixturevalue(request.param)

@pytest.fixture(scope='session')
def cloud_port_c() -> str:
    return 'cloud_port_c'

@pytest.fixture(scope='session')
def cloud_port_d() -> str:
    return 'cloud_port_d'

@pytest.fixture(params=[
    'cloud_port_c',
    'cloud_port_d'
], scope='session')
def cloud_port(request: pytest.FixtureRequest) -> str:
    return request.getfixturevalue(request.param)

@pytest.fixture(scope='session')
def port(primary_location: str) -> str:
    return f'port-{primary_location}'

@pytest.fixture(scope='session')
def port_group(primary_location: str) -> str:
    return f'port_group-{primary_location}'

@pytest.fixture(scope='session')
def secondary_port(secondary_location: str) -> str:
    return f'secondary_port-{secondary_location}'

@pytest.fixture(scope='session')
def port_attachment(secondary_port: str) -> str:
    return f'port_attachment-{secondary_port}'

@pytest.fixture(scope='session')
def cloud_attachment(cloud_port: str) -> str:
    return f'cloud_attachment-{cloud_port}'

def test_port_to_port_attachment(port: str, port_attachment: str) -> None:
    print(f'{port}-{port_attachment}')

def test_port_to_cloud_attachment(port: str, cloud_attachment: str) -> None:
    print(f'{port}-{cloud_attachment}')

def test_port_group_to_port_attachment(port_group: str, port_attachment: str) -> None:
    print(f'{port_group}-{port_attachment}')

def test_port_group_to_cloud_attachment(port_group: str, cloud_attachment: str) -> None:
    print(f'{port_group}-{cloud_attachment}')

[mtraynham]

After looking at the pytest documentation, maybe this is expected. Kind of unfortunate, some of my fixtures are a bit expensive to create and teardown.

Pytest only caches one instance of a fixture at a time, which means that when using a parametrized fixture, pytest may invoke a fixture more than once in the given scope.

https://docs.pytest.org/en/6.2.x/fixture.html#fixture-scopes