Allow to dynamically create fixture in plugin

[pbrezina]

What's the problem this feature will solve?

I am the author of next-actions/pytest-mh which is a multihost testing plugin for pytest. We are doing some black magic to dynamically create test fixtures and map them to some custom objects. Some user-focused details are described here but TLDR we are doing this:

@pytest.fixture
def pytest_fixture():
    pass

def test_example(client, ldap, pytest_fixture):
    pass

In this scenario, client and ldap are passed to the test_example by setting it in item.funcargs in runtest hook to internal objects created per-tests. pytest_fixture is standard pytest fixture. This works nicely and it is using only public fields (I'm not sure if funcargs is documented, but at the very least it is not prefix with _).

However, accessing these data from pytest fixture fails, obviously:

@pytest.fixture
def pytest_fixture(client):
    pass

def test_example(client, ldap, pytest_fixture):
    pass

... because client is not a registered fixture. I probably could manage that by accessing pytest private stuff, but I'd like to avoid that.

I would like to be able to create and delete fixture (at least function-scoped) inside hooks.

Describe the solution you'd like

Provide API to dynamically register and unregister fixtures from pytest hooks.

Alternative Solutions

Currently, deep and private pytest objects and attributes are required.

Additional context

[RonnyPfannschmidt]

The ask here is very unclear

Unfortunately it's pretty much a open research question

[pbrezina]

What part should I clarify?

[RonnyPfannschmidt]

Both the ask here and the documentation of the plugin are entirely unclear

I'm not going to guess and I'm not going to try and figure it from the source code

[pbrezina]

At this moment, you can only create fixture using @pytest.fixture, decorated functions are collected and registered as fixtures at some point of pytest setup, right?

In pytest-mh, we are passing custom values to each test. The values are objects that are unique for each tests and they are dynamically created as needed by the test. For example:

def test_example(client: ClientRole, ldap: LDAPRole):
    pass

This will pass to the test client and ldap objects that provide high-level API to manage relevant services. We can't convert these parameters into fixtures using @pytest.fixture decorator and static definition since they really need to be created dynamically for each test for various reasons.

One of the reason is a thing we call "topology parametrization", which means that single test can be run against different hosts (backends) and then parameter called provider can take different values based on the topology that is currently being executed). It's something like @pytest.mark.parametrize but on different level of abstraction.

The way I am doing it currently is by putting these parameters into item.funcargs, which works nicely. The downside is, that they can not be accessed from fixtures. So if I try this:

@pytest.fixture
def my_fixture(client: ClientRole):
    pass

def test_example(client: ClientRole, ldap: LDAPRole, my_fixture):
    pass

Then pytest raises exception because client is not a fixture and it is not available. When digging through the code, I found this commit: 3234c79ee57feb7ec481811af1925bb0f81acdea which is pretty much what I need, only still private. The comment shows the intention to make this public at some point therefore I decided to open this request.

What I would like to have is an option to dynamically register and unregister fixture. So when I am at pytest_runtest_setup hook, I'd like to register a fixture client so it is available to both the test and the function-scoped fixture and I want to unregister the fixture when the code gets into pytest_runtest_teardown.

I hope this clarifies things. Please, let me know if it is still unclear.

[bluetech]

@pbrezina Yes it is my intention to make register_function public in some way. Also see discussion in #11662. There are still some design issues we need to work on, like deciding how to expose it (FixtureManager is private so it needs to be exposed in some other way) and dulling a few sharp edges (clear error if not run during collection; difference between None and '' nodeid).

However, I don't think it it will satisfy your requirements, since in pytest fixtures must be registered during the collection phase, and there is no way to unregister them, while you want to do it on setup/teardown.

[nicoddemus]

Just to comment on this bit:

(FixtureManager is private so it needs to be exposed in some other way)

I don't think we need to expose FixtureManager, perhaps just a public function would suffice?

Say:

def register_fixture(function: Callable, scope: FixtureScope, location: Node) -> None:

[bluetech]

@nicoddemus The implementation itself does need to get at the FixtureManager. So it needs to take config: Config, or alternatively fetch it from a Node.

[nicoddemus]

Yep, that's why I added Node to the signature (besides needing to anchor the fixture at some location for visibility purposes of course).

[bluetech]

Ah missed the type. The minor problem here is one I hinted at above, namely that there a case where node=None (fixturedef.has_location == False), which is subtly different from registering on node=Session (has_location = True):

https://github.com/pytest-dev/pytest/blob/48cb8a2b329d0e8beaf30804c503eddb5e531385/src/_pytest/fixtures.py#L1668-L1676

[nicoddemus]

I see, well no problem adding a config parameter too and make location Optional.

[pbrezina]

Well... having this functionality would be very nice.

In the mean time, I have implemented the following workaround:

def mh_fixture(scope: Literal["function"] = "function"):
    def decorator(fn):
        full_sig = inspect.signature(fn)
        mh_args = []
        for arg, hint in get_type_hints(fn).items():
            if issubclass(hint, MultihostRole):
                mh_args.append(arg)
                continue

        @wraps(fn)
        def wrapper(mh: MultihostFixture, *args, **kwargs):
            if 'mh' in full_sig.parameters:
                kwargs['mh'] = mh

            for arg in mh_args:
                if arg not in mh.fixtures:
                    raise KeyError(f"{fn.__name__}: Parameter {arg} is not a valid topology fixture")

                kwargs[arg] = mh.fixtures[arg]

            return fn(*args, **kwargs)

        cb = wraps(fn)(partial(wrapper, **{arg: None for arg in mh_args}))
        fixture = pytest.fixture(scope="function")(cb)

        partial_parameters = [inspect.Parameter('mh', inspect._POSITIONAL_OR_KEYWORD)]
        partial_parameters.extend([param for key, param in full_sig.parameters.items() if key != 'mh' and key not in mh_args])
        fixture.__pytest_wrapped__.obj.func.__signature__ = inspect.Signature(partial_parameters, return_annotation=full_sig.return_annotation)

        return fixture

    return decorator

But I don't really like that it is using internal pytest knowledge (__pytest_wrapped__.obj.func), though I suppose this stuff can be considered stable?

And of course, it quite a hack and it requires user to use type hints. But this works:

@mh_fixture
def my_fixture(client: Client, request):
    print(client)
    print(request)