Using Fixtures as Parameters

[jgersti]

Discussed in https://github.com/pytest-dev/pytest/discussions/11412

^{Originally posted by **jgersti** September 7, 2023} In #11284 @RonnyPfannschmidt mentioned that he would like to incorporate `pytest-lazy-fixture`[^1] and building blocks/hooks for `pytest-cases`[^2] like behaviour into the `pytest` core. I have experimented the last month with a `pytest-cases`[^2] replacement since its author has not been active for some time. I would like to share some my observations and conclusion to start a discussion about what the scope of the feature should be and what steps need to be taken to implement it. I will start by giving a brief overview what these plugins do and how they work, followed by my opinion what is in scope for addition to the core of `pytest`. In case of `pytest-cases` this description is heavily simplified. ## Plugin Overview ### `pytest-lazy-fixture`[^1] The plugin provides a single feature: it introduces a `LazyFixture` object to reference a fixture inside `pytest.mark.parametrize` and the `params` argument of `pytest.fixture`. Because the referenced fixture maybe parametrized itself or have a parametrized dependency, a `pytest_generate_tests` hook is used after the core test generation to discover transitive parameters by inspecting each `Metafunc.callspec`s `funcargs` and `params` entries for `LazyFixture` objects and __recursively descending__ until no further parameters are found. This is done by recalculating a new fixture closure with the referenced fixture included and calling `FixtureManager.pytest_generate_tests` with a new (deep) copied `Metafunc` object with the new fixture closure and finally replacing the `callspec`s in the old `Metafunc` are replaced by the newly calculated ones when ascending. Because the added fixture names in the new fixture closure cannot be passed to the `Metafunc` object without influencing all calls, they are dropped at this point and in turn are __not__ considered when reordering the tests and higher scoped fixture maybe initialised late and/or multiple times. Also since the additional parametrizations are applied last instead of in order of the dependencies the parameter id order is wrong. The `LazyFixture` objects are resolved by using the `pytest_run_setup`, `pytest_fixture_setup` and `pytest_run_call` hooks. In `pytest_runtest_steup` `item._request._fillfixtures` is replaced with a wrapper that inspects `item.callspec.params` and `item.funcargs` and resolves found `LazyFixtures` before calling the original `_fillfixture` method. During iteration `item.callspec.params` is reordered to account for dependency order and scopes. In `pytest_fixture_setup` `request.param` is inspected and if it is a `LazyFixture` resolved, resp. in `pytest_runtest_call` each `item.funcargs` entry is inspected and resolved. ### `pytest-cases`[^2] The plugin provides an (_opiniated_[^3]) unified alternative to conventional parametrization. It features a two new decorators for parametrization, `parametrize` and `parametrize_with_cases`, a replacement decorator for `pytest.fixture` and lazily evaluated functions as parameters and fixture reference in form of `FixtureRef` (_`parametrize` can auto detect fixtures and wrap them into a `FixtureRef`_). The plugin also provides some more features but to keep the description short I will concentrate on the mentioned featured minus lazy functions I will also not delve into how cases are discovered and just assume we are given a list of functions as cases. The basic idea is offload parametrization to the new `parametrize` decorator and use it for test functions and fixtures (_as well as cases_) to have a unified UX. This is achieved by wrapping the function that is decorated to manipulate the signature, creating an intermediate fixture that is parametrized and creating a parameter fixture for each parameter (_if the parameter is a `FixtureRef` this parameter fixture is obviously skipped and the referenced fixture is used_). The intermediate fixture depends on all parameter fixtures but during test collection and execution fixtures that are created using the new fixture decorator are selectively disabled. Indirect parametrization is not allowed if any new feature is used. The new fixture decorator does not support the `params` argument but detects parametrization marks and create the parametrization similar to the above mechanism. It also wraps the decorated function to inject the mechanism for skipping. Case parametrization decorator takes a list of function (or a class) and applies the fixture decorator to it and forwards `FixtureRef`s for these to `parametrize`. To achieve the proper parametrized test and proper skipping of all the parameters that are not currently active, the plugin replaces `FixtureManager.getfixtureclosure` and wraps `Metafunc.parametrize` to inject facades into `FuncFixtureInfo.names_closure` and `Metafunc.callspec`. _Note: This is a extremely simplified description that hopefully conveys the major points._ ## Feature Scope While a unified parametrization UX would be nice to have it is most definitely out of scope because it would break most existing code bases and would involve quite a bit of black magic behind the curtains. What I think is in scope is a reimplementation of `pytest-lazy-fixture`, though I would prefer a name like `FixtureRef`/`FixtureReference` because it better conveys the intended usage/meaning, and the proper calculation of the dependency graph tied to the `Callspec2` objects instead of the `Metafunc` object. I think that test reordering might also need to be touched. But with a feature that allows fixtures in parametrization and proper dependency calculation writing a plugin the behaves similar to `pytest-cases` and offers a unified parametrization UX is relative simple. As already stated i have an experimental (internal) implementation for an replacement uses a heavily modified version of `pytest-lazy-fixtures` under the hood that mostly works but does some sketchy stuff to inject dependencies. ## Proposed Changes Following is a loose and incomplete list of changes/tasks i would propose to tackle this. - Investigate and implement calculation of the complete and exact[^4] dependency graph instead of the current fixture closure. Then `names_closure`/`fixturenames` is the iteration in topological sort order, if this order does not exist the graph has an cycle and is invalid. This would also address #5303, #11350 and maybe #2844. _I am aware that this is computationally expensive. But I think it is either this or recursive algorithms further down the line._ - Do not share the `FuncFixtureInfo` object between all calls to a test and attach it to the callspecs instead of `MetaFunc`. This is similar to what is included in #11298. - Extend `Metafunc.parametrize` to recalculate the dependency graph. For now this would be only to prune the graph in case of direct parametrization. - Implement `LazyFixture`/`FixtureRef`. This can be a simple `dataclass` with the name of the fixture and an optional field id for parameter id generation. - Extend/reimplement `Metafunc.parametrize` to recalculate the dependency graph by adding branches and iterate along it to discover all parametrizations. Ideally this is done none-recursive. - Check if the dependency graphs can be used elsewhere. Reordering? Fixture Setup? - Check how ordering in deeply parametrized dependencies of higher scopes is impacted. I would like to reiterate that this post is intended as a starting point for a discussion and not as an definite 'this needs to happen' roadmap. I would appreciate feedback, comments and any help in making this happen. [^1]: https://github.com/TvoroG/pytest-lazy-fixture [^2]: https://github.com/smarie/python-pytest-cases [^3]: my words, not the authors [^4]: i.e. resolved to a single `FixtureDef` instead all of them.

[glatterf42]

Would love to see pytest-lazy-fixture become a part of core pytest since the existing outside solutions are not compatible with pytest 8.x (and not maintained) or missing functionality, it seems.

[nicoddemus]

Previous discussion on the topic: #3244.

Now that pytest-lazy-fixture is no longer being maintained (it seems), I definitely agree that this feature should be integrated into the core.

My hunch is that because it can be implemented directly in the internals, the implementation can be simpler than pytest-lazy-fixture itself.

[jgersti]

Unfortunately it is not as simple. Both pytest-cases and pytest-lazy-fixture have bugs with transitive dependencies, because pytest-cases explicitly and pytest-lazy-fixture implicitly assume that the dependencies form a tree. In reality the dependencies from a DAG (directed acyclic graph) and if there is a diamond pattern in the transitive dependencies both plugins can break by either not loading a required fixture, loading too many fixtures, and/or using inappropriate parametrization.

As I tried to convey in the post above and the attached discussion, I think the first step should be fixing/refactoring the dependency calculation and internal depenency representation in pytest itself, so so the dependency graph is explicitly known and it can be manipulated . There is a experimental branch where I implemented an algorithm to build the graph, but i realized quite fast that the current priority of autouse fixtures is difficult to replicate. I did take some notes which tests failed with some guesses as to why, but i am currently unable to find these.

This change more or less equates to ripping out most/all of the fixture dependency and replacing them.