Keyword-based parameter sets

[ikonst]

In many of my tests I've been using pytest.param(..., id="...") as means of providing more readable identifiers for my tests inline, since otherwise with multiple parameters, the autogenerated id is hard to read.

Some of my parameter-driven tests end up growing beyond 2-3 arguments and maintaining the parameters positionally becomes error-prone and hurts readability. The reader has to mentally "zip" a list of parameters (of a parameter set) to the parameter names defined earlier, and if you need to add a parameter, now you need to add them to each and every test, and if you want to insert it (e.g. to maintain grouping), you'd have to carefully count parameters in each of the sets.

One pattern I've been introducing in such cases is defining a "ParamSet" attrs base class:

import attr

@attr.s(auto_attribs=True, kw_only=True)
class ParamSet:
    """
    Use as base class for sets for parameters for @pytest.mark.parametrize
    when you find yourself passing too many parameters positionally.
    """

    id: str

    @property
    def __name__(self) -> str:  # indicate the id to pytest
        return self.id

From there, a test would look like:

@attr.s(auto_attribs=True, kw_only=True)
class MyComplexTestParamSet(ParamSet):
   foo: int
   bar: str = 'some_default'
   expected_baz: float

@pytest.mark.parameterize('params', [
   MyComplexTestParamSet(foo=42, expected_baz=42.42, id="happy path"),
   ...
])
def test_complex(params: MyComplexTestParamSet):
   ...

Before we discuss specifics, is this something we'd want pytest.param to enable?

To give an idea, I'd imagine usage like:

@pytest.mark.parameterize('params', [
   pytest.param(foo=42, expected_baz=42.42, id="happy path"),
   ...
])
def test_complex(params: Any):
   ...

or something more typed like this: (I like the fact that the type hint enables better developer experience when working on the test)

class MyComplexTestParamSet(pytest.ParameterSet):
  foo: int
  ...

[RonnyPfannschmidt]

I believe you just described namedtupe/dataclasses

[ikonst]

@RonnyPfannschmidt I'm essentially using dataclasses (the attrs variant of). The only interesting parts are:

• the base class (that I'm reusing in various tests) makes you provide an 'id' (encourages naming param-sets) and it id's itself (that's why it has a __str__)
• I like attr's concept of enforcing kw-only (namedtuple lacks that)
• lacking kw-only, all namedtuple's defaults must be ordered last

As you can see, I already have a pattern in practice. What I'm wondering is whether I can put an end to copy-pasting this "paramset.py" between my projects, by instead getting pytest.param (or perhaps the currently-private API ParameterSet) to encompass a pattern for keyword params.

If we don't want to enable derived types, then perhaps:

1. having pytest.param access kwargs and store them in a "ParameterSet.named_values" namedtuple field

If we do, then perhaps:

1. making ParameterSet public (right now it's not "exported")
2. basing it on top of attrs instead of namedtuple to allow kw-only and arbitrary default ordering

I don't want to be too prescriptive, so I'm trying to gauge first if it sounds like a worthwhile change in general.

[RonnyPfannschmidt]

I would like to see more details on your actual use case

My initial instinct is to provide something like

def as_param(self, id=None, marks=None):
  return pytest.param(self, marks=marks, id=id) 

But that may very well miss the mark for your particular use case.

However its critical to make a good initial choice on where the complexity will be as the down payment will be a drag otherwise.

[The-Compiler]

Personally I'd love to have some kind of syntax to pass keyword arguments into parametrize, without having to declare a dataclass for every test. However, using pytest.param that way poses a challenge: It itself has keyword arguments to pass meta information about the parameter, namely id and marks. That means that it wouldn't be possible to pass those to tests, but also, perhaps more importantly, it wouldn't allow us to add new arguments to pytest.param in the future.

I have some more thoughts in a comment on an earlier related issue (which was closed by the author): https://github.com/pytest-dev/pytest/issues/7568#issuecomment-665628455

[kalekundert]

Another possible syntax that I don't think I saw mentioned is:

pytest.param(id=..., meta=...).values(a=..., b=....)

This would require that no parameters be specified in the constructor, but doesn't limit either the arguments to param or the parameters themselves.

[ikonst]

it wouldn't allow us to add new arguments to pytest.param in the future

That's a great point. Indeed id and marks I can see myself giving up on, but wouldn't want to block evolution. We can opt for a "builder pattern" like @kalekundert is suggesting. Perhaps we can agree that passing named args is the more common and "trivial" use case, and reserve the "builder pattern" for the features we cannot foresee yet.

pytest.param(foo=..., bar=..., id="what I'm testing").feature_i_cannot_foresee(42).another_feature('foobar')

[ikonst]

I think my proposal was conflating a few things:

1. Named parameters in parameter sets.
2. Accepting parameter sets as a single argument. 2.1. ... which is a data-class.

Most of us can agree on the utility of (1) in complex tests. The motivation stems from: a. Difficulty reading and updating parameter sets. b. Difficulty mapping between parameter-set arguments and function arguments, in tests which also use fixtures.

The second reason has also been the motivation for (2) and (2.1). In complex tests with dozens of arguments, it's hard to tell whether an argument is provided by a fixture, a parameter-set or perhaps a decorator like @unittest.mock.patch (the latter make it particularly nefarious since they're also positional, and apply in "reverse order" to the decorator's stacking).

I do find (2) and (2.1) to be slightly unsavory. It makes tests bulkier and splits them across two units of code, but perhaps what I find the most unsavory is - that the idiomatic mechanism for passing parameters in Python should be function args, while here we'd be forfeiting it to dependency injection and relegating parameters/arguments to a second-tier mechanism.

Another rationale for (2.1) is type-checking. Perhaps effort should be instead put into implementing a pytest plugin for mypy, which would impose the test function's type annotations on the parameters passed to pytest.param. Though, of course, type-checking tests is of secondary importance really...

I'd love to hear more of your experiences with complex tests (involving parameters and fixtures) and hear what you've been doing to keep it maintainable.

[RonnyPfannschmidt]

i would propose experimentations with other ways to spell not just pram, but parametrize outside of pytest so we can have experimentation, its very easy to get things wrongly involved and i still have some horrors from the mark smearing, (and marks are still not sanely represented )

[kalekundert]

I'd love to hear more of your experiences with complex tests (involving parameters and fixtures) and hear what you've been doing to keep it maintainable.

To me, the biggest maintainability issues with parametrized tests are (i) long parameter lists end up dwarfing the test code and (ii) python syntax is not very conducive to data entry (e.g. multiline-string parameters are always hard to read). I wrote a package called parametrize_from_file that solves both problems by loading parameters from external files (e.g. YAML, TOML, etc). It doesn't address exactly the same issues that you've brought up here, but I'd recommend giving it a look; it works really well for me.

[ntextreme3]

I came here looking for ultimately the same thing: a way to make passing long lists of params to parametrize more readable (without creating a custom class each time). With many params per test case, it can be hard to keep track of which param (by index) aligns with which of the function arguments.

Another thing I thought would be nice is if you could use dict to label all the params (in the case you don't really care about providing a custom test ID). Along the lines of:

def dict_parametrize(test_cases: list[dict]):
    keys = list(test_cases[0].keys())
    params = [list(d.values()) for d in test_cases]
    return pytest.mark.parametrize(keys, params)

@dict_parametrize(
    [
        {
            "a": 1,
            "b": 2,
            "c": 3,
        },
        {
            "a": 2,
            "b": 2,
            "c": 3,
        },
    ]
)
def test(a, b, c):
    assert a == 1 and b == 2 and c == 3

(I know there's a lot this doesn't account for, just throwing it out there)

Edit: after digging around, looks like something like this was already suggested in the issue @The-Compiler linked to, ex: https://github.com/pytest-dev/pytest/issues/7568#issuecomment-665966315

[seandstewart]

Popping in here to say I wrote a plugin which solves for this limitation as well after submitting my own issue (#10518): https://github.com/seandstewart/pytest-parametrize-suite

@pytest.mark.suite(
    case1=dict(arg1=1, arg2=2),
    case2=dict(arg1=1, arg2=2),
)
def test_valid_suite(arg1, arg2):
    # Given
    expected_result = arg1
    # When
    result = arg2 - arg1
    # Then
    assert result == expected_result

@pytest.mark.suite(
    case1=dict(arg1="cross1"),
    case2=dict(arg1="cross2"),
)
@pytest.mark.suite(
    case3=dict(arg2="product1"),
    case4=dict(arg2="product2"),
)
def test_suite_matrix(arg1, arg2):
    # Given
    combination = arg1 + arg2
    # When
    possible_combinations.remove(combination)
    # Then
    assert combination not in possible_combinations

possible_combinations = {
    "cross1product1",
    "cross1product2",
    "cross2product1",
    "cross2product2",
}

Essentially, the top-level keyword argument is your test ID. Then you pass in a mapping of {argname->argvalue, ...}.

The plugin does the work of validating that the structure of all the mappings for a given marker are the same structure, then translates it into the standard parametrize input and passes it on to metafunc.parametrize.

All-in-all, it's fairly simple and naive, but it results in parametrized tests which are extremely easy to read, reason about, and maintain while providing a clean test output.

I don't say all this to advertise for myself. Rather, I think this interface is an improvement over the standard interface and would love to see a version of it accepted into pytest core. Until then, I have my plugin to bridge the gap.

[RonnyPfannschmidt]

The proposed plugin seems to intentionally ignore pytest.param to make the example of the builtin way look as painful as possible, as such I'm taken back

[seandstewart]

The proposed plugin seems to intentionally ignore pytest.param to make the example of the builtin way look as painful as possible, as such I'm taken back

Apologies, that's definitely not my intent. I've got skin in the game only insomuch as I want a simple way to define these parameters. You're correct, I didn't include an example using pytest.param, but this is more-so because I've never seen it used beyond a few examples in the documentation. By and large, I've only really seen folks just use the argnames and friends.

I work in pytest everyday for my job and for my other side projects, and I reference the documentation frequently, all in all, I'm a huge fan. The use of pytest.param has never been immediately obvious to me, but I stand to be corrected.