Description

Brief

I am trying to provide pytest a idfn after it has generated a large combinatorial dataset via modular parametrized fixtures.

Context

pytest.mark.parametrize is really useful for quickly generating multiple sets of data for a particular test. It's particularly great when stacked to generate all combinations of multiple parametrized arguments.

pytest provides a mechanism for automatically generating a test id from the params based on whether the param is primitive/non-primitive. It also allows the user to provide an idfn so that the test input can be expressed in a way more relevant to the problem domain.

@pytest.fixture is allowed to be parametrized via params. It is also modular and can use other fixtures, which in turn can be parametrized, but it need not be aware of that.

Using the above 3 properties of pytest, I am able to pass in a single fixture to a test function that uses two other parametrized fixtures, and converts their raw values into a namedtuple that better reflects the problem domain.

However, I am more interested in the composite data produced by the single fixture than the primitive data it acquires from the other fixtures, and would like to reflect this in the output of pytest.

I have tried:

to pass a idfn to @pytest.fixture(ids=
to pass a idfn to @pytest.mark.parametrize(..., ids=
to implement pytest_make_parametrize_id in conftest.py

all without success.

pip list

Package            Version
------------------ -------
atomicwrites       1.3.0
attrs              19.3.0
importlib-metadata 0.23
more-itertools     7.2.0
packaging          19.2
pip                19.3.1
pluggy             0.13.0
py                 1.8.0
pyparsing          2.4.5
pytest             5.2.2
setuptools         41.6.0
six                1.13.0
wcwidth            0.1.7
wheel              0.33.6
zipp               0.6.0

versions

platform darwin -- Python 3.7.3, pytest-5.2.2

Example

# regex.py

import re

C_DEFINE_REGEX = re.compile(r'#define\s([a-zA-Z]+)\s(\d+)')

# test_regex.py

import pytest

from collections import namedtuple
from regex import C_DEFINE_REGEX

C_Define = namedtuple("C_Define", ["name", "value", "string"])

@pytest.fixture(params=["foo", "FOO"])
def c_define_name(request):
    return request.param

@pytest.fixture(params=["1", "10"])
def c_define_value(request):
    return request.param

@pytest.fixture
def c_define(c_define_name, c_define_value):
    string = f"#define {c_define_name} {c_define_value}"

    return C_Define(c_define_name, c_define_value, string)

def test_c_define_regex(c_define):
    match = C_DEFINE_REGEX.match(c_define.string)

    assert match.group(1) == c_define.name
    assert match.group(2) == c_define.value

Output

===== test session starts =====
platform darwin -- Python 3.7.3, pytest-5.2.2, py-1.8.0, pluggy-0.13.0 -- /{path}/pytest-issue/venv/bin/python
cachedir: .pytest_cache
rootdir: /{path}/pytest-issue
collected 4 items

test_regex.py::test_c_define_regex[foo-1] PASSED                [ 25%]
test_regex.py::test_c_define_regex[foo-10] PASSED               [ 50%]
test_regex.py::test_c_define_regex[FOO-1] PASSED                [ 75%]
test_regex.py::test_c_define_regex[FOO-10] PASSED               [100%]

===== 4 passed in 0.06s =====

Desired Output

test_regex.py::test_c_define_regex[#define foo 1] PASSED          [ 25%]
test_regex.py::test_c_define_regex[#define foo 10] PASSED         [ 50%]
test_regex.py::test_c_define_regex[#define FOO 1] PASSED          [ 75%]
test_regex.py::test_c_define_regex[#define FOO 10] PASSED         [100%]

Quickfix

For this small scenario it's possible to write something along the lines:

@pytest.mark.parametrize("c_define", [
    get_c_define(foo, 1),
    ...
    get_c_define(FOO, 10),
    ],
    ids=lambda v: v.string
)
def test_c_define_regex(c_define):
    ...

Which will achieve the desired output, but at the cost of easily extending the tests.

Flaws in the regex could quickly be found with test params:

c_define.name = "foo_bar"
c_define.value = "0xF"
c_define.string = "#define{space}{space}foo{space}{space}1"

And it would be much easier to append these instances to a pytest.mark.parametrize list.

blueyed commented 5 years ago

Only skimmed the issue, but sounds like idfnset (to generate IDs for the complete set, given index, argvalues, item as arguments) might help? (ref https://github.com/blueyed/pytest/pull/104, after https://github.com/pytest-dev/pytest/pull/6174).

hamish-miller commented 5 years ago

Thanks for the quick response.

generating IDs for the complete set

sounds like what I'm trying to achieve.

I'll have a look at your PRs to verify.

pytest-dev / pytest

Is it possible to assign an idfn after stacked parameterization? #6185