Parametrize based on another parameter

[Sup3rGeo]

Hi,

This feature is related to a concept I think that pytest is missing. Let's say I have this very simple test:

@pytest.mark.parametrize("param", [1,2,3])
def test(param):
    mysetupcode(param)
    myassertfunc(param)

Now, I want to refactor the setup as a fixture, but still have the same parametrization for both the test and the fixture. In that case, I end up having to create a fixture:

@pytest.fixture(params=[1,2,3])
def param():
    # This is not a real fixture because it does not run any code
    return request.param

@pytest.fixture
def mysetupcode(param):
    # ... setup code
    pass

def test(param, mysetupcode):
    pass

So basically the concept that pytest is missing is this param fixture that is not a real fixture as does not run any code, but is just a parameter broadcast entity that is different from parametrizing at the individual fixture/test level.

Another example where this would be useful, with some API suggestions:

datasetA = [data1_a, data2_a, data3_a]
datasetB = [data1_b]
datasetC = [data1_c, data2_c]

# HERE the concept missing. The following line is equivalent to:
# @pytest.fixture(params=[datasetA, datasetB])
# def dataset(request)
#     return request.param
# Also you can think as a pytest.mark.parametrize not tied to any function
pytest.parameterize('dataset', [datasetA, datasetB])

# Fixtures could use it just as well (as it is just like a fixture)
# Actually becomes another way to parametrize
@pytest.fixture
def fixture1(dataset):
    setup(dataset)
    pass

# tests can use this concept as well, as we would with fixtures
def test_one(dataset)

# Following line is almost as if we are parametrizing the parametrize
# So it will be doubly parametrized: 
# test_data[datasetA-data1_a]
# test_data[datasetA-data2_a]
# test_data[datasetA-data3_a]
# test_data[datasetB-data1_b]
# test_data[datasetC-data1_c]
# test_data[datasetC-data2_c]
@pytest.mark.parametrize('data', pytest.parameter('dataset')])
def test_data(data):
    #do test

This feature was discussed in https://github.com/TvoroG/pytest-lazy-fixture/issues/25 https://github.com/pytest-dev/pytest/issues/349#issuecomment-339658616

I decided to open a new feature request because I believe it is a different use case/implementation than #349, although they might be related. The difference is that we don't need to worry about this that @nicoddemus mentioned:

Under the current design, pytest needs to generate all items during the collection stage. Your examples generate new items during the session testing stage (during fixture instantiation), which is not currently possible.

Because we don't need to run any code to have all the parameters in place at collection time.

[RonnyPfannschmidt]

covariant parameterize is long-sought ^^

[nicoddemus]

Gave a quick and dirty stab at this:

# conftest.py
import attr

import pytest

@attr.s
class Covariant:
    name = attr.ib()
    values = attr.ib()

    def make_fixture(self):
        @pytest.fixture(name=self.name, params=self.values)
        def fixture_func(request):
            return request.param

        return fixture_func

def pytest_configure():
    pytest.covariant_param = Covariant

@pytest.hookimpl(hookwrapper=True)
def pytest_collect_file():
    outcome = yield
    modules = outcome.get_result()
    for module in modules:
        if isinstance(module, pytest.Module):
            print(module)
            for name in dir(module.obj):
                member = getattr(module.obj, name)
                if isinstance(member, Covariant):
                    fixture = member.make_fixture()
                    setattr(module.obj, name, fixture)
                    print(fixture)
            module.session._fixturemanager.parsefactories(module)

Usage:

import pytest

dataset = pytest.covariant_param('dataset', [1, 2, 3])

def test_foobar(dataset):
    pass

λ pytest .tmp\covariant\ -v
======================== test session starts ========================
platform win32 -- Python 3.6.6, pytest-3.10.1.dev78+gb83e9780.d20181119, py-1.6.0, pluggy-0.8.0 -- c:\pytest\.env36\scripts\python.exe
...
collected 3 items

.tmp\covariant\test_foo.py::test_foobar[1] PASSED              [ 33%]
.tmp\covariant\test_foo.py::test_foobar[2] PASSED              [ 66%]
.tmp\covariant\test_foo.py::test_foobar[3] PASSED              [100%]

===================== 3 passed in 0.03 seconds ======================

But the idea is to do some post processing after collection, where we look for Covariant instances and replace them with proper fixtures.

It needs some refining to collect from test classes as well, possibly other features like scope and ids, but I believe the underlying implementation is surprisingly solid because we are using fixtures behind the scenes. It might be missing some functionality as I only tested with test_foo.py above.

This can easily be done in an external plugin too.

[Sup3rGeo]

Hey @nicoddemus that is really interesting! However, although it tackles the first thing I described (the missing concept of a independent parameter), I believe it still would not have the covariant aspect which is basically the "parametrization of second order" (parameters coming from another parameter - hence the covariance). Or maybe I am getting the terms wrong as well! Anyway we call it, any ideas for this?

[nicoddemus]

Sorry about that, wrote that right before going to bed.

believe it still would not have the covariant aspect which is basically the "parametrization of second order"

You mean this last example right?

# Following line is almost as if we are parametrizing the parametrize
# So it will be doubly parametrized: 
# test_data[datasetA-data1_a]
# test_data[datasetA-data2_a]
# test_data[datasetA-data3_a]
# test_data[datasetB-data1_b]
# test_data[datasetC-data1_c]
# test_data[datasetC-data2_c]
@pytest.mark.parametrize('data', pytest.parameter('dataset')])
def test_data(data):
    #do test

Perhaps extending the idea further, I think this might be possible to implement:

import pytest

datasetA = pytest.covariant_param('datasetA', ['data1_a', 'data2_a', 'data3_a'])
datasetB = pytest.covariant_param('datasetB', ['data1_b'])
datasetC = pytest.covariant_param('datasetC', ['data1_c', 'data2_c'])

dataset = pytest.covariant_param('dataset', [datasetA, datasetB, datasetC])

def test_foobar(dataset):
    pass

What do you think?

[Sup3rGeo]

If it results in the set of tests described in the comments above the test function, then I think it should do the job!

Probably we would have to iterate a bit to get the interface 100% correct but it seems to be going in the right direction

[smarie]

Just for the sake of traceability, @Sup3rGeo you requested this feature in pytest-cases and it is now available.

Note: when it is officially fixed in pytest the pytest-cases version will fallback on it, but I will keep the current implementation available for older versions of pytest.