New scope designed for parametrized tests functions

[pfreixes]

Today I came across with an issue that I couldn't solve nicely with pytest, basically due to the lack of another level of granularity for the scope related with the fixtures, so let me give you an example:

class TestFoo:
    @pytest.fixture
    def db(self):
        db.insert(['foo', 'bar'])
        yield db
        db.delete()

    @pytest.parametrize('param',[
        'foo',
        'bar'
    ])
    def test_search(self, db):
        assert db.search(param)

    def test_insert(self, db):
        assert db.insert('x') 

The previous snippet shows a simple example of a test class that has two different tests, one of them parametrized that will end up executing many times the same test and another test function that will be called just once, it's not parametrized. The fixture will be instantiated as many times as many tests we have, also considering the different instances of the test_search that is parametrized.

If we want to reuse the fixture, let's think that the fixture implies insert many documents in a DB and later in the teardown delete these documents, we have to change the scope of the fixture. Perhaps using the class scope.

As the example shows the fixture in the first test does not imply mutate it, so can be reused securely at least within the first test. The main problem with this scenario is the test that checks the insert, this test will mutate the fixture so the class scope can't be used as it is.

So to circumvent this issue, either we have to move the tests to another class to avoid the intersection or create some specific fixtures ad-hoc within the same class.

My proposal here is implementing a new scope called parametrize. This scope will suppose for those tests that are parametrized reuse the fixtures that have been marked with this scope and for those test that are not parametrized the scope will be considered just as a function scope.

Comments?

[nicoddemus]

Hi @pfreixes thanks for writing.

IIUC this new scope is useful only for parametrized tests which don't mutate the fixture. Wouldn't that be dangerous to misuse, for example if people later change the test or the fixture so this operation is no longer safe? I'm just thinking that this might be useful only in a very narrow set of scenarios, so not much useful.

[RonnyPfannschmidt]

its a sensible idea, but wrong in some sense for the scope name, and the usage since it completely ignores parameterization at higher scopes

however a less problemmatic spcification is in line with making the currently internal FunctionDefinition part of the actual collection tree, that would have the scope "funcdef" and would sit in the middle between higher scopes and actual functions

[pfreixes]

@nicoddemus

IIUC this new scope is useful only for parametrized tests which don't mutate the fixture.

The same as the other scopes that impose share the fixture between different execution tests, so I wouldn't consider this as an anti-pattern raised because of this feature.

@RonnyPfannschmidt

its a sensible idea, but wrong in some sense for the scope name, and the usage since it completely ignores parameterization at higher scopes

Naming is hard, let's find a better name. But the idea is the same. Regarding the usage since it completely ignores parameterization at higher scopes, what do you mean?

[nicoddemus]

The same as the other scopes that impose share the fixture between different execution tests, so I wouldn't consider this as an anti-pattern raised because of this feature.

True, I was thinking that might be more surprising in a parametrized-scoped fixture, but you are absolutely right.

[brianmaissy]

I don't see an inherent connection in the fact that the test in which you want the fixture to be scoped differently is parameterized. A feature that behaved this way would be useful only in your very specific use case. What if I wanted to parametrize both the non-destructive and the destructive tests?

It seems to me that the problem you really want to solve is a way to define a fixture once and then use it in multiple scopes.

The easy way to do this without a new feature is just to define the fixture twice, db_functionscope and db_classscope, but I understand that this smells of duplication.

Maybe it would help if we could somehow mark a fixture as being multiple-scoped, and add a way for the test function to specify which scope it wants.

Or, a slight variation, a way for the test function to say, "Hi, I'm going to damage my fixture. Even though it's not function-scoped, delete it and recreate it after calling me". Something like pytest.mark.use_disposable_fixture('db').

[nicoddemus]

Closing this in favor of https://github.com/pytest-dev/pytest/issues/1552.

[RonnyPfannschmidt]

@nicoddemus unlike #1552 i believe this one is a lot more doable (and lined up as part of the node cleanup)

[nicoddemus]

Sorry, thanks for double checking. 👍