Pytest 3 to 4 issues to take into account when handling markers and parametrizations

[goanpeca]

I had a question regarding if the use of markers was the standard way to be able to parametrize fixtures? I was trying to look for documentation on pytest but found this instead

https://docs.pytest.org/en/latest/historical-notes.html#update-marker-code

It might not affect us or maybe it does. it seems the way to parametrize a fixture is with the use of indirect

The documentation in pytest is horrible, let me write an example to explain how parametrize with indirect works.

Basically the thing we want to do is:

How can I pass different values to a fixture so I can have the default fixture, but to use a different parameter like use_working_dir=True

Which led me to https://stackoverflow.com/questions/18011902/py-test-pass-a-parameter-to-a-fixture-function, where they talk that indirect is the official though obscure and with not the clearest documentation.

When we parametrize a fixture and use indirect True, the parameters are available on the request.param object for use. So

import pytest

class Vehicle(object):
    def __init__(self, wheels=4, honks=True):
         self.wheels = wheels
         self.honks = honks

@pytest.fixture()
def vehicle(request):
    param = getattr(request, 'param', None)
    if param:
        kwargs = request.param[0]
        vehicle = Vehicle(**kwargs)
    else:
        vehicle = Vehicle()

    return vehicle

def test_defaults(vehicle):
    assert vehicle.wheels == 4

@pytest.mark.parametrize('vehicle', [[{'wheels': 6, 'honks': False}]], indirect=True)
@pytest.mark.parametrize('other', [(1, ), (2, )])
def test_other(vehicle, other):
    assert vehicle.wheels == 6

In this example the first parametrize is changing things on the fixture, whereas the second one is used as we have in other tests, to run the same test but changing the parameters. The test_other tests runs twice with a modified fixture, on the first run with other = 1 and on the second one with other = 2.

It feels a bit awkward at first glance, but once we get use to this, I believe is the better way to avoid collisions with markers.

If we agree on some convention, we can have the first item be `kwargs and the second things to change on the CONF etc...

[ccordoba12]

I had a question regarding if the use of markers was the standard way to be able to parametrize fixtures?

Actually, we see use several ways of parametrizing fixtures (according to the contributor's taste):

1. Markers.
2. indirect usage: (there are a couple of tests in test_mainwindow.py and in other places).
3. Lazy-fixtures: For instance, lsp_client_and_completion in spyder/plugins/completion/languageserver/tests/test_client.py.
4. Making the fixture return a callable: For instance, lsp_context in spyder/plugins/completion/languageserver/tests/conftest.py.

If we agree on some convention, we can have the first item be `kwargs and the second things to change on the CONF etc...

That was already done by me, for instance in test_opengl_implementation present in test_mainwindow.py.

So, I don't know if there's something else to discuss here.

[goanpeca]

Not much, except that markers might be problematic from 3 to 4.

[goanpeca]

@ccordoba12 I will like to write the guide to testing (for beta5) for new devs to include examples of this.

Regarding

Actually, we see use several ways of parametrizing fixtures (according to the contributor's taste):

Should we favor just a couple and not so many paradigms?

[ccordoba12]

Not much, except that markers might be problematic from 3 to 4.

The changes between Pytest 3 and 4 were big, so I don't know if we could move back to use Pytest 3 (if that's what you're implying, because we're already using Pytest 4).

Should we favor just a couple and not so many paradigms?

The simplest ones are markers, callables and indirect usage (in that order, I think). So we should probably focus on them only (I don't know how lazy fixtures work, to be honest).

[goanpeca]

The changes between Pytest 3 and 4 were big, so I don't know if we could move back to use Pytest 3 (if that's what you're implying, because we're already using Pytest 4).

My bad! I thought we were using pytest3

The simplest ones are markers, callables and indirect usage (in that order, I think). So we should probably focus on them only (I don't know how lazy fixtures work, to be honest).

A lazy fixture is a way to use different fixtures as parameters so for example:

import pytest

@pytest.fixture(params=[
    pytest.lazy_fixture('one'),  # These are evaluated on runtime... hence lazy
    pytest.lazy_fixture('two'),  # These are evaluated on runtime... hence lazy
])
def fixture_one_or_two(request):
    return request.param

@pytest.fixture
def one():
    return 1

@pytest.fixture
def two():
    return 2

def test_func(fixture_one_or_two):
    assert fixture_one_or_two in [1, 2]

I understand the use case, but I think having too many ways of doing things might be confusing.