Allow db access to determine length of parameterized fixture list

[rolfhnelson]

I have a periodically-increasing set of regression test-cases in my test database, but I don't see a documented way to look in the django db and use that data to set the length of my parameterized 'regression_test_data' fixture. As a hack, before 3.0, inside 'pytest_generate_tests' I would check if regression tests are being run and, if so, hit the database using the django db manager to determine the number of regression tests, but even this hack fails during collection on 3.0 with 'Failed: Database access not allowed, use the "django_db" mark to enable it.'

Ideally test generation referenced from a django_db test could be enhanced to automatically allow db access. As an added bonus it would be great if any collection of a django_db test could be completely skipped if the command-line is run to ignore tests marked django_db. If this were implemented in version X.Y, then I could do something like:

# test_my_regression.py
@pytest.mark.django_db
def test_my_regression_for_my_function(my_regression_fixture): ...

# conftest.py
def pytest_generate_tests(metafunc):
    if 'my_regression_fixture' in metafunc.fixturenames:
        # This code won't get called during fast tests, because none of
        # my fast tests use my_regression_fixture, and pytest X.Y is smart
        # enough to not even call pytest_generate_tests on those skipped tests.

        # pytest X.Y checks for django_db tag on test_my_regression_for_my_function,
        # and says 'OK, you can use db', rather than raise error
        regressions = Regression.objects.filter(...)
        metafunc.parametrize('my_regression_fixture', regressions)

-Rolf

[blueyed]

Not really clear to me (and I don't think pytest-django can skip pytest_generate_tests), but this seems to be related: https://github.com/pytest-dev/pytest-django/issues/368.

[rolfhnelson]

I don't think pytest-django can skip pytest_generate_tests

Fair enough. In that case, I would request a supported way that I can use the django object manager from inside pytest_generate_tests in 3.0 without the 'Failed: Database access not allowed, use the "django_db" mark to enable it' error.

[pelme]

This is a bit tricky -- pytest collects test first, then executes them. When the first test that needs the database is executed (the db fixture is requested), then the database will be constructed. Fixtures execute with the tests, not during test collection.

Do you use pytest-djangos database setup? If so, how is the Regression model objects created? By using default pytest-django Regression.objects.all() would just be empty anyways?

Storing test cases in the same database that is also used for testing (being rollbacked, flushed etc between tests) seems error prone.

Can you store your regression data outside of the actual test database? Then you could access it from pytest_generate_tests without any problems.

[rolfhnelson]

By using default pytest-django Regression.objects.all() would just be empty anyways?

Correct, I have a regression db with a schema main db, that stores any persistent regression data. Regression.objects.all() is empty, Regression.objects.using('regression').count() is what I want to access from within pytest_generate_tests (since I can defer everything but the count of tests), and what I access using pytest-django 2.9.1. (I should have included the 'using' in my desired code above).

Can you store your regression data outside of the actual test database? Then you could access it from pytest_generate_tests without any problems.

It's already stored outside the test database, but uses the same schema (using django's manual database selection) If I can't use the convenient django ORM, I'll just do a raw query. Is django.db.connections['regression'] supported at pytest_generate_tests time, or should I not even take that for granted and use the vanilla Python Database API?

[daishi]

I believe I'm hitting this same issue. Have there been any updates regarding this situation?

I can see @pelme 's point regarding trickiness, but perhaps there would be a way to either explicitly pre-provision the DB? Is there a way to directly access the fixture instantiation logic?