pytest-dev / pytest-django

A Django plugin for pytest.
https://pytest-django.readthedocs.io/
Other
1.4k stars 345 forks source link

Unblock not cleaned up at the end of pytest execution? #1147

Closed boxed closed 2 months ago

boxed commented 2 months ago

I realize this is a bit of an edge case, but there seems to be something off about the cleanup for pytest.mark.django_db. If you run pytest twice in the same process all tests that are marked django_db fail with the error reproduced below.

A reproduction repo is at https://github.com/boxed/pytest_django_block_db_issue. Run run_pytest_twice.py after initializing a venv with pytest-django, and django.

(The reason I'm doing this is because I'm working on the next version of mutmut, which is looking like it could go ~100 times faster than the current version, but this issue stops me from trying out out iommi)

I have tracked down the exception to _blocking_wrapper (which was annoyingly hard because of __tracebackhide__), but I'm at a bit of a loss as to what my next step is.

_______________________________________________________ ERROR at setup of test_issue ________________________________________________________

request = <SubRequest '_django_db_marker' for <Function test_issue>>

    @pytest.fixture(autouse=True)
    def _django_db_marker(request: pytest.FixtureRequest) -> None:
        """Implement the django_db marker, internal to pytest-django."""
        marker = request.node.get_closest_marker("django_db")
        if marker:
>           request.getfixturevalue("_django_db_helper")

venv/lib/python3.12/site-packages/pytest_django/plugin.py:533: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
venv/lib/python3.12/site-packages/pytest_django/fixtures.py:144: in django_db_setup
    db_cfg = setup_databases(
venv/lib/python3.12/site-packages/django/test/utils.py:206: in setup_databases
    connection.creation.create_test_db(
venv/lib/python3.12/site-packages/django/db/backends/base/creation.py:78: in create_test_db
    call_command(
venv/lib/python3.12/site-packages/django/core/management/__init__.py:194: in call_command
    return command.execute(*args, **defaults)
venv/lib/python3.12/site-packages/django/core/management/base.py:459: in execute
    output = self.handle(*args, **options)
venv/lib/python3.12/site-packages/django/core/management/base.py:107: in wrapper
    res = handle_func(*args, **kwargs)
venv/lib/python3.12/site-packages/django/core/management/commands/migrate.py:118: in handle
    executor = MigrationExecutor(connection, self.migration_progress_callback)
venv/lib/python3.12/site-packages/django/db/migrations/executor.py:18: in __init__
    self.loader = MigrationLoader(self.connection)
venv/lib/python3.12/site-packages/django/db/migrations/loader.py:58: in __init__
    self.build_graph()
venv/lib/python3.12/site-packages/django/db/migrations/loader.py:235: in build_graph
    self.applied_migrations = recorder.applied_migrations()
venv/lib/python3.12/site-packages/django/db/migrations/recorder.py:89: in applied_migrations
    if self.has_table():
venv/lib/python3.12/site-packages/django/db/migrations/recorder.py:63: in has_table
    with self.connection.cursor() as cursor:
venv/lib/python3.12/site-packages/django/utils/asyncio.py:26: in inner
    return func(*args, **kwargs)
venv/lib/python3.12/site-packages/django/db/backends/base/base.py:320: in cursor
    return self._cursor()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <DatabaseWrapper vendor='sqlite' alias='default'>, name = None

    def _cursor(self, name=None):
        self.close_if_health_check_failed()
>       self.ensure_connection()
E       RuntimeError: Database access not allowed, use the "django_db" mark, or the "db" or "transactional_db" fixtures to enable it.

venv/lib/python3.12/site-packages/django/db/backends/base/base.py:296: RuntimeError
boxed commented 2 months ago

I think what happens is that _dj_db_wrapper saves a copy to the "real implementation", but that misfires in this situation and stores the blocking implementation instead.

If I do this:

        if self._real_ensure_connection is None:
            assert 'blocking' not in BaseDatabaseWrapper.ensure_connection.__name__
            self._real_ensure_connection = BaseDatabaseWrapper.ensure_connection

that assert fails the second run

boxed commented 2 months ago

Success! If I do:

        if not hasattr(BaseDatabaseWrapper, '_real_ensure_connection'):
            BaseDatabaseWrapper._real_ensure_connection = BaseDatabaseWrapper.ensure_connection
        self._real_ensure_connection = BaseDatabaseWrapper._real_ensure_connection

then this bug disappears

boxed commented 2 months ago

PR: https://github.com/pytest-dev/pytest-django/pull/1148