django-pytest raising DatabaseError in all tests

[Mendes11]

Hi,

I was using django-pytest version 3.3.2 and my tests using Django TestCase were running just fine. Now I started a new project and decided do use the latest django-pytest version, but all my tests are raising an error related to the default database being created in one thread but used in another.

Python: 3.7.2+and 3.6.9 Django Version: 2.2.9 and 2.2 pytest Version: 5.3.2 pytest-django Version: 3.8.0 psycopg2-binary: 2.8.4 (I'm using Postgres)

pytest.ini:

[pytest]
DJANGO_SETTINGS_MODULE = data_api.settings
python_files = tests.py test_*.py *_tests.py

The error:

request = <SubRequest '_django_setup_unittest' for <TestCaseFunction test_data_insert_event_dispatched>>
django_db_blocker = <pytest_django.plugin._DatabaseBlocker object at 0x7fbe81b51550>

    @pytest.fixture(autouse=True, scope="class")
    def _django_setup_unittest(request, django_db_blocker):
        """Setup a django unittest, internal to pytest-django."""
        if not django_settings_is_configured() or not is_django_unittest(request):
            yield
            return

        from _pytest.unittest import TestCaseFunction

        if "debug" in TestCaseFunction.runtest.__code__.co_names:
            # Fix pytest (https://github.com/pytest-dev/pytest/issues/5991), only
            # if "self._testcase.debug()" is being used (forward compatible).
            from _pytest.monkeypatch import MonkeyPatch

            def non_debugging_runtest(self):
                self._testcase(result=self)

            mp_debug = MonkeyPatch()
            mp_debug.setattr("_pytest.unittest.TestCaseFunction.runtest", non_debugging_runtest)
        else:
            mp_debug = None

>       request.getfixturevalue("django_db_setup")

/home/mendes/.virtualenvs/data_domain/lib/python3.7/site-packages/pytest_django/plugin.py:524: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/home/mendes/.virtualenvs/data_domain/lib/python3.7/site-packages/pytest_django/fixtures.py:108: in django_db_setup
    **setup_databases_args
/home/mendes/.virtualenvs/data_domain/lib/python3.7/site-packages/django/test/utils.py:174: in setup_databases
    serialize=connection.settings_dict.get('TEST', {}).get('SERIALIZE', True),
/home/mendes/.virtualenvs/data_domain/lib/python3.7/site-packages/django/db/backends/base/creation.py:60: in create_test_db
    self.connection.close()
/home/mendes/.virtualenvs/data_domain/lib/python3.7/site-packages/django/db/backends/base/base.py:279: in close
    self.validate_thread_sharing()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <django.db.backends.postgresql.base.DatabaseWrapper object at 0x7fbe810e3a90>

    def validate_thread_sharing(self):
        """
        Validate that the connection isn't accessed by another thread than the
        one which originally created it, unless the connection was explicitly
        authorized to be shared between threads (via the `inc_thread_sharing()`
        method). Raise an exception if the validation fails.
        """
        if not (self.allow_thread_sharing or self._thread_ident == _thread.get_ident()):
            raise DatabaseError(
                "DatabaseWrapper objects created in a "
                "thread can only be used in that same thread. The object "
                "with alias '%s' was created in thread id %s and this is "
                "thread id %s."
>               % (self.alias, self._thread_ident, _thread.get_ident())
            )
E           django.db.utils.DatabaseError: DatabaseWrapper objects created in a thread can only be used in that same thread. The object with alias 'default' was created in thread id 140456219838272 and this is thread id 140456135896760.

/home/mendes/.virtualenvs/data_domain/lib/python3.7/site-packages/django/db/backends/base/base.py:547: DatabaseError

[blueyed]

Can you make sure that it is related to pytest-django only, and then maybe git-bisect it?

[Mendes11]

Can you make sure that it is related to pytest-django only ... ?

When I run manage.py test using Django's default test runner all tests passed.

[Mendes11]

After multiple atempts I found that the problem is with pluggy requirement.

.pluggy >=0.12 (requirement for pytest >=4) 
.pytest >=4
.pytest-django==3.8.0

this error occurs.

pluggy==0.11
pytest>=3.6,<4
pytest-django==3.8.0

All tests have passed.

After this analysis think pytest-django should have a requirement for pytest>=3.6,<4. Or perhaps someone that knows how to fix it (if it has something to do with this repo) posting a PR...

Hope this was helpfull

[blueyed]

After multiple atempts I found that the problem is with pluggy requirement.

You mean pytest, right? I plan to drop support for < 4.2 there (https://github.com/pytest-dev/pytest-django/pull/744).

(possibly related: https://github.com/pytest-dev/pytest-django/issues/710, https://github.com/pytest-dev/pytest-django/issues/753)

Can you provide a minimal example to trigger this?

[Mendes11]

You mean pytest, right?

No.

If I keep pytest version<4 and pluggy==0.11 all tests pass If i update pluggy==0.12 and keep pytest<4 all tests fail.

Can you provide a minimal example to trigger this? Sure.

With this model:

class DataInput(models.Model):
    INPUT_CHOICES = (('continuous', 'Continuous'), ('categoric', 'Categoric'))

    input_name = models.SlugField()
    verbose_name = models.CharField(max_length=200)
    unit = models.CharField(max_length=100, null=True, blank=True)
    input_type = models.CharField(max_length=100, null=True, blank=True)
    owner = models.UUIDField()

And this test:

class TestDataInputAPI(APITestCase): # I'm using DRF.
    def setUp(self) -> None:
        self.c = APIClient()
        self.owner1 = uuid.uuid4()
        self.input1 = DataInput.objects.create(
            input_name='input_1', verbose_name='Input 1',
            input_type='continuous', owner=self.owner1
        )

    def test_db_failure(self):
        self.assertEqual(1, 1)

It fails with pluggy==0.12 and works with pluggy==0.11

[oesgalha]

I had the same problem and the same workaround worked for me.

I was trying to test a Django 2.2.x app with pytest >= 4 (tried with 6.0.2) and pytest-django 3.10. Downgrading pyttest to < 4 and pinning pluggy==0.11 resolved the issue.

---

But upgrading the app to Django 3.1 also resolved the issue, which is a better option than downgrading pytest.

[atleta]

The above did not solve the issue for me. I now have:

pytest==3.10.1
pytest-django==3.8.0
pytest-pythonpath==0.7.3
pluggy==0.11.0

and it still produces the above error. Upgrading Django may help, though it's pretty disappointing in general that a lot of packages seem not to set an upper bound on their dependencies, and thus their older versions simply break and become unusable. (A very similar bug happened to ipython a few weeks ago.)

In this case it's even more disappointing since it means that this package does not support the current LTS version of django, which will be maintained for another year (and which happens to be the only LTS at the moment). Not that I'm that much keen on using an LTS version, but some people definitely are - LTS exists for a reason.

[atleta]

... but what seems to work, as a temporary solution is providing module or even package names on the command line. Here is how it works for me... I have the following configuration in my pytest.ini:

python_files = *.py
testpaths = tests/apps/ tests/e2e/ tests/libs/
django_find_project = false
python_paths = .

(The last two line may or may not be influencing this behaviour.) Now if I run pytest test then I get the error above. However, if I do pytest tests/apps/ tests/somemodule.py then it works. (And this is equivalent, in my case of running all tests. Note, that test/apps has several subdirectories and is a pyton package having an __init__.py file.)

This works with the following requirements without any problem:

Django==2.2
pytest ==6.2.1
pytest-django = "==4.1.0"
pytest-pythonpath = "==0.7.3"
pluggy==0.13.1

(I didn't pin pluggy, it's what gets installed with the above requirements as of now. But I've checked and it works with 0.12 too.)