Using the same database with `xdist` and `--create-db` results in error.

[cansin]

We are trying to avoid creating databases for each xdist process since our migrations are taking a very long time (They have network calls. I know, I know). The pytest-django documentation at https://pytest-django.readthedocs.io/en/latest/database.html?highlight=xdist#use-the-same-database-for-all-xdist-processes suggests it would be possible to use xdist and let processes rely on the same database.

But in reality, there is a race condition when the same database is tried to be used, and it causes a bunch of "duplicate * value" errors.

Is there a recommended way around this problem? Thanks!

[cansin]

Here is a sample error:

pytest.ini

[pytest]
addopts =
  --cov=hyke
  --cov-report=html
  --cov-report=term
  --cov-report=xml
  --failed-first
  --no-cov-on-fail
  --numprocesses=auto
  --reuse-db

bash

pipenv run pytest --create-db
============================= test session starts ==============================
platform linux -- Python 3.9.10, pytest-7.1.1, pluggy-1.0.0
django: settings: **omitted** (from option)
rootdir: **omitted**, configfile: pytest.ini
plugins: cov-3.0.0, Faker-13.3.4, forked-1.4.0, xdist-2.5.0, django-4.5.2, ddtrace-0.60.2
gw0 I / gw1 I
gw0 [1614] / gw1 [1614]

_______ ERROR at setup of **omitted** ________
[gw1] linux -- Python 3.9.10 **omitted**/bin/python

self = <django.db.backends.utils.CursorWrapper object at 0x7f4a84c39550>
sql = 'CREATE DATABASE "test_postgres" ', params = None
ignored_wrapper_args = (False, {'connection': <django.db.backends.postgresql.base.DatabaseWrapper object at 0x7f4a84c3[90](https://github.com/**omitted**/runs/6013133846?check_suite_focus=true#step:5:90)70>, 'cursor': <django.db.backends.utils.CursorWrapper object at 0x7f4a84c3[95](https://github.com/**omitted**/runs/6013133846?check_suite_focus=true#step:5:95)50>})

    def _execute(self, sql, params, *ignored_wrapper_args):
        self.db.validate_no_broken_transaction()
        with self.db.wrap_database_errors:
            if params is None:
                # params default might be backend specific.
>               return self.cursor.execute(sql)
E               psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "pg_database_datname_index"
E               DETAIL:  Key (datname)=(test_postgres) already exists.

[cansin]

I think pytest-django needs a logic that would make sure the database is created for the first connection, and the other processes just shared a connection to that DB, similar to what is being done at Django's DiscoverRunner: https://github.com/django/django/blob/main/django/test/utils.py#L196

[cansin]

And that logic needs to be implemented at https://github.com/pytest-dev/pytest-django/blob/master/pytest_django/fixtures.py#L101

[cansin]

To be clear, I am referring to the code section within if parallel > 1:. Just a theory though.

[cansin]

Actually no. I guess you are already doing it manually in your code at django_db_modify_db_settings_xdist_suffix. Hmm 😕

[cansin]

I guess the default Django-test runner's parallel option does not even allow re-using the same DB on different processes.

[cansin]

I wonder what @blueyed and @adamantike think.

[cansin]

I am guessing we need some kind of a mutex so that one of the runners calls setup_databases while others wait for it. I am not sure if that is possible through what pydest-xdist provides though.

[cansin]

Through trial and error, I have realized an underlying problem with our migrations having network calls. It somehow was causing the test run to start running before applied migrations. I think something was erroring out and breaking a wait loop.