Uprading to pytest-xdist v2.x breaks parallelism

[skoot]

Now when we run our tests with -n4, we get errors that the database already exists.

[D3X]

This was fixed by #861. That fix is included in version 3.10. There are some issues with pypi release, though: https://github.com/pytest-dev/pytest-django/issues/863#issuecomment-680099820

[bluetech]

Closing per @D3X's comment; the version is released now.

[jgb]

@D3X are you sure the new version fixes this bug? I've got pytest-xdist 2.1.0 and pytest-django 4.0.0 and my tests still fail when running with -n4 with these type of errors:

MySQLdb._exceptions.OperationalError: (1050, "Table 'django_content_type' already exists")

[jgb]

@bluetech this bug isn't solved AFAIK. Using the latest pytest-xdist and pytest-django, I still fail to run pytest with -n4. Downgrading pytest-xdist fixes the problem.

[bluetech]

OK thanks @jgb, reopening.

[ricky-sb]

@jgb what version of pytest-xdist is working for you?

[jgb]

@jgb what version of pytest-xdist is working for you?

@ricky-sb Hello, the last version that works without problems for me is pytest-xdist 1.34.0 Anything higher and I get the error mentioned above in the comment from 20th of october.

[bluetech]

I tried to reproduce this but couldn't. This is with

platform linux -- Python 3.9.1, pytest-6.2.1, py-1.10.0, pluggy-0.13.1
django: settings: pytest_django_test.settings_mysql_innodb (from env)
rootdir: /home/ran/src/pytest-django, configfile: setup.cfg
plugins: xdist-2.2.0, forked-1.3.0, django-4.1.1.dev8+g5cc98c1

and 10 workers.

Please provide more details:

• Django version
• pytest version
• pytest-django version
• pytest-xdist version
• Which DB you're using
• Are you using --reuse-db?

And a standalone way to reproduce would be best.

[jgb]

Hello @bluetech,

what we are using:

• python 3.6.9
• ubuntu LTS 18.04
• django 2.2.17
• pytest 6.2.1
• pytest-django 4.1.0
• pytest-xdist 1.34.0
• mariadb 10.1.48

Yes we are using --reuse-db. With this configuration, as soon as I upgrade to any version of pytest-xdist higher than the one I mentioned, the failures like I commented on 20th of october pop up. Downgrading back to 1.34.0 solves it.

[shacker]

We have a very db-intensive test suite (needs to be for this app). We use Postgres, pytest, pytest-django, and pytest-xdist on MacOS. Previously, we could run our test suite with -n 8 and the parallelism would work just fine; 8 databases were created and destroyed. But lately we've been seeing a lot of errors like the one described here. Specifically:

                self.log('Got an error creating the test database: %s' % e)
>               sys.exit(2)
E               SystemExit: 2

I've done some testing and come up with the example below, in case it's helpful debugging this.

Decorating any test with @pytest.mark.django_db invokes db creation, even if no data is created or touched in the test body. Note that this sample test does not use any code from our project -- it's "bare":

# sometest.py

import pytest

@pytest.mark.django_db
def test_a():
    assert 1 == 1

@pytest.mark.django_db
def test_b():
    assert 1 == 1

• If I run pytest sometest.py it works just fine.
• If I run pytest -n 2 sometest.py it fails with a database creation error similar to the one above.
• If I comment out one of the two decorators and again run pytest -n 2 sometest.py it's fine again.

In other words, the problem only comes up when xdist is invoked AND the pytest collector finds more than one test that touches the database.

Current testrunner libs are:

plugins: testinfra-1.16.0, sugar-0.9.4, html-3.1.1, cov-2.11.1, metadata-1.11.0, xdist-2.2.1, faker-2.0.0, Faker-4.1.2, django-4.1.0, flakefinder-1.0.0, forked-1.3.0

Going on other suggestions, I've tried downgrading python-xdist to 1.34.0 but it did not fix the problem.

It is not clear to me whether the problem is in pytest-django or in python-xdist. I'm going to cross-post this to issues on both projects.

pytest-xdist Issue here

[danmash]

In my case, it was caused by the wrong config for databases:

DATABASES = {
    'default': {
        'NAME': 'postgres',
    },
    'caching_database': {
        'NAME': 'postgres',
    },
    'user_db': {
        'NAME': 'postgres',
    }
}

So pytest-xdist tried to create a new database using the database name test_postgres_gw0 three times and as a result, failed with SystemExit: 2