Unit Tests + Django REST Framework = schema "test" does not exist

[bartmika]

I am receiving this error when running the unit tests. This code was confirmed working in django-tenants but now that I ported to `django-tenant-schemas`` this code errors.

It is interesting to note that the application actually works when I run and play with it. But when running unit tests I get the following:

======================================================================
ERROR: tearDownClass (tenant_reward.tests.TenantRewardTestCases)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/johnappleseed/Developer/smegurus/django-smegurus/env/lib/python3.5/site-packages/django/db/backends/utils.py", line 62, in execute
    return self.cursor.execute(sql)
psycopg2.ProgrammingError: schema "test" does not exist

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/johnappleseed/Developer/smegurus/django-smegurus/env/lib/python3.5/site-packages/tenant_schemas/test/cases.py", line 25, in tearDownClass
    cursor.execute('DROP SCHEMA test CASCADE')
  File "/Users/johnappleseed/Developer/smegurus/django-smegurus/env/lib/python3.5/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/Users/johnappleseed/Developer/smegurus/django-smegurus/env/lib/python3.5/site-packages/django/db/utils.py", line 94, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/Users/johnappleseed/Developer/smegurus/django-smegurus/env/lib/python3.5/site-packages/django/utils/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/Users/johnappleseed/Developer/smegurus/django-smegurus/env/lib/python3.5/site-packages/django/db/backends/utils.py", line 62, in execute
    return self.cursor.execute(sql)
django.db.utils.ProgrammingError: schema "test" does not exist

The code I use is as follows:

from django.db import transaction
from django.contrib.auth.models import User, Group
from django.utils import translation
from django.core.urlresolvers import resolve, reverse
from rest_framework import status
from rest_framework.authtoken.models import Token
from rest_framework.test import APITestCase
from tenant_schemas.test.cases import TenantTestCase
from tenant_schemas.test.client import TenantClient
from smegurus import constants
from foundation_tenant.models.me import TenantMe
from foundation_tenant.models.postaladdress import PostalAddress
from foundation_tenant.models.contactpoint import ContactPoint

TEST_USER_EMAIL = "xxx@xxx.com"
TEST_USER_USERNAME = "X"
TEST_USER_PASSWORD = "Y"
TEST_USER_FIRST_NAME = "A"
TEST_USER_LAST_NAME = "B"

class TenantRewardTestCases(APITestCase, TenantTestCase):
    fixtures = []

    @classmethod
    def setUpTestData(cls):
        user = User.objects.create_user(  # Create our User.
            email=TEST_USER_EMAIL,
            username=TEST_USER_USERNAME,
            password=TEST_USER_PASSWORD
        )
        user.is_active = True
        user.save()

    @transaction.atomic
    def setUp(self):
        translation.activate('en')  # Set English
        super(TenantRewardTestCases, self).setUp()
        # Initialize our test data.
        self.user = User.objects.get(username=TEST_USER_USERNAME)
        token = Token.objects.get(user=self.user)

        # Setup.
        self.unauthorized_client = TenantClient(self.tenant)
        self.authorized_client = TenantClient(self.tenant, HTTP_AUTHORIZATION='Token ' + token.key)
        self.authorized_client.login(
            username=TEST_USER_USERNAME,
            password=TEST_USER_PASSWORD
        )

        # Update Organization.
        self.tenant.users.add(self.user)
        self.tenant.save()

        # Setup User.
        TenantMe.objects.create(
            owner=self.user,
        )

    @transaction.atomic
    def tearDown(self):
        PostalAddress.objects.delete_all()
        ContactPoint.objects.delete_all()
        TenantMe.objects.delete_all()
        users = User.objects.all()
        for user in users.all():
            user.delete()
        # super(TenantRewardTestCases, self).tearDown()

    @transaction.atomic
    def test_reward_page(self):
        url = reverse('tenant_reward')
        response = self.authorized_client.get(url)
        self.assertEqual(response.status_code, status.HTTP_200_OK)
        self.assertTrue(len(response.content) > 1)
        self.assertIn(b'Rewards',response.content)

And I am using the following versions:

Django==1.10.2
djangorestframework==3.4.7
django-tenant-schemas==1.6.4

Any help in this issue would be much appreciated!

[oplahcinski]

I'm on Django 1.8, same DRF and DTS having a similar issue.

django.db.utils.OperationalError: no such table: bar_a

From what i've found so far, Try this hack to see if it solves something for you too

In your tests settings file put

import tenant_schemas
DATABASE_ROUTERS = []
DATABASES = {
    "default": {
        "ENGINE": "custom.testing.tenants.sqlite",
        "TEST": {
            "NAME": ":memory:",
        }
    },
}

I found DATABASE_ROUTERS to be causing me some problems that im still trying to narrow down. Doing this lets my tests run a little further.

[ericpalakovichcarr]

I'm also getting a similar issue. It appears the TenantTestCase isn't creating the test schema during setUpClass. I'm applying django-tenant-schemas to an existing project, and outside of running the existing tests, everything seems to be running well (i.e. using the site from local runserver webserver).

Here's how I updated my tests:

from django import test

from canvas.factories import (
    WalkSheetFactory,
)

class UnicodeTests(test.TestCase):
    def test_WalkSheet(self):
        '%s' % WalkSheetFactory()

Became:

from tenant_schemas.test.cases import TenantTestCase as TestCase

from canvas.factories import (
    WalkSheetFactory,
)

class UnicodeTests(TestCase):
    def test_WalkSheet(self):
        '%s' % WalkSheetFactory()

The test above is creating a WalkSheet record in the database and then ensuring the WalkSheet model's __str__ method doesn't crash. I'm pretty the changes I made follow what's in the documentation here. I've confirmed setupClass is getting called, which in turn calls migrate_schemas before the test runs.

But the test fails with this exception:

Traceback (most recent call last):
  File "/home/vagrant/venv_victoryguide/lib/python3.5/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
psycopg2.ProgrammingError: relation "canvas_walksheet" does not exist
LINE 1: INSERT INTO "canvas_walksheet" ("id", "created_at", "modifie...

And then the tearDownClass method from TenantTestCase fails with the following:

Traceback (most recent call last):
  File "/home/vagrant/venv_victoryguide/lib/python3.5/site-packages/tenant_schemas/test/cases.py", line 25, in tearDownClass
    cursor.execute('DROP SCHEMA test CASCADE')
  File "/home/vagrant/venv_victoryguide/lib/python3.5/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/home/vagrant/venv_victoryguide/lib/python3.5/site-packages/django/db/utils.py", line 95, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/home/vagrant/venv_victoryguide/lib/python3.5/site-packages/django/utils/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/home/vagrant/venv_victoryguide/lib/python3.5/site-packages/django/db/backends/utils.py", line 62, in execute
    return self.cursor.execute(sql)
django.db.utils.ProgrammingError: schema "test" does not exist

Have I missed something obvious to TenantTestCase to work with my existing test suite? FWIW:

Python 3.5.1 Django==1.9.6 django-tenant-schemas==1.6.10 djangorestframework==3.3.3 # though I don't think DRF has anything to do with it

[alexandervaneck]

Hi @bigsassy,

What is the structure of your SHARED_APPS and TENANT_APPS?

Does your canvas app have to be in the public schema? or is it in the tenant schema? TenantTestCase automatically sets your connection to the newly created tenant, public schema apps can use the normal django TestCase base I think.

If the canvas app is in the SHARED_APPS and therefore in the public schema this is what would cause the exception. It can't be found in the tenant schema...

The tearDownClass failing seems weird to me, I don't understand why it's simply called 'test'. Normally it's '[yourdb]_test'.

[ericpalakovichcarr]

Hey @AlexvEck,

Here's the structure of my SHARED_APPS and TENANT_APPS

SHARED_APPS = (
    'tenant_schemas',
    'account',  # this is the app with the TENANT_MODEL
    'django.contrib.contenttypes',
)
TENANT_APPS = (
    'canvas',
    # ... other apps
)

The canvas app is only applicable to individual tenants, so I think I have it in the right place. Should it be somewhere else?

[alexandervaneck]

@bigsassy, that seems properly configured.

Does the WalkSheetFactory or model do anything fancy? and where does that exception originate from? I take it there's some kind of statement before it it tries to execute?

[ericpalakovichcarr]

Nothing too fancy. Here's the WalkSheet model:

class WalkSheet(models.Model):
    """A list of voters to visit on a door knocking run."""
    id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False)
    created_at = models.DateTimeField(editable=False, blank=True, default=timezone.now)
    modified_at = models.DateTimeField(editable=False, blank=True, auto_now=True)

    name = models.TextField()
    households = models.ManyToManyField('map.Household', through='WalkSheetHousehold')
    candidates = models.ManyToManyField('people.Candidate')

    def __str__(self):
        return self.name

Here's a simplified version of the test.

from tenant_schemas.test.cases import TenantTestCase as TestCase
from canvas.models import WalkSheet

class UnicodeTests(TestCase):
    def test_WalkSheet(self):
        walksheet = WalkSheet.objects.create(name="Test Walk Sheet")
        '%s' % walksheet

I ran that updated version of the test and it failed the same way as before. When I move all my apps into SHARED_APPS the test itself passes, so this error goes away:

psycopg2.ProgrammingError: relation "canvas_walksheet" does not exist
LINE 1: INSERT INTO "canvas_walksheet" ("id", "created_at", "modifie...

But the tearDownClass method still fails with:

======================================================================
ERROR: tearDownClass (canvas.tests.UnicodeTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vagrant/venv_victoryguide/lib/python3.5/site-packages/django/db/backends/utils.py", line 62, in execute
    return self.cursor.execute(sql)
psycopg2.ProgrammingError: schema "test" does not exist

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/vagrant/venv_victoryguide/lib/python3.5/site-packages/tenant_schemas/test/cases.py", line 25, in tearDownClass
    cursor.execute('DROP SCHEMA test CASCADE')
  File "/home/vagrant/venv_victoryguide/lib/python3.5/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/home/vagrant/venv_victoryguide/lib/python3.5/site-packages/django/db/utils.py", line 95, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/home/vagrant/venv_victoryguide/lib/python3.5/site-packages/django/utils/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/home/vagrant/venv_victoryguide/lib/python3.5/site-packages/django/db/backends/utils.py", line 62, in execute
    return self.cursor.execute(sql)
django.db.utils.ProgrammingError: schema "test" does not exist

[alexandervaneck]

Do you have auto_drop_schema set in your TenantMixin model? canvas can be in your TENANT_APPS, that should be no problem.

I think the tearDownClass fails because the cls.tenant.delete() already deletes the schema if auto_drop_schema is set to True on your TenantMixin model. This is probably a bug ;)

However the other error you're receiving... might have to do with auto_create_schema. Is it set to False on your TenantMixin model? In that case when TenantTestCase creates the tenant it never creates/changes the schema to it because there's no new schema. and tearDownClass will also fail as a result of this.

Any thoughts?

[ericpalakovichcarr]

Nope. It looks like this:

import uuid

from django.db import models
from django.utils import timezone
from tenant_schemas.models import TenantMixin

class Account(TenantMixin):
    id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False)
    created_at = models.DateTimeField(editable=False, blank=True, default=timezone.now)
    modified_at = models.DateTimeField(editable=False, blank=True, auto_now=True)

    name = models.TextField()

Should I?

[ericpalakovichcarr]

Ah, right! It's when cls.tenant.save(verbosity=0) is called in setUpClass that the schema is supposed to be created. Let me put in a breakpoint there and see what's actually happening...

[alexandervaneck]

I'm most interested in a breakpoint at if is_new and self.auto_create_schema: line 47 in tenant_schemas/models.py. and then the value of is_new and self.auto_create_schema.

[ericpalakovichcarr]

https://www.youtube.com/watch?v=vhu3NTOHz-M

Right, and that's where the problem was. So, here are the bits that matter in TenantMixin:

def save(self, verbosity=1, *args, **kwargs):
    is_new = self.pk is None  # <-- here's the problem

    if is_new and self.auto_create_schema:
        try:
            self.create_schema(check_if_exists=True, verbosity=verbosity)
        except:
            # We failed creating the tenant, delete what we created and
            # re-raise the exception
            self.delete(force_drop=True)
            raise
        else:
            post_schema_sync.send(sender=TenantMixin, tenant=self)

Here's a reminder of my WalkSheet model:

class WalkSheet(models.Model):
    """A list of voters to visit on a door knocking run."""
    id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False)
    created_at = models.DateTimeField(editable=False, blank=True, default=timezone.now)
    modified_at = models.DateTimeField(editable=False, blank=True, auto_now=True)

    name = models.TextField()
    households = models.ManyToManyField('map.Household', through='WalkSheetHousehold')
    candidates = models.ManyToManyField('people.Candidate')

    def __str__(self):
        return self.name

My model uses a UUID for the primary key, and since it's not using a SERIAL field in the database it sets the key in Django before executing the SQL to create the record. So when is_new = self.pk is None is False, because the PK was already set because of default=uuid.uuid4 in the id field.

Ok, so that makes sense. Hmmm...that's tricky. Any ideas on how to work around this? I'd be happy to try and fix this and submit a pull request.

[alexandervaneck]

Nice! Glad we found it. :)

I think the workaround would be to not use a UUIDfield for the PK of the Account model, or use one at all. Why are you using it? and if it's just for reference then why does it have to be the PK?

[ericpalakovichcarr]

Nah, I think I'll just make a subclass of TenantMixin and update the save method to handle my edge case. I think that should solve the issue for me.

Thanks for the help :)

[alexandervaneck]

No problem!

[bernardopires]

See issue #364