Crash when there is no relational database at all

[Karmak23]

Hi,

on our jenkins server, there is no "real" django database at all: we just need the test one created on the fly by ./manage.py test or ./manage.py jenkins. Still, the latest version of mongoadmin crashes:

Traceback (most recent call last):
  File "/var/lib/jenkins/jobs/1flow/workspace/venv/local/lib/python2.7/site-packages/django/core/management/base.py", line 222, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/var/lib/jenkins/jobs/1flow/workspace/venv/local/lib/python2.7/site-packages/django/core/management/base.py", line 250, in execute
    translation.activate('en-us')
  File "/var/lib/jenkins/jobs/1flow/workspace/venv/local/lib/python2.7/site-packages/django/utils/translation/__init__.py", line 90, in activate
    return _trans.activate(language)
  File "/var/lib/jenkins/jobs/1flow/workspace/venv/local/lib/python2.7/site-packages/django/utils/translation/trans_real.py", line 183, in activate
    _active.value = translation(language)
  File "/var/lib/jenkins/jobs/1flow/workspace/venv/local/lib/python2.7/site-packages/django/utils/translation/trans_real.py", line 172, in translation
    default_translation = _fetch(settings.LANGUAGE_CODE)
  File "/var/lib/jenkins/jobs/1flow/workspace/venv/local/lib/python2.7/site-packages/django/utils/translation/trans_real.py", line 154, in _fetch
    app = import_module(appname)
  File "/var/lib/jenkins/jobs/1flow/workspace/venv/local/lib/python2.7/site-packages/django/utils/importlib.py", line 35, in import_module
    __import__(name)
  File "/var/lib/jenkins/jobs/1flow/workspace/venv/src/mongoadmin/mongoadmin/__init__.py", line 1, in <module>
    from .options import *
  File "/var/lib/jenkins/jobs/1flow/workspace/venv/src/mongoadmin/mongoadmin/options.py", line 6, in <module>
    from mongoadmin.contenttypes.models import ContentType
  File "/var/lib/jenkins/jobs/1flow/workspace/venv/src/mongoadmin/mongoadmin/contenttypes/models.py", line 1, in <module>
    from .utils import HAS_RELATIONAL_DB, get_model_or_document
  File "/var/lib/jenkins/jobs/1flow/workspace/venv/src/mongoadmin/mongoadmin/contenttypes/utils.py", line 14, in <module>
    ContentType.objects.all()[0]
  File "/var/lib/jenkins/jobs/1flow/workspace/venv/local/lib/python2.7/site-packages/django/db/models/query.py", line 215, in __getitem__
    return list(qs)[0]
  File "/var/lib/jenkins/jobs/1flow/workspace/venv/local/lib/python2.7/site-packages/django/db/models/query.py", line 92, in __len__
    self._result_cache.extend(self._iter)
  File "/var/lib/jenkins/jobs/1flow/workspace/venv/local/lib/python2.7/site-packages/django/db/models/query.py", line 301, in iterator
    for row in compiler.results_iter():
  File "/var/lib/jenkins/jobs/1flow/workspace/venv/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 775, in results_iter
    for rows in self.execute_sql(MULTI):
  File "/var/lib/jenkins/jobs/1flow/workspace/venv/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 840, in execute_sql
    cursor.execute(sql, params)
  File "/var/lib/jenkins/jobs/1flow/workspace/venv/local/lib/python2.7/site-packages/django/db/backends/util.py", line 41, in execute
    return self.cursor.execute(sql, params)
  File "/var/lib/jenkins/jobs/1flow/workspace/venv/local/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/base.py", line 58, in execute
    six.reraise(utils.DatabaseError, utils.DatabaseError(*tuple(e.args)), sys.exc_info()[2])
  File "/var/lib/jenkins/jobs/1flow/workspace/venv/local/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/base.py", line 54, in execute
    return self.cursor.execute(query, args)
DatabaseError: relation "django_content_type" does not exist
LINE 1: ..."."app_label", "django_content_type"."model" FROM "django_co...

If we git reset --hard HEAD~2 the mongoadmin/ repository, everything is fine again.

So the problem lies in one of your latest 2 patches. I didn't have the time to investigate the "why", but at a glance ContentType.objects.all()[0] seems to be a little too much assumption that a relational DB exists with Django's own tables (we don't even run syncdb on the jenkins machine because it will be run for the test database anyway).

regards,

Olivier

[jschrewe]

Well, the problem here is that mongoadmin runs fine without any relational db as long as it is set to

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.dummy'
    }
}

The only problem is that the admin tends to use contenttypes all over the place. So in order to make the admin run without any kind of relational db mongoadmin tries to figure out if there is a relational db and if there isn't then uses the contenttypes provided by mongoengine. I think the auth system needs them at least.

I do very much agree though that the current test for a relational db sucks and I'd love to have a better (and more robust) test. I do remember that just catching every error the database might throw makes running syncdb next to impossible. Ah, well seems I need to figure out a better test now.

If you can tell jenkins to just catch and ignore it you should be fine though (maybe set HAS_RELATIONAL_DB to False too).

[Karmak23]

Correct me if I'm wrong, but I feel like setting the default database engine to dummy will imply the same for the test database ? It is obviously not what we want, but I will investigate this on my side.

I tend to think the same about HAS_RELATIONAL_DB In fact, we have a relational db, but only for the tests.

[jschrewe]

I have no idea but I suppose you'll get the same for tests. And, well, I can't tell you what you want of course, only the state the code is in. Which isn't all that great apparently.

And the only reason HAS_RELATIONAL_DB is there is because it didn't really work for me without. I'm a bit amazed that you got it working without any relational db.

But I take a look at it tomorrow and hopefully come up with a better way to test this.

[jschrewe]

Give d5633d2cc5c194be48cf4ceb6b95ad0e071de5b3 a try. If you add MONGOADMIN_CHECK_CONTENTTYPE = False to settings.py it should not try to figure out wether a relational db is there or not. I'd actually like to make the real test more robust, but failed miserably. So I hope this works for you too.

[Karmak23]

Latest commit works, regarding this issue, on my already setup project.

The subsequent question is: is there a way to simply delay this test, like with a *_lazy() function ? Because the crash still arise when I start a Django project from scratch.

The main problem beiing that even with a relational database and everything correctly configured, ./manage createdb will fail because the ContentType model has not created the SQL table yet. This happens a lot in our environment, either in Jenkins which always start from scratch to be sure the project is completely installable at any time, or when (re-)deploying a full preview environment to test new features.

I understand that I can either temporarily remove mongoadmin from INSTALLED_APPS or define MONGOADMIN_CHECK_CONTENTTYPE = False until createdb is done, but this is still awkward: you would have to document somewhere that any project starting from scratch with mongoadmin included will fail to start, and provide these 2 possible workarounds.

[jschrewe]

Aww, I thought createdb would run without issue and the problem was that Jenkins just didn't create a database. I do wonder if there is a pattern I'm missing that runs tests like this only once but later.

Thinking a bit more about this I think looking at the database setting and assuming that the ContentType stuff only needs replacement if the DB is explicitly set to dummy is the way to go. This is how it works at the moment anyway. Only uglier...

[Karmak23]

I close this issue because I didn't have time to test further this particular config. I will reopen it if I hit the bug again.