harvest uses the production database when --keepdb is used

[electroniceagle]

If you use --keepdb, aloe will use your production database 'foo' rather than 'test_foo' in some circumstances. I think I've hit this bug in django-nose using --keepdb: https://github.com/django-nose/django-nose/issues/76

I only noticed this because postgresql hit a constraint error on a table that does not currently exist in Django (we removed the app, but forgot to clean the tables).

(Pdb) down
> /Users/bschott/.virtualenvs/sites/lib/python2.7/site-packages/django/db/backends/base/base.py(142)_commit()
-> return self.connection.commit()
(Pdb) p self.connection
<connection object at 0x7ff6e7bda700; dsn: 'dbname=test_site_awesim user=bschott port=5432', closed: 0>
(Pdb)
======================================================================
ERROR: Failure: IntegrityError (update or delete on table "filer_image" violates foreign key constraint "django_main_image_id_55d81830a31519a_fk_filer_image_file_ptr_id" on table "djangocms_blog_post"
DETAIL:  Key (file_ptr_id)=(19) is still referenced from table "djangocms_blog_post".
)

Despite the connection claiming it is connected to the 'test_site_awesim' database, the constraint error is for a table that only exists in my local copy of our production database.

test_site_awesim=# select * from djangocms_blog_post;
ERROR:  relation "djangocms_blog_post" does not exist
LINE 1: select * from djangocms_blog_post;
                      ^

site_awesim=# select * from djangocms_blog_post;
 id |         date_created          |         date_modified         |     date_published     | date_published_end | publish | enable_comments | author_id | content_id | main_image_id | main_image_full_id | main_image_thumbnail_id | app_config_id
----+-------------------------------+-------------------------------+------------------------+--------------------+---------+-----------------+-----------+------------+---------------+--------------------+-------------------------+---------------
  1 | 2016-01-05 10:42:15.958389-05 | 2016-01-05 10:52:42.823421-05 | 2016-01-05 10:32:39-05 |                    | t       | t               |        75 |        277 |            19 |                    |                         |             1
(1 row)

Here is our current config.

Django==1.8.12
django-nose==1.4.3
nose==1.3.7
psycopg2==2.6.1

Our workaround is to manually prepend 'test' to the NAME in the DATABASES setting.

DATABASE_NAME = os.environ.get('DATABASE_NAME', PROJECT_NAME)
if 'harvest' in sys.argv and '--keepdb' in sys.argv:
    DATABASE_NAME = 'test_' + DATABASE_NAME
DATABASES = {
    'default': {
        'ENGINE': os.environ.get(
        'DATABASE_ENGINE', 'django.db.backends.postgresql_psycopg2'),
        'NAME': DATABASE_NAME,
        ...

[electroniceagle]

Another point, the code that is hitting this in a before.all hook. This code appears to be executing against the production database:

@before.all
def _cleanup_user_models():
    """
    Clean up possibly modified user tables in case
    a user exits uncleanly.  Control-c from pdb causes
    data to be left in tables, for example.
    """

    Session.objects.all().delete()
    EmailAddress.objects.all().delete()
    UserProfile.objects.all().delete()
    User.objects.all().delete()

[electroniceagle]

And the bug only appears when executing a specific feature file. Here is the derived command from my makefile:

./manage.py harvest --pdb --attr=\!not_site_awesim --keepdb --color -v 3 --logging-level=INFO --all-modules site_common/features/sessions.feature

[koterpillar]

I did not verify this but won't be surprised if this is inherited from django-nose. I am currently investigating if it would be feasible to switch away the whole project from Nose to either Py.test, Nose 2 or plain unittest.

[electroniceagle]

Py.test seems to have a significant following, but I don't have much experience with nose or Py.test, so it is hard for me to compare.

I was reading over py.test and trolling the issues and came across this: https://github.com/pytest-dev/pytest-django/pull/182

Our bug might be triggered by django-cms opening the database very early. The proposed patches in both testing tools haven't been picked up probably because this is such a corner case.

[koterpillar]

This looks similar to aloetesting/aloe#118. Are you running a single feature or letting Aloe/Nose discover them?

[koterpillar]

@electroniceagle can you please try pure-unittest branch of both Aloe and Aloe-Django (https://github.com/aloetesting/aloe/pull/117, #50) and see if the problem still exists?