register_hstore not getting called properly during tests

ddohler commented 9 years ago

We have an application with a test that looks roughly like this:

from django.test import TestCase
from things.models import Thing

class ThingCreationTestCase(TestCase):
    """Ensure hstore fields work."""
    def test_create_thing(self):
        thing1 = Thing(name='new-thing', data={'testkey': 'testvalue'})
        thing1.save()
        thing2 = Thing.objects.all()[0]
        self.assertEqual(thing2.data['testkey'], thing1.data['testkey'])

After updating to django_hstore >= 1.2.1, this test fails with the following error:

======================================================================
ERROR: Ensure hstore fields work
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/projects/myproject/python/django/things/tests.py", line 95, in test_create_thing
    self.assertEqual(thing2.data['testkey'], thing1.data['testkey'])
TypeError: string indices must be integers, not str

The application itself runs fine, however, and hstore fields on models work properly--only the tests fail.

I was able to trace the error to this set of commits: https://github.com/djangonauts/django-hstore/compare/8ac21be921905d0e2d87e865148e101f8cc8e731...2e9164f3b23f5f6ead4893aa2312dd360a72fd01

In other words, this test works with versions of django_hstore prior to Jan. 12, 2014, while it fails with versions after Jan. 12, 2014.

I was able to resolve the issue by placing the following at the top of the test:

from django.test import TestCase
from things.models import Thing

class ThingCreationTestCase(TestCase):
    """Ensure hstore fields work."""
    def test_create_thing(self):
        from django.db import connection
        from psycopg2.extras import register_hstore
        register_hstore(connection.connection)
        thing1 = Thing(name='new-thing', data={'testkey': 'testvalue'})
        thing1.save()
        thing2 = Thing.objects.all()[0]
        self.assertEqual(thing2.data['testkey'], thing1.data['testkey'])

It looks like django_hstore isn't calling register_hstore properly at the start of each test case; I suspect that this is due to the way the connection_created signal is being handled in django_hstore 1.2.1 and after, but I wasn't able to figure out exactly why register_hstore isn't being called.

We are using the django.contrib.gis.db.backends.postgis database engine. We store and deal with spatial data, so we can't switch to django.db.backends.postgresql_psycopg2 as suggested in #79. When running tests under GeoDjango, the database is created from template_postgis (link), and we have the hstore extension installed on that database:

vagrant@vagrant-ubuntu-trusty-64:~$ sudo -u postgres psql template_postgis
psql (9.4.0, server 9.3.5)
Type "help" for help.

template_postgis=# CREATE EXTENSION hstore;
ERROR:  extension "hstore" already exists
template_postgis=#

nemesifier commented 9 years ago

failing test case please

alukach commented 9 years ago

We're running into a similar problem. When switching from the django.db.backends.postgresql_psycopg2 backend to django.contrib.gis.db.backends.postgis, all of our tests relying on Django Hstore suddenly start failing. On further investigation, we found that the hstore.DictionaryField was returning a string rather than a dictionary (in issue https://github.com/djangonauts/django-hstore/issues/79, notice the last test assertion failure is doing the exact same thing: AssertionError: '"v"=>"1", "v2"=>"3"' is not an instance of ...). It appears that this is what @ddohler is experiencing (TypeError: string indices must be integers, not str because thing2.data is a str rather than a dict).

This leads me to believe that there are some issues surrounding using django-hstore with PostGIS. @nemesisdesign could you comment on what version of PostGIS is being used in the Travis tests?

We are using: posgresql 9.3.4 postgis 2.1.2 r12389 ("POSTGIS="2.1.2 r12389" GEOS="3.4.2-CAPI-1.8.2 r3921" PROJ="Rel. 4.8.0, 6 March 2012" GDAL="GDAL 1.10.1, released 2013/08/26" LIBXML="2.9.1" LIBJSON="UNKNOWN" RASTER") django 1.7.4 django-hstore 1.2.4

@ddohler Can you please post your versions?

nemesifier commented 9 years ago

According to http://docs.travis-ci.com/user/using-postgresql/ and https://github.com/djangonauts/django-hstore/blob/master/.travis.yml we have postgresql 9.1 and postgis 2.1.

it's correct, this started happening after https://github.com/djangonauts/django-hstore/compare/8ac21be921905d0e2d87e865148e101f8cc8e731...2e9164f3b23f5f6ead4893aa2312dd360a72fd01 before then the "CREATE EXTENSION HSTORE;" command was being called at each connection. That was cleaned up, but it caused my tests to fail at that time too.

@alukach what happens if you run: "SELECT t.oid FROM pg_type t JOIN pg_namespace ns ON typnamespace = ns.oid WHERE typname = 'hstore' and nspname = 'public';" on your database? (the broken one I mean)

nemesifier commented 9 years ago

I think it has to do with some postgresql internals we are missing.

Have you also tried doing something like:

DROP EXTENSION hstore;
CREATE EXTENSION hstore;

alukach commented 9 years ago

Okay, I feel like I figured it out. It's a pretty strange one.

I was able to reproduce this bug with the two following conditions:

Creating a DB off of a template DB that did not have hstore enabled.
Using the django.contrib.gis.db.backends.postgis backend.

If you create a django project database from a template that does not have hstore enabled, for some reason the postgis backend will not properly use hstore on any test database that is created and instead return the data as a string ("prop1"=>"1", "prop2"=>"test_value"). ~~I had originally thought that the test database was created only from the template database at test runtime, but this does not appear to be true~~ (EDIT: Read below, this is not the issue). If you enable hstore on the template database and the django project database used, django-hstore will still fail on tests if you did not originally create the django project database from a template with the hstore extension. This bug does not occur with the django.db.backends.postgresql_psycopg2 backend.

I am still not sure why this exists but was able to reproduce the issue as shown below. Currently, my only suggestion to resolve this issue is to drop the django project database and recreate it from a template with the hstore extension enabled. Obviously, this is extremely inconvenient to do with a production database (backing up the data, renaming the db, creating a new db, loading in the data). There is probably a better solution but, at this time, I am unsure of what that is.

Example:

$ mkvirtualenv hstore
New python executable in hstore/bin/python
Installing setuptools, pip...done.
(hstore)$ git clone https://github.com/djangonauts/django-hstore.git
(hstore)$ cd django-hstore/
(hstore)$ pip install -r requirements.txt 
(hstore)$ pip install django
(hstore)$ createdb django_hstore  # template1 currently has hstore enabled
(hstore)$ python runtests.py  # Tests will pass
Creating test database for alias 'default'...
../home/alukach/.virtualenvs/hstore/local/lib/python2.7/site-packages/django/db/models/base.py:1029: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
  if f.blank and raw_value in f.empty_values:

/home/alukach/.virtualenvs/hstore/local/lib/python2.7/site-packages/django/db/models/fields/__init__.py:458: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
  if value in self.empty_values:

........................................./home/alukach/Projects/django-hstore/django_hstore/query.py:57: RemovedInDjango18Warning: 'is_managed' is deprecated.
  if not transaction.is_managed(using=self.db):

...........................................................................
----------------------------------------------------------------------
Ran 118 tests in 1.915s

OK
Destroying test database for alias 'default'...
(hstore)$ psql django_hstore -c "SELECT t.oid, typarray FROM pg_type t JOIN pg_namespace ns ON typnamespace = ns.oid WHERE typname = 'hstore';"
  oid   | typarray 
--------+----------
 907772 |   907777
(1 row)

(hstore)$ dropdb django_hstore  # Drop db
(hstore)$ psql template1 -c "DROP EXTENSION hstore;"  # Remove hstore support on template
DROP EXTENSION
(hstore)$ createdb django_hstore  # Recreate DB from template without hstore
(hstore)$ psql template1 -c "CREATE EXTENSION hstore;"  # Add hstore to template (test DBs should be based off of this, right?)
CREATE EXTENSION
(hstore)$ psql django_hstore -c "CREATE EXTENSION hstore;"  # Add hstore to created DB
CREATE EXTENSION
(hstore)$ python runtests.py  # Tests will fail
Creating test database for alias 'default'...
.E/home/alukach/.virtualenvs/hstore/local/lib/python2.7/site-packages/django/db/models/base.py:1029: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
  if f.blank and raw_value in f.empty_values:

/home/alukach/.virtualenvs/hstore/local/lib/python2.7/site-packages/django/db/models/fields/__init__.py:458: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
  if value in self.empty_values:

EE..E...........E...E.......EE.E.........FE....F........EE..........EF...FE....F...F./home/alukach/Projects/django-hstore/django_hstore/query.py:57: RemovedInDjango18Warning: 'is_managed' is deprecated.
  if not transaction.is_managed(using=self.db):

F.......EE.........EF......F.FF
======================================================================
ERROR: test_admin_add (tests.django_hstore_tests.tests.SchemaTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/alukach/Projects/django-hstore/tests/django_hstore_tests/tests.py", line 718, in test_admin_add
    self.assertEqual(d.number, 3)
  File "/home/alukach/Projects/django-hstore/django_hstore/virtual.py", line 49, in __get__
    return getattr(instance, self.hstore_field_name).get(self.name, self.default)
AttributeError: 'str' object has no attribute 'get'

======================================================================
ERROR: test_admin_add_utf8 (tests.django_hstore_tests.tests.SchemaTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/alukach/Projects/django-hstore/tests/django_hstore_tests/tests.py", line 730, in test_admin_add_utf8
    self.assertEqual(d.number, 3)
  File "/home/alukach/Projects/django-hstore/django_hstore/virtual.py", line 49, in __get__
    return getattr(instance, self.hstore_field_name).get(self.name, self.default)
AttributeError: 'str' object has no attribute 'get'

======================================================================
ERROR: test_admin_change (tests.django_hstore_tests.tests.SchemaTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/alukach/Projects/django-hstore/tests/django_hstore_tests/tests.py", line 741, in test_admin_change
    response = self.client.get(url)
  File "/home/alukach/.virtualenvs/hstore/local/lib/python2.7/site-packages/django/test/client.py", line 470, in get
    **extra)
  File "/home/alukach/.virtualenvs/hstore/local/lib/python2.7/site-packages/django/test/client.py", line 286, in get
    return self.generic('GET', path, secure=secure, **r)
  File "/home/alukach/.virtualenvs/hstore/local/lib/python2.7/site-packages/django/test/client.py", line 358, in generic
    return self.request(**r)
  File "/home/alukach/.virtualenvs/hstore/local/lib/python2.7/site-packages/django/test/client.py", line 440, in request
    six.reraise(*exc_info)
  File "/home/alukach/.virtualenvs/hstore/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 111, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/alukach/.virtualenvs/hstore/local/lib/python2.7/site-packages/django/contrib/admin/options.py", line 583, in wrapper
    return self.admin_site.admin_view(view)(*args, **kwargs)
  File "/home/alukach/.virtualenvs/hstore/local/lib/python2.7/site-packages/django/utils/decorators.py", line 105, in _wrapped_view
    response = view_func(request, *args, **kwargs)
  File "/home/alukach/.virtualenvs/hstore/local/lib/python2.7/site-packages/django/views/decorators/cache.py", line 52, in _wrapped_view_func
    response = view_func(request, *args, **kwargs)
  File "/home/alukach/.virtualenvs/hstore/local/lib/python2.7/site-packages/django/contrib/admin/sites.py", line 206, in inner
    return view(request, *args, **kwargs)
  File "/home/alukach/.virtualenvs/hstore/local/lib/python2.7/site-packages/django/contrib/admin/options.py", line 1456, in change_view
    return self.changeform_view(request, object_id, form_url, extra_context)
  File "/home/alukach/.virtualenvs/hstore/local/lib/python2.7/site-packages/django/utils/decorators.py", line 29, in _wrapper
    return bound_func(*args, **kwargs)
  File "/home/alukach/.virtualenvs/hstore/local/lib/python2.7/site-packages/django/utils/decorators.py", line 105, in _wrapped_view
    response = view_func(request, *args, **kwargs)
  File "/home/alukach/.virtualenvs/hstore/local/lib/python2.7/site-packages/django/utils/decorators.py", line 25, in bound_func
    return func.__get__(self, type(self))(*args2, **kwargs2)
  File "/home/alukach/.virtualenvs/hstore/local/lib/python2.7/site-packages/django/db/transaction.py", line 394, in inner
    return func(*args, **kwargs)
  File "/home/alukach/.virtualenvs/hstore/local/lib/python2.7/site-packages/django/contrib/admin/options.py", line 1419, in changeform_view
    form = ModelForm(instance=obj)
  File "/home/alukach/.virtualenvs/hstore/local/lib/python2.7/site-packages/django/forms/models.py", line 319, in __init__
    object_data = model_to_dict(instance, opts.fields, opts.exclude)
  File "/home/alukach/.virtualenvs/hstore/local/lib/python2.7/site-packages/django/forms/models.py", line 151, in model_to_dict
    data[f.name] = f.value_from_object(instance)
  File "/home/alukach/.virtualenvs/hstore/local/lib/python2.7/site-packages/django/db/models/fields/__init__.py", line 848, in value_from_object
    return getattr(obj, self.attname)
  File "/home/alukach/Projects/django-hstore/django_hstore/virtual.py", line 49, in __get__
    return getattr(instance, self.hstore_field_name).get(self.name, self.default)
AttributeError: 'str' object has no attribute 'get'

======================================================================
ERROR: test_create (tests.django_hstore_tests.tests.SchemaTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/alukach/Projects/django-hstore/tests/django_hstore_tests/tests.py", line 844, in test_create
    self.assertEqual(s1.number, 2)
  File "/home/alukach/Projects/django-hstore/django_hstore/virtual.py", line 49, in __get__
    return getattr(instance, self.hstore_field_name).get(self.name, self.default)
AttributeError: 'str' object has no attribute 'get'

======================================================================
ERROR: test_schemadatabag_save (tests.django_hstore_tests.tests.SchemaTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/alukach/Projects/django-hstore/tests/django_hstore_tests/tests.py", line 680, in test_schemadatabag_save
    self.assertEqual(d.number, 4)
  File "/home/alukach/Projects/django-hstore/django_hstore/virtual.py", line 49, in __get__
    return getattr(instance, self.hstore_field_name).get(self.name, self.default)
AttributeError: 'str' object has no attribute 'get'

======================================================================
ERROR: test_utf8 (tests.django_hstore_tests.tests.SchemaTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/alukach/Projects/django-hstore/tests/django_hstore_tests/tests.py", line 821, in test_utf8
    self.assertEqual(d.char, u'è')
  File "/home/alukach/Projects/django-hstore/django_hstore/virtual.py", line 49, in __get__
    return getattr(instance, self.hstore_field_name).get(self.name, self.default)
AttributeError: 'str' object has no attribute 'get'

======================================================================
ERROR: test_boolean (tests.django_hstore_tests.tests.TestDictionaryField)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/alukach/Projects/django-hstore/tests/django_hstore_tests/tests.py", line 110, in test_boolean
    self.assertEqual(json.loads(databag.data['boolean']), True)
TypeError: string indices must be integers, not str

======================================================================
ERROR: test_decimal (tests.django_hstore_tests.tests.TestDictionaryField)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/alukach/Projects/django-hstore/tests/django_hstore_tests/tests.py", line 62, in test_decimal
    self.assertEqual(databag.data['dec'], force_text(Decimal('1.01')))
TypeError: string indices must be integers, not str

======================================================================
ERROR: test_dictionary (tests.django_hstore_tests.tests.TestDictionaryField)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/alukach/Projects/django-hstore/tests/django_hstore_tests/tests.py", line 101, in test_dictionary
    self.assertEqual(json.loads(databag.data['dict']), {'subkey': 'subvalue'})
TypeError: string indices must be integers, not str

======================================================================
ERROR: test_hslice (tests.django_hstore_tests.tests.TestDictionaryField)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/alukach/Projects/django-hstore/tests/django_hstore_tests/tests.py", line 405, in test_hslice
    self.assertEqual(DataBag.objects.hslice(id=alpha.id, attr='data', keys=['v']), {'v': '1'})
  File "/home/alukach/Projects/django-hstore/django_hstore/managers.py", line 33, in hslice
    return self.filter(**params).hslice(attr, keys)
  File "/home/alukach/Projects/django-hstore/django_hstore/query.py", line 46, in selector
    return method(self, query, *args, **params)
  File "/home/alukach/Projects/django-hstore/django_hstore/query.py", line 294, in hslice
    return dict((key, field._value_to_python(value)) for key, value in result[0].items())
AttributeError: 'str' object has no attribute 'items'

======================================================================
ERROR: test_list (tests.django_hstore_tests.tests.TestDictionaryField)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/alukach/Projects/django-hstore/tests/django_hstore_tests/tests.py", line 96, in test_list
    self.assertEqual(json.loads(databag.data['list']), ['a', 'b', 'c'])
TypeError: string indices must be integers, not str

======================================================================
ERROR: test_long (tests.django_hstore_tests.tests.TestDictionaryField)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/alukach/Projects/django-hstore/tests/django_hstore_tests/tests.py", line 76, in test_long
    self.assertEqual(databag.data['long'], force_text(l))
TypeError: string indices must be integers, not str

======================================================================
ERROR: test_number (tests.django_hstore_tests.tests.TestDictionaryField)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/alukach/Projects/django-hstore/tests/django_hstore_tests/tests.py", line 88, in test_number
    self.assertEqual(databag.data['num'], '1')
TypeError: string indices must be integers, not str

======================================================================
ERROR: test_serialization_deserialization (tests.django_hstore_tests.tests.TestDictionaryField)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/alukach/Projects/django-hstore/tests/django_hstore_tests/tests.py", line 430, in test_serialization_deserialization
    self.assertEqual(json.loads(str(DataBag.objects.get(name='alpha').data)), json.loads(str(alpha.data)))
  File "/usr/lib/python2.7/json/__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.7/json/decoder.py", line 369, in decode
    raise ValueError(errmsg("Extra data", s, end, len(s)))
ValueError: Extra data: line 1 column 4 - line 1 column 20 (char 3 - 19)

======================================================================
ERROR: test_hremove (tests.django_hstore_tests.tests.TestReferencesField)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/alukach/Projects/django-hstore/tests/django_hstore_tests/tests.py", line 1057, in test_hremove
    self.assertEqual(RefsBag.objects.get(name='alpha').refs['0'], alpha.refs['0'])
  File "/home/alukach/Projects/django-hstore/django_hstore/dict.py", line 149, in __getitem__
    value = super(self.__class__, self).__getitem__(*args, **kwargs)
  File "/home/alukach/Projects/django-hstore/django_hstore/dict.py", line 73, in __getitem__
    value = super(HStoreDict, self).__getitem__(*args, **kwargs)
KeyError: '0'

======================================================================
ERROR: test_hslice (tests.django_hstore_tests.tests.TestReferencesField)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/alukach/Projects/django-hstore/tests/django_hstore_tests/tests.py", line 1071, in test_hslice
    self.assertEqual(RefsBag.objects.hslice(id=alpha.id, attr='refs', keys=['0']), {'0': refs[0]})
  File "/home/alukach/Projects/django-hstore/django_hstore/managers.py", line 33, in hslice
    return self.filter(**params).hslice(attr, keys)
  File "/home/alukach/Projects/django-hstore/django_hstore/query.py", line 46, in selector
    return method(self, query, *args, **params)
  File "/home/alukach/Projects/django-hstore/django_hstore/query.py", line 294, in hslice
    return dict((key, field._value_to_python(value)) for key, value in result[0].items())
AttributeError: 'str' object has no attribute 'items'

======================================================================
ERROR: test_simple_retrieval (tests.django_hstore_tests.tests.TestReferencesField)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/alukach/Projects/django-hstore/tests/django_hstore_tests/tests.py", line 995, in test_simple_retrieval
    self.assertEqual(Ref.objects.get(name='0'), alpha.refs['0'])
  File "/home/alukach/Projects/django-hstore/django_hstore/dict.py", line 149, in __getitem__
    value = super(self.__class__, self).__getitem__(*args, **kwargs)
  File "/home/alukach/Projects/django-hstore/django_hstore/dict.py", line 73, in __getitem__
    value = super(HStoreDict, self).__getitem__(*args, **kwargs)
KeyError: '0'

======================================================================
FAIL: test_hremove (tests.django_hstore_tests.tests.TestDictionaryField)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/alukach/Projects/django-hstore/tests/django_hstore_tests/tests.py", line 395, in test_hremove
    self.assertEqual(DataBag.objects.get(name='alpha').data, alpha.data)
AssertionError: '"v"=>"1", "v2"=>"3"' != {'v2': '3', 'v': '1'}

======================================================================
FAIL: test_hupdate (tests.django_hstore_tests.tests.TestDictionaryField)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/alukach/Projects/django-hstore/tests/django_hstore_tests/tests.py", line 411, in test_hupdate
    self.assertEqual(DataBag.objects.get(name='alpha').data, alpha.data)
AssertionError: '"v"=>"1", "v2"=>"3"' != {'v2': '3', 'v': '1'}

======================================================================
FAIL: test_properties_hstore (tests.django_hstore_tests.tests.TestDictionaryField)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/alukach/Projects/django-hstore/tests/django_hstore_tests/tests.py", line 589, in test_properties_hstore
    self.assertEqual(type(instance.data), HStoreDict)
AssertionError: <type 'str'> != <class 'django_hstore.dict.HStoreDict'>

======================================================================
FAIL: test_replace_full_dictionary (tests.django_hstore_tests.tests.TestDictionaryField)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/alukach/Projects/django-hstore/tests/django_hstore_tests/tests.py", line 229, in test_replace_full_dictionary
    self.assertEqual(replacement, DataBag.objects.get(name='foo').data)
AssertionError: {'added': 'new', 'change': 'new value'} != '"added"=>"new", "change"=>"new value"'

======================================================================
FAIL: test_unicode_processing (tests.django_hstore_tests.tests.TestDictionaryField)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/alukach/Projects/django-hstore/tests/django_hstore_tests/tests.py", line 207, in test_unicode_processing
    self.assertEqual(greets, DataBag.objects.get(name='multilang').data)
AssertionError: {u'en': u'hello, world', u'zh': u'\u4f60\u597d\uff0c\u4e16\u754c', u'de': u'Gr\xfc\xdfe, Welt', u'jp': u'\u3053\u3093\u306b\u3061\u306f\u3001\u4e16\u754c', u'es': u'hola, ma\xf1ana', u'he': u'\u05e9\u05dc\u05d5\u05dd, \u05e2\u05d5\u05dc\u05dd'} != '"de"=>"Gr\xc3\xbc\xc3\x9fe, Welt", "en"=>"hello, world", "es"=>"hola, ma\xc3\xb1ana", "he"=>"\xd7\xa9\xd7\x9c\xd7\x95\xd7\x9d, \xd7\xa2\xd7\x95\xd7\x9c\xd7\x9d", "jp"=>"\xe3\x81\x93\xe3\x82\x93\xe3\x81\xab\xe3\x81\xa1\xe3\x81\xaf\xe3\x80\x81\xe4\xb8\x96\xe7\x95\x8c", "zh"=>"\xe4\xbd\xa0\xe5\xa5\xbd\xef\xbc\x8c\xe4\xb8\x96\xe7\x95\x8c"'

======================================================================
FAIL: test_location_create (tests.django_hstore_tests.tests.TestDictionaryFieldPlusGIS)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/alukach/Projects/django-hstore/tests/django_hstore_tests/tests.py", line 1216, in test_location_create
    self.assertEqual(other_loc.data, {'prop1': '1', 'prop2': 'test_value'})
AssertionError: '"prop1"=>"1", "prop2"=>"test_value"' != {'prop1': '1', 'prop2': 'test_value'}

======================================================================
FAIL: test_location_hupdate (tests.django_hstore_tests.tests.TestDictionaryFieldPlusGIS)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/alukach/Projects/django-hstore/tests/django_hstore_tests/tests.py", line 1222, in test_location_hupdate
    self.assertEqual(loc.data, {'prop1': '2', 'prop2': 'test_value'})
AssertionError: '"prop1"=>"2", "prop2"=>"test_value"' != {'prop1': '2', 'prop2': 'test_value'}

======================================================================
FAIL: test_simple_retrieval_get (tests.django_hstore_tests.tests.TestReferencesField)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/alukach/Projects/django-hstore/tests/django_hstore_tests/tests.py", line 1000, in test_simple_retrieval_get
    self.assertEqual(Ref.objects.get(name='0'), alpha.refs.get('0'))
AssertionError: <Ref: Ref object> != None

======================================================================
FAIL: test_location_create (tests.django_hstore_tests.tests.TestReferencesFieldPlusGIS)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/alukach/Projects/django-hstore/tests/django_hstore_tests/tests.py", line 1248, in test_location_create
    self.assertEqual(loc_1.data, {'prop1': '1', 'prop2': 'test_value'})
AssertionError: '"prop1"=>"1", "prop2"=>"test_value"' != {'prop1': '1', 'prop2': 'test_value'}

======================================================================
FAIL: test_location_hupdate (tests.django_hstore_tests.tests.TestReferencesFieldPlusGIS)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/alukach/Projects/django-hstore/tests/django_hstore_tests/tests.py", line 1256, in test_location_hupdate
    self.assertEqual(loc.data, {'prop1': '2', 'prop2': 'test_value'})
AssertionError: '"prop1"=>"2", "prop2"=>"test_value"' != {'prop1': '2', 'prop2': 'test_value'}

======================================================================
FAIL: test_hstore_registring_in_transaction_block (tests.django_hstore_tests.tests.NotTransactionalTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/alukach/Projects/django-hstore/tests/django_hstore_tests/tests.py", line 942, in test_hstore_registring_in_transaction_block
    self.assertIsInstance(qs[0].data, HStoreDict)
AssertionError: '"v"=>"1", "v2"=>"3"' is not an instance of <class 'django_hstore.dict.HStoreDict'>

----------------------------------------------------------------------
Ran 118 tests in 1.962s

FAILED (failures=11, errors=17)
Destroying test database for alias 'default'...

(hstore)$ dropdb django_hstore
(hstore)$ createdb django_hstore  # Recreate DB (from template with hstore enabled)
(hstore)$ python runtests.py  # Tests will pass
Creating test database for alias 'default'...
../home/alukach/.virtualenvs/hstore/local/lib/python2.7/site-packages/django/db/models/base.py:1029: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
  if f.blank and raw_value in f.empty_values:

/home/alukach/.virtualenvs/hstore/local/lib/python2.7/site-packages/django/db/models/fields/__init__.py:458: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
  if value in self.empty_values:

........................................./home/alukach/Projects/django-hstore/django_hstore/query.py:57: RemovedInDjango18Warning: 'is_managed' is deprecated.
  if not transaction.is_managed(using=self.db):

...........................................................................
----------------------------------------------------------------------
Ran 118 tests in 1.994s

OK
Destroying test database for alias 'default'...
(hstore)$

alukach commented 9 years ago

@nemesisdesign And to answer your question, this is the output of SELECT t.oid FROM pg_type t JOIN pg_namespace ns ON typnamespace = ns.oid WHERE typname = 'hstore' and nspname = 'public'; on my database and test database when the conditions for the bug are met:

(hstore)$ psql test_django_hstore -c "SELECT t.oid FROM pg_type t JOIN pg_namespace ns ON typnamespace = ns.oid WHERE typname = 'hstore' and nspname = 'public';"
  oid   
--------
 913637
(1 row)

(hstore)$ psql django_hstore -c "SELECT t.oid FROM pg_type t JOIN pg_namespace ns ON typnamespace = ns.oid WHERE typname = 'hstore' and nspname = 'public';"
  oid   
--------
 913497
(1 row)

ddohler commented 9 years ago

@alukach -- Great sleuthing! That looks like it explains our problem as well, because our database setup process is similar:

Create template_postgis
Create django app database from template_postgis
Run CREATE EXTENSION hstore; on django app database
RunCREATE EXTENSION hstore; on template_postgis

In other words, template_postgis is not hstore-enabled until after the django app database is created, which then (apparently) causes the postgis backend to erroneously believe that the test database is not hstore-enabled when in fact it is, while the postgresql_psycopg2 backend correctly detects and utilizes the hstore extension.

alukach commented 9 years ago

Okay, doing some late night research, I think I'm beginning to see what the issue is. Here are my notes so far:

When running tests with the postgis backend:

This function is called during setup with settings_dict['NAME'] set as the Django db name (ex. django_hstore) (this is set to None when using the postgresql_psycopg backend).
That opens a connection, calling django-hstore's connection handler.
That calls register_hstore from psycopg2.extras, which iterates through the connection's oids, discovering & registering the hstore oid for the prod db.

It isn't until later that the connection is opened for the test db (test_django_hstore). When that connection is opened, register_hstore is not rerun. This means that psycopg2 will be looking for hstore at the oid that it existed under on django_hstore, rather than the oid from test_django_hstore. If they're both created from the same template, the oids will match and everything will work fine. However, if the django_hstore db has the hstore extension created later, then it's oid will NOT match the template oid, thus causing errors.

It appears that a possible solution could be to rerun register_hstore when the DB changes from django_hstore to test_django_hstore. I haven't really looked into how to do that.

nemesifier commented 9 years ago

@alukach thank you for your research.

Any other suggestion of improvement?

Federico

rklyne commented 9 years ago

I think I've found the problem and solution. My reproduction was different though. Django signals are used to observe every new connection and call psycopg2's register_hstore function on it. Django signals will, by default, store references to the receiver function as weakrefs. In a low memory situation these can go away and future connections are not registered to use hstore features. My suggested fix is on PR-128 - https://github.com/djangonauts/django-hstore/pull/128

Deepakdubey90 commented 8 years ago

Hey Guys,

          I'm also facing the same issue with "HstoreField" , while running the django unit Test-Cases .

I don't understand why its throwing error Test_Database but not throwing error for actual Database creation. postgres ==9.3.10. django ==1.8.6

////////////////////////////// Error :: "django.db.utils.ProgrammingError: can't adapt type 'dict'" /////////////////////////// Is there a way to fixed it.????

rklyne commented 8 years ago

That's not the error I was seeing, but possibly that's a 1.6-1.8 difference. There's a fix for this issue in master - install from there and see if that resolves your issue?

Deepakdubey90 commented 8 years ago

Thanks @rklyne, https://docs.djangoproject.com/en/1.8/ref/contrib/postgres/fields/ I got solution from above url.

nemesifier commented 8 years ago

I'm closing this, hoping it is will be finally solved with #128

djangonauts / django-hstore

register_hstore not getting called properly during tests #93