Incorrect handling of default value when adding a new field to existing model via Django 1.7 migrations

[lanzz]

Let's say we have an existing model Foo, with a bunch of records stored in its table. Let's add a JSON field to model Foo:

class Foo(models.Model):
    # ...other fields...
    bar = JsonField(blank=True, default={})

Let's now make a Django 1.7 migration and apply it:

./manage.py makemigrations
./manage.py migrate

Suddenly, all existing records have a new field bar, populated with a value of '"{}"' — i.e. a JSON string containing the representation of an empty Python dict, instead of an empty JSON object.

[niwinz]

Thanks for report it! I will reasearch and try fix it as soon as I can!

[niwinz]

I have research about it and every thing works correctly. Json is a serialization format, and a result is a text, and "{}" is a valid representation of json empty object. Can you explain it little more, because I does not understand completelly the problem that you are having.

I just added tests for check the default value handling, and everything is working as expected (https://github.com/djangonauts/django-pgjson/blob/ac4976260fb0bd15db5361718c72dae438b54e61/tests/pg_json_fields/tests.py#L20)

[lanzz]

The value stored in the database is not an empty JSON object ({}), but a JSON string containing the two literal characters ("{}"). It is not the value of JSON.dumps({}), but the value of JSON.dumps(JSON.dumps({})) instead.

[niwinz]

I understand, but I can not reproduce it, tests are working as expected. Having JSON.dumps(JSON.dumps({})) stored in a field, the reverse value (on retrieving the object from db) will be a string, but in tests is a properly deserialized python dict.

Can you provide some testcase for reproduce it? I have this package working as expectly in many environments, I really don't understand why that happens...

[lanzz]

Steps to reproduce:

1. Create a new model:

   from django.db import models
   
   class Example(models.Model):
   
      foo = models.CharField(max_length=100)

2. Create a record in the new model:

   >>> Example.objects.create(foo="Hello")

3. Add a JSON field to the model:

   from django.db import models
   from django_pgjson.fields import JsonField
   
   class Example(models.Model):
   
      foo = models.CharField(max_length=100)
      bar = JsonField(blank=True, default={})

4. Migrate the model:

   $ ./manage.py makemigrations
   $ ./manage.py migrate

5. Observe database contents and note that quotes are included in the JSON contents, i.e. it is a string, not a dictionary:

   pgjsonrepro=> select * from example_example;
   id |  foo  | bar  
   ----+-------+------
    1 | Hello | "{}"
   (1 row)

6. Attempt to access that JSON field's contents in a custom query and fail miserably:

   pgjsonrepro=> select * from example_example where bar->>'key' = 'value';
   ERROR:  cannot extract element from a scalar

7. Confirm that django_pgjson can indeed deserialize that malformed field:

   >>> ex = Example.objects.get(foo="Hello")
   >>> type(ex.bar)
   <type 'dict'>

8. Save that model instance:

   >>> ex.save()

9. Observe database contents and note that quotes are no longer included in the JSON contents, i.e. it has become an actual JSON dictionary now:

   pgjsonrepro=> select * from example_example;
   id |  foo  | bar  
   ----+-------+------
    1 | Hello | {}
   (1 row)

10. Attempt to access that JSON field's contents in a custom query and note that it now works without raising an error:

    pgjsonrepro=> select * from example_example where bar->>'key' = 'value';
     id | foo | bar 
    ----+-----+-----
    (0 rows)

I hope this step-by-step example makes it clear what the problem is, namely that the presence of JSON strings instead of dictionaries in the table makes it impossible to use native JSON introspection operators in custom queries (step 6).

Tested with latest packages available from pip: Django-1.7.1, psycopg2-2.5.4 and django-pgjson-0.2.0.

[niwinz]

Thank you very much, with this detailed information I'll try to reproduce it.

[niwinz]

Check the latest 0.2.1 version. It should fix that.