Problem pickling SortedDictWithMetadata type output of Serializer.data

[roberts81]

I assumed that I would be able to pickle the .data from Serializers without an issue, but it is failing on a certain nested Serializer of mine.

This bug has been been really hard to pinpoint what the problem is, since it happens deeply nested in some recursive pickle juice, but it looks like its trying to pickle the class for the model where the dictionary came from, for some reason related to the fact that the dict is actually a SortedDictWithMetadata. I assume it may be the Metadata?

When I run pickle.dumps on .data from my serializer i get:

PicklingError: Can't pickle <class 'menus.models.Day'>: attribute lookup menus.models.Day failed

or

PicklingError: Can't pickle <class 'menus.models.Day'>: it's not found as menus.models.Day

depending on the path to run it (the latter is if I try to pickle it directly, former comes from if i try to let pylibmc pickle it).

One of the few useful clues that I have is that if I cast the problem dictionary to a plain dict from the SortedDictWithMetadata then it becomes picklable, so I assume something about SortedDictWithMetadata is the problem. And I've double checked my dictionary and its all native types.

While I realize I probably haven't provided enough info for a resolution, I wanted to open the issue to see if it rings a bell for anyone, and then I'll follow up with a failed test or something in the next few days when i get some time to do so. In the mean time, i'm going to recursively cast all my SortedDictWithMetadata's to dicts before pickling to get past this issue.

Also, I'm using nested Serializers, 4 deep. The error happens on any of the resulting dictionaries.

menu_week (MenuWeekSerializer) --> days (MenuDaySerializer) --> menu_items (MenuItemSerializer) --> food (FoodSerializer)

Here's the mostly useless stack trace:

(food is the inner most dictionary of the rats nest that raises the error (rounded_nutrition_info is picklable) )

>>> food
{'id': 12, 'name': u'some name', 'description': u'some description', 'image_url': u'/media/food_images/a.png', 'hoverpic_url': u'/media/food_images/a150x0_q85.jpg', 'price': None, 'ingredients': u'ingredients', 'rounded_nutrition_info': {'mg_sodium': 0, 'g_carbs': 0, 'g_sugar': 0}}
>>> type(food)
<class 'rest_framework.serializers.SortedDictWithMetadata'>
>>> df = dict(food)
>>> pickle.dumps(df)
'blob.'
>>> pickle.dumps(f)
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 1374, in dumps
    Pickler(file, protocol).dump(obj)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 224, in dump
    self.save(obj)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 419, in save_reduce
    save(state)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 663, in _batch_setitems
    save(v)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 663, in _batch_setitems
    save(v)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 419, in save_reduce
    save(state)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 663, in _batch_setitems
    save(v)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 419, in save_reduce
    save(state)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 663, in _batch_setitems
    save(v)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 419, in save_reduce
    save(state)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 663, in _batch_setitems
    save(v)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 419, in save_reduce
    save(state)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 663, in _batch_setitems
    save(v)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 419, in save_reduce
    save(state)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 663, in _batch_setitems
    save(v)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 419, in save_reduce
    save(state)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 663, in _batch_setitems
    save(v)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 600, in save_list
    self._batch_appends(iter(obj))
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 615, in _batch_appends
    save(x)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 401, in save_reduce
    save(args)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 562, in save_tuple
    save(element)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 748, in save_global
    (obj, module, name))
PicklingError: Can't pickle <class 'menus.models.Day'>: it's not found as menus.models.Day

[tomchristie]

The fix would be to define getstate on SortedDictWithMetadata, to ignore the metadata (references to the fields that were used to serializer the data) and only pickle the dict content.

See also this related ticket on pickling Response objects: https://github.com/tomchristie/django-rest-framework/issues/346

On 29 Nov 2012, at 00:20, roberts81 notifications@github.com wrote:

I assumed that I would be able to pickle the .data from Serializers without an issue, but it is failing on a certain nested Serializer of mine.

This bug has been been really hard to pinpoint what the problem is, since it happens deeply nested in some recursive pickle juice, but it looks like its trying to pickle the class for the model where the dictionary came from, for some inexplicable reason related to the fact that the dict is actually a SortedDictWithMetadata. I assume it may be the Metadata?

When I run pickle.dumps on .data from my serializer i get:

PicklingError: Can't pickle : attribute lookup menus.models.Day failed

I found out that if I cast the problem dictionary to a plain dict from the SortedDictWithMetadata that it becomes picklable.

While I realize I probably haven't provided enough info for a resolution, I wanted to open the issue to see if it rings a bell for anyone, and then I'll follow up with a failed test or something in the next few days when i get some time to do so. In the mean time, i'm going to recursively cast all my SortedDictWithMetadata's to dicts before pickling to get past this issue.

— Reply to this email directly or view it on GitHub.

[roberts81]

After I started writing tests to demonstrate the problem I realized that its probably more of a problem that pickle has with my code than one with RestFramework. The problem was that I was trying to serialize (non-model) objects whose class (Day) was an inner class of another class (Week). This doesn't seem to cause problems in any other DRF usage that I'd encountered, but due to the Metadata of the fields on the dict, Pickle is trying to import the Day class directly from the module and borks when it can't find it to import. So the error messaging I was getting from pickle turns out to be spot on and not that confusing, I just couldn't see the plain answer in front of my face.

Anyhow, the simple solution was to move my inner class to be a top level class in the module (which was probably the right thing to do anyhow). And things seem to pickle nicely now, even with the Metadata on the dicts.

So, as far as I'm concerned the problem is resolved. And it seems to be more of an issue between pickle and inner classes than something DRF-specific. But in case with greater understanding, Tom or someone decides this is a problem that's worth fixing via the getstate override mentioned above, I'll leave that up to y'all to decide whether to close the issue or not.

Here's the test I wrote that led me to figure it out. As you can see, if either the Serializer or the instance's Class (not demonstrated) isn't at the top level of the module, it will fail . (I figured its probably not worth commiting to the framework, although you're welcome to if you decide to fix this issue)

    def test_pickle_simple_model_serializer(self):
        pickle.dumps(PersonSerializer(Person(name="foob")).data) #passes

    def test_pickle_inner_serializer(self):
            class InnerPersonSerializer(serializers.ModelSerializer):
                class Meta:
                    model = Person
                    fields = ('name', 'age')
            pickle.dumps(InnerPersonSerializer(Person(name="methusela", age=969)).data) #fails

[roberts81]

I had this problem again for something different thats not so easily resolved, and am convinced its a legitimate issue that will probably come up again. Thanks Tom for pointing me to where to fix it. I've issued a pull request with a test and a functional fix. https://github.com/tomchristie/django-rest-framework/pull/477

[tomchristie]

Closing now that #477 is merged