'cache_response' decorator pickle error.

[caxap]

I run into the problem after update to django-rest-framework==3.0.4 (redis==2.10.3, django-redis-cache==0.13.0):

Traceback (most recent call last):
  File "/home/app/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 111, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/app/local/lib/python2.7/site-packages/django/views/decorators/csrf.py", line 57, in wrapped_view
    return view_func(*args, **kwargs)
  File "/home/app/local/lib/python2.7/site-packages/django/views/generic/base.py", line 69, in view
    return self.dispatch(request, *args, **kwargs)
  File "/home/app/local/lib/python2.7/site-packages/rest_framework/views.py", line 407, in dispatch
    response = self.handle_exception(exc)
  File "/home/app/local/lib/python2.7/site-packages/rest_framework/views.py", line 404, in dispatch
    response = handler(request, *args, **kwargs)
  File "/home/app/local/lib/python2.7/site-packages/rest_framework/generics.py", line 278, in get
    return self.retrieve(request, *args, **kwargs)
  File "/home/app/local/lib/python2.7/site-packages/rest_framework_extensions/cache/decorators.py", line 34, in inner
    kwargs=kwargs,
  File "/home/app/local/lib/python2.7/site-packages/rest_framework_extensions/cache/decorators.py", line 56, in process_cache_response
    self.cache.set(key, response, self.timeout)
  File "/home/app/local/lib/python2.7/site-packages/redis_cache/cache.py", line 239, in set
    result = self._set(key, pickle.dumps(value), timeout, client, _add_only)
  File "/home/app/lib/python2.7/copy_reg.py", line 70, in _reduce_ex
    raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: can't pickle function objects

[chibisov]

Could you provide more code snippets from your app? For example, view implementation.

[caxap]

Sure, I found that problem related to defautdict w/ lambda expression:

from collections import defaultdict

class TestView(generics.RetrieveAPIView):

    def calculate_cache_key(self, view_instance, view_method, request, args, kwargs):
        return 'test'

    @cache_response(10 * 60, key_func='calculate_cache_key')
    def retrieve(self, request, *args, **kwargs):
        return Response({'test': defaultdict(lambda: 1.0)})   # <-- cause pickle error

But in fact if you remove @cache_response decorator or replace defaultdict(lambda: 1.0) to something w/o lambda defaultdict(int) it works fine.

[chibisov]

Hm, does it work without @cache_response decorator? Python can't pickle and store in cache your lambda function:

>>> {'test': defaultdict(lambda: 1.0)}
{'test': defaultdict(<function <lambda> at 0x100ffaaa0>, {})}

[caxap]

I think it works because of https://github.com/tomchristie/django-rest-framework/blob/master/rest_framework/utils/encoders.py#L58

DRF performs additional steps to encode response data.

[elioscordo]

Hi, any solutions for this? I am using a ModelSerializer... It is also hard to understand where the pickle error comes from.

[nemesifier]

I just upgraded a project to DRF 3.2.3 and I bumped into this issue.

Here's a code sample of nodeshot.core.cms.views:

class PageList(ACLMixin, generics.ListAPIView):
    """
    Retrieve the list of all the available pages.
    """
    authentication_classes = (authentication.SessionAuthentication,)
    queryset = Page.objects.published()
    serializer_class = PageListSerializer

    @cache_response(86400, key_func=cache_by_group)
    def get(self, request, *args, **kwargs):
        return super(PageList, self).get(request, *args, **kwargs)

Stacktrace:

  File "django/core/handlers/base.py", line 132, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "django/views/generic/base.py", line 71, in view
    return self.dispatch(request, *args, **kwargs)
  File "rest_framework/views.py", line 466, in dispatch
    response = self.handle_exception(exc)
  File "rest_framework/views.py", line 463, in dispatch
    response = handler(request, *args, **kwargs)
  File "rest_framework_extensions/cache/decorators.py", line 43, in inner
    kwargs=kwargs,
  File "rest_framework_extensions/cache/decorators.py", line 60, in process_cache_response
    response = self.cache.get(key)
  File "django_redis/cache.py", line 25, in _decorator
    return method(self, *args, **kwargs)
  File "django_redis/cache.py", line 73, in get
    client=client)
  File "django_redis/client/default.py", line 205, in get
    return self.decode(value)
  File "django_redis/client/default.py", line 303, in decode
    value = self._serializer.loads(value)
  File "django_redis/serializers/pickle.py", line 43, in loads
    return pickle.loads(force_bytes(value))

[nemesifier]

Just wondering, could it be a DRF bug? We bumped into a similar bug in DRF-gis.

[auvipy]

@nemesisdesign did you fixed the issue on your branch? if so feel free to send a PR