Encoding error when passing a unicode string to an Activity's extra_context

[ahumeau]

Hello, First, thank you for this useful work. I encountered an error when trying to store text containing non ASCII characters in an activity's extra_context and passing it as a unicode string (actually, the error occurs when fetching the activity from storage).

Here is a fork of the feedly_pin repo modified to reproduce the error: https://github.com/ahumeau/feedly_pin

And here is the traceback:

Environment:

Request Method: POST
Request URL: http://localhost:8083/pin/

Django Version: 1.5.5
Python Version: 2.7.3
Installed Applications:
('django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'core',
 'south',
 'djcelery',
 'compressor')
Installed Middleware:
('django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware')

Traceback:
File "/home/vagrant/feedly/local/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response
  115.                         response = callback(request, *callback_args, **callback_kwargs)
File "/home/vagrant/feedly/local/lib/python2.7/site-packages/django/contrib/auth/decorators.py" in _wrapped_view
  25.                 return view_func(request, *args, **kwargs)
File "/home/vagrant/feedly/feedly_pin/core/views.py" in pin
  93.             pin = form.save()
File "/home/vagrant/feedly/feedly_pin/core/forms.py" in save
  39.         feedly.add_pin(pin)
File "/home/vagrant/feedly/feedly_pin/core/pin_feedly.py" in add_pin
  20.         self.add_user_activity(pin.user_id, activity)
File "/home/vagrant/feedly/local/lib/python2.7/site-packages/feedly/feed_managers/base.py" in add_user_activity
  152.                     fanout_priority=priority_group
File "/home/vagrant/feedly/local/lib/python2.7/site-packages/feedly/feed_managers/base.py" in create_fanout_tasks
  319.                 operation_kwargs=operation_kwargs
File "/home/vagrant/feedly/local/lib/python2.7/site-packages/celery/app/task.py" in delay
  453.         return self.apply_async(args, kwargs)
File "/home/vagrant/feedly/local/lib/python2.7/site-packages/celery/app/task.py" in apply_async
  547.                               link=link, link_error=link_error, **options)
File "/home/vagrant/feedly/local/lib/python2.7/site-packages/celery/app/task.py" in apply
  735.                                         request=request, propagate=throw)
File "/home/vagrant/feedly/local/lib/python2.7/site-packages/celery/app/trace.py" in eager_trace_task
  354.         uuid, args, kwargs, request)
File "/home/vagrant/feedly/local/lib/python2.7/site-packages/celery/app/trace.py" in trace_task
  253.                     I, R, state, retval = on_error(task_request, exc, uuid)
File "/home/vagrant/feedly/local/lib/python2.7/site-packages/celery/app/trace.py" in trace_task
  240.                     R = retval = fun(*args, **kwargs)
File "/home/vagrant/feedly/local/lib/python2.7/site-packages/feedly/tasks.py" in fanout_operation_hi_priority
  17.     return fanout_operation(feed_manager, feed_class, user_ids, operation, operation_kwargs)
File "/home/vagrant/feedly/local/lib/python2.7/site-packages/celery/app/task.py" in __call__
  420.             return self.run(*args, **kwargs)
File "/home/vagrant/feedly/local/lib/python2.7/site-packages/feedly/tasks.py" in fanout_operation
  11.     feed_manager.fanout(user_ids, feed_class, operation, operation_kwargs)
File "/home/vagrant/feedly/local/lib/python2.7/site-packages/feedly/feed_managers/base.py" in fanout
  346.                     operation(feed, **operation_kwargs)
File "/home/vagrant/feedly/local/lib/python2.7/site-packages/feedly/feed_managers/base.py" in add_operation
  24.     feed.add_many(activities, batch_interface=batch_interface, trim=trim)
File "/home/vagrant/feedly/local/lib/python2.7/site-packages/feedly/feeds/aggregated_feed/base.py" in add_many
  90.             current_activities = self[:self.merge_max_length]
File "/home/vagrant/feedly/local/lib/python2.7/site-packages/feedly/feeds/base.py" in __getitem__
  302.                 start, bound)
File "/home/vagrant/feedly/local/lib/python2.7/site-packages/feedly/feeds/base.py" in get_activity_slice
  346.             activities = self.hydrate_activities(activities)
File "/home/vagrant/feedly/local/lib/python2.7/site-packages/feedly/feeds/base.py" in hydrate_activities
  324.         activity_list = self.activity_storage.get_many(activity_ids)
File "/home/vagrant/feedly/local/lib/python2.7/site-packages/feedly/storage/base.py" in get_many
  179.         activities_data = self.get_from_storage(activity_ids, *args, **kwargs)
File "/home/vagrant/feedly/local/lib/python2.7/site-packages/feedly/storage/redis/activity_storage.py" in get_from_storage
  23.         activities = dict((k, unicode(v)) for k, v in activities.items() if v)
File "/home/vagrant/feedly/local/lib/python2.7/site-packages/feedly/storage/redis/activity_storage.py" in <genexpr>
  23.         activities = dict((k, unicode(v)) for k, v in activities.items() if v)

Exception Type: UnicodeDecodeError at /pin/
Exception Value: 'ascii' codec can't decode byte 0xe6 in position 67: ordinal not in range(128)

[ahumeau]

The error does not appear when using json instead of pickle to serialize activities: https://github.com/ahumeau/feedly_pin/tree/serialize_activities_with_json

[jquacinella]

+1 this issue. I have having issues with this as well.

@ahumeau when you say you don't have this issue with JSON, is there an option to use JSON serializing versus pickling? Thanks for any insight!

[izhan]

+1

encountering this issue on my end as well when using a Redis Notification Feed @jquacinella what was your solution in the end?

[jquacinella]

@izhan No solution as of yet :(

[izhan]

@jquacinella if you are using a Redis store and Python 2, have you tried the fix I proposed in https://github.com/tschellenbach/Stream-Framework/issues/147? Let me know if that works for you! :)