Serialization Error / KeyError in get_hydrated (?)

tschellenbach / Stream-Framework

Stream Framework is a Python library, which allows you to build news feed, activity streams and notification systems using Cassandra and/or Redis. The authors of Stream-Framework also provide a cloud service for feed technology:

https://getstream.io/

Other

4.73k stars 542 forks source link

Serialization Error / KeyError in get_hydrated (?) #15

Closed andrenro closed 10 years ago

andrenro commented 10 years ago

I have an issue similar to that in issue #8 .

Im trying to make a REST service with this, I use django-rest-framework.

Seems adding activities to redis works, but when i try to fetch them back i will get this

KeyError: 13869294460000000000005005L

seems like the "L" is causing some sort of error ?

function calls and stack trace: wishlists/views.py

feedly = UserWishFeedly()

//function for adding activities: activity = Activity(wishlist.user, WishVerb, in_prod.id) feed = UserWishFeed(wishlist.user.id) feed.add(activity)

//test-function for getting the feed def wish_feed(request,args,*kwargs): feed = feedly.get_user_feed(request.user.id) activities = feed[:10] content = {"activities":activities} json_data = json.dumps(content) return Response(json_data)

File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py", line 115, in get_response response = callback(request, _callback_args, *_callback_kwargs)

File "/usr/local/lib/python2.7/dist-packages/django/views/decorators/csrf.py", line 77, in wrapped_view return view_func(_args, *_kwargs)

File "/usr/local/lib/python2.7/dist-packages/django/views/generic/base.py", line 68, in view return self.dispatch(request, _args, *_kwargs)

File "/usr/local/lib/python2.7/dist-packages/django/views/decorators/csrf.py", line 77, in wrapped_view return view_func(_args, *_kwargs)

File "/usr/local/lib/python2.7/dist-packages/rest_framework/views.py", line 327, in dispatch response = self.handle_exception(exc)

File "/usr/local/lib/python2.7/dist-packages/rest_framework/views.py", line 324, in dispatch response = handler(request, _args, *_kwargs)

File "/usr/local/lib/python2.7/dist-packages/rest_framework/decorators.py", line 49, in handler return func(_args, *_kwargs)

File "/usr/local/lib/python2.7/dist-packages/django/contrib/auth/decorators.py", line 25, in _wrapped_view return view_func(request, _args, *_kwargs)

File "/home/andrenro/dev_giftit_webapp/env/Scripts/giftit_webapp/wishlists/views.py", line 437, in wish_feed activities = feed[:10]

File "/usr/local/lib/python2.7/dist-packages/feedly/feeds/base.py", line 261, in get item start, bound)

File "/usr/local/lib/python2.7/dist-packages/feedly/feeds/base.py", line 304, in get_activity_slice activities = self.hydrate_activities(activities)

File "/usr/local/lib/python2.7/dist-packages/feedly/feeds/base.py", line 285, in hydrate_activities return [activity.get_hydrated(activity_data) for activity in activities]

File "/usr/local/lib/python2.7/dist-packages/feedly/activity.py", line 41, in get_hydrated activity = activities[int(self.serialization_id)]

KeyError: 13869294460000000000005005L

If i try redis-cli i get:

127.0.0.1:6379> zrange "feed:user:6" 0 25 1) "13869294460000000000005005"

Im running python 2.7.3 and django 1.5.4 Im kinda stuck with this, hope someone can shed some light on it. Great project btw!!

tbarbugli commented 10 years ago

Can you post the code you are using to insert activities in feedly ? My guess is that only the reference to the activity is stored on redis

andrenro commented 10 years ago

I actually experimented with it and got passed that issue, but now I'm getting a connection, "Error: [Errno 111] Connection refused " I know that the redis-server is running, I have also tried to restart the server. Im trying this on a live server with the settings used in the docs:

FEEDLY_NYDUS_CONFIG { 'CONNECTIONS': { 'redis': { 'engine': 'nydus.db.backends.redis.Redis', 'router': 'nydus.db.routers.redis.PrefixPartitionRouter', 'hosts': { 0: {'prefix': 'default', 'db': 0, 'host': 'localhost', 'port': 6379}, } }, } }

Stack trace: Traceback (most recent call last):

File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py", line 115, in get_response response = callback(request, _callback_args, *_callback_kwargs)

File "/usr/local/lib/python2.7/dist-packages/django/views/generic/base.py", line 68, in view return self.dispatch(request, _args, *_kwargs)

File "/usr/local/lib/python2.7/dist-packages/django/views/decorators/csrf.py", line 77, in wrapped_view return view_func(_args, *_kwargs)

File "/usr/local/lib/python2.7/dist-packages/rest_framework/views.py", line 327, in dispatch response = self.handle_exception(exc)

File "/usr/local/lib/python2.7/dist-packages/rest_framework/views.py", line 324, in dispatch response = handler(request, _args, *_kwargs)

File "/home/andrenro/dev_giftit_webapp/env/Scripts/giftit_webapp/wishlists/views.py", line 185, in put feedly.add_wish(wishlist.user,activity)

File "/home/andrenro/dev_giftit_webapp/env/Scripts/giftit_webapp/wishlists/models.py", line 109, in add_wish self.add_user_activity(user.id, activity)

File "/usr/local/lib/python2.7/dist-packages/feedly/feed_managers/base.py", line 149, in add_user_activity fanout_priority=priority_group

File "/usr/local/lib/python2.7/dist-packages/feedly/feed_managers/base.py", line 314, in create_fanout_tasks operation_kwargs=operation_kwargs

File "/usr/local/lib/python2.7/dist-packages/celery/app/task.py", line 442, in delay return self.apply_async(args, kwargs)

File "/usr/local/lib/python2.7/dist-packages/celery/app/task.py", line 544, in apply_async _dict(self._get_exec_options(), _options)

File "/usr/local/lib/python2.7/dist-packages/celery/app/base.py", line 317, in send_task reply_to=reply_to or self.oid, **options

File "/usr/local/lib/python2.7/dist-packages/celery/app/amqp.py", line 298, in publish_task **kwargs

File "/usr/local/lib/python2.7/dist-packages/kombu/messaging.py", line 166, in publish routing_key, mandatory, immediate, exchange, declare)

File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 462, in _ensured interval_max)

File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 374, in ensure_connection interval_start, interval_step, interval_max, callback)

File "/usr/local/lib/python2.7/dist-packages/kombu/utils/init.py", line 228, in retry_over_time return fun(_args, *_kwargs)

File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 245, in connect return self.connection

File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 752, in connection self._connection = self._establish_connection()

File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 711, in _establish_connection conn = self.transport.establish_connection()

File "/usr/local/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 111, in establish_connection conn = self.Connection(**opts)

File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", line 148, in init self.transport = create_transport(host, connect_timeout, ssl)

File "/usr/local/lib/python2.7/dist-packages/amqp/transport.py", line 300, in create_transport return TCPTransport(host, connect_timeout)

File "/usr/local/lib/python2.7/dist-packages/amqp/transport.py", line 109, in init raise socket.error(last_err)

error: [Errno 111] Connection refused

Seems to be an issue with ampq / kombu rather than in the feedly code?

tbarbugli commented 10 years ago

Hi, feedly uses celery to execute fanouts in the background. By default celery uses rabbitmq as broker, you can short-circuit celery and skip the broker part setting CELERY_ALWAYS_EAGER as True. (not suggested for production!) or you can setup a task broker (celery support redis too but I would not suggest it for large deployments)

Tommaso On 16 Dec 2013 21:49, "Andreas Røed" notifications@github.com wrote:

I actually experimented with it and got passed that issue, but now I'm getting a connection, "Error: [Errno 111] Connection refused " I know that the redis-server is running, I have also tried to restart the server. Im trying this on a live server with the settings used in the docs:

FEEDLY_NYDUS_CONFIG { 'CONNECTIONS': { 'redis': { 'engine': 'nydus.db.backends.redis.Redis', 'router': 'nydus.db.routers.redis.PrefixPartitionRouter', 'hosts': { 0: {'prefix': 'default', 'db': 0, 'host': 'localhost', 'port': 6379}, } }, } }

Stack trace: Traceback (most recent call last):

File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py", line 115, in get_response response = callback(request, _callback_args, *_callback_kwargs)

File "/usr/local/lib/python2.7/dist-packages/django/views/generic/base.py", line 68, in view return self.dispatch(request, _args, *_kwargs)

File "/usr/local/lib/python2.7/dist-packages/django/views/decorators/csrf.py", line 77, in wrapped_view return view_func(_args, *_kwargs)

File "/usr/local/lib/python2.7/dist-packages/rest_framework/views.py", line 327, in dispatch response = self.handle_exception(exc)

File "/usr/local/lib/python2.7/dist-packages/rest_framework/views.py", line 324, in dispatch response = handler(request, _args, *_kwargs)

File "/home/andrenro/dev_giftit_webapp/env/Scripts/giftit_webapp/wishlists/views.py", line 185, in put feedly.add_wish(wishlist.user,activity)

File "/home/andrenro/dev_giftit_webapp/env/Scripts/giftit_webapp/wishlists/models.py", line 109, in add_wish self.add_user_activity(user.id, activity)

File "/usr/local/lib/python2.7/dist-packages/feedly/feed_managers/base.py", line 149, in add_user_activity fanout_priority=priority_group

File "/usr/local/lib/python2.7/dist-packages/feedly/feed_managers/base.py", line 314, in create_fanout_tasks operation_kwargs=operation_kwargs

File "/usr/local/lib/python2.7/dist-packages/celery/app/task.py", line 442, in delay return self.apply_async(args, kwargs)

File "/usr/local/lib/python2.7/dist-packages/celery/app/task.py", line 544, in apply_async _dict(self._get_exec_options(), _options)

File "/usr/local/lib/python2.7/dist-packages/celery/app/base.py", line 317, in send_task reply_to=reply_to or self.oid, **options

File "/usr/local/lib/python2.7/dist-packages/celery/app/amqp.py", line 298, in publish_task **kwargs

File "/usr/local/lib/python2.7/dist-packages/kombu/messaging.py", line 166, in publish routing_key, mandatory, immediate, exchange, declare)

File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 462, in _ensured interval_max)

File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 374, in ensure_connection interval_start, interval_step, interval_max, callback)

File "/usr/local/lib/python2.7/dist-packages/kombu/utils/init.py", line 228, in retry_over_time return fun(_args, *_kwargs)

File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 245, in connect return self.connection

File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 752, in connection self._connection = self._establish_connection()

File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 711, in _establish_connection conn = self.transport.establish_connection()

File "/usr/local/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 111, in establish_connection conn = self.Connection(**opts)

File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", line 148, in init self.transport = create_transport(host, connect_timeout, ssl)

File "/usr/local/lib/python2.7/dist-packages/amqp/transport.py", line 300, in create_transport return TCPTransport(host, connect_timeout)

File "/usr/local/lib/python2.7/dist-packages/amqp/transport.py", line 109, in init raise socket.error(last_err)

error: [Errno 111] Connection refused

Seems to be an issue with ampq / kombu rather than in the feedly code?

— Reply to this email directly or view it on GitHubhttps://github.com/tschellenbach/Feedly/issues/15#issuecomment-30698978 .

andrenro commented 10 years ago

Hi again! So I will actually need to set celery to use redis in settings.py to get it going? Any suggestions for an easy setup? Seems Ive got some research and testing to do,maybe.. I will eventually need something that works well on a production server though. Haven't really used this kind of stuff on previous projects.. Thanks for good and helpful answers anyways!

tbarbugli commented 10 years ago

Hi, you can find plenty of docs about celery here http://docs.celeryproject.org/en/latest/ We built an example app with feedly here https://github.com/tbarbugli/feedly_pin you can use to have a look at settings (for dev environments)

My general suggestion is to disable celery during development (CELERY_ALWAYS_EAGER=True in settings) and to use rabbitmq or redis for production environments.

2013/12/16 Andreas Røed notifications@github.com

Hi again! So I will actually need to set celery to use redis in settings.py to get it going? Any suggestions for an easy setup? Seems Ive got some research and testing to do,maybe.. I will eventually need something that works well on a production server though. Haven't really used this kind of stuff on previous projects.. Thanks for good and helpful answers anyways!

— Reply to this email directly or view it on GitHubhttps://github.com/tschellenbach/Feedly/issues/15#issuecomment-30703062 .

andrenro commented 10 years ago

Hi, thanks a bunch! I'll check it out some more!

Den 17. des. 2013 kl. 09:07 skrev Tommaso Barbugli notifications@github.com:

Hi, you can find plenty of docs about celery here http://docs.celeryproject.org/en/latest/ We built an example app with feedly here https://github.com/tbarbugli/feedly_pin you can use to have a look at settings (for dev environments)

My general suggestion is to disable celery during development (CELERY_ALWAYS_EAGER=True in settings) and to use rabbitmq or redis for production environments.

2013/12/16 Andreas Røed notifications@github.com

Hi again! So I will actually need to set celery to use redis in settings.py to get it going? Any suggestions for an easy setup? Seems Ive got some research and testing to do,maybe.. I will eventually need something that works well on a production server though. Haven't really used this kind of stuff on previous projects.. Thanks for good and helpful answers anyways!

— Reply to this email directly or view it on GitHubhttps://github.com/tschellenbach/Feedly/issues/15#issuecomment-30703062 .

— Reply to this email directly or view it on GitHub.

lucianolev commented 10 years ago

Hi!

I'm having the same issue as the original post.

Here's my code

def test_a_like_of_a_guide_from_a_follower_is_registered_correclty_in_users_feed(self):
    a_follower = User(1)
    a_guide = Guide(1)
    a_follower_activity = Activity(a_follower, Like, a_guide)
    the_user = User(2)
    user_feed = RedisFeed(the_user)
    user_feed.add(a_follower_activity)
    registered_feed = user_feed[0]
    self.assertEquals(a_follower_activity, registered_feed)

User and Guide are just mock objects with an 'id' attribute (value passed on init). Like is a Verb subclass.

This is the traceback I got

Error
Traceback (most recent call last):
  File "/home/luciano/eryx/projects/pc_newsfeed/core/tests.py", line 32, in test_a_like_of_a_guide_from_a_follower_is_registered_correclty_in_users_feed
    registered_feed = user_feed[0]
  File "/home/luciano/eryx/projects/pc_newsfeed/env/local/lib/python2.7/site-packages/feedly/feeds/base.py", line 261, in __getitem__
    start, bound)
  File "/home/luciano/eryx/projects/pc_newsfeed/env/local/lib/python2.7/site-packages/feedly/feeds/base.py", line 304, in get_activity_slice
    activities = self.hydrate_activities(activities)
  File "/home/luciano/eryx/projects/pc_newsfeed/env/local/lib/python2.7/site-packages/feedly/feeds/base.py", line 285, in hydrate_activities
    return [activity.get_hydrated(activity_data) for activity in activities]
  File "/home/luciano/eryx/projects/pc_newsfeed/env/local/lib/python2.7/site-packages/feedly/activity.py", line 41, in get_hydrated
    activity = activities[int(self.serialization_id)]
KeyError: 13896234100000000000001001L

I'm just starting with Feedly, but it's difficult to understand what's going on.

Thanks!

tbarbugli commented 10 years ago

Hi, can you flush your redis storage and rerun your test ? perhaps some data leaked between tests and feedly fails because it cant find some activities in redis.

2014/1/13 lucianolev notifications@github.com

Hi!

I'm having the same issue as the original post.

Here's my code

def test_a_like_of_a_guide_from_a_follower_is_registered_correclty_in_users_feed(self): a_follower = User(1) a_guide = Guide(1) a_follower_activity = Activity(a_follower, Like, a_guide) the_user = User(2) user_feed = RedisFeed(the_user) user_feed.add(a_follower_activity) registered_feed = user_feed[0] self.assertEquals(a_follower_activity, registered_feed)

User and Guide are just mock objects with an 'id' attribute (value passed on init). Like is a Verb subclass.

This is the traceback I got

Error Traceback (most recent call last): File "/home/luciano/eryx/projects/pc_newsfeed/core/tests.py", line 32, in test_a_like_of_a_guide_from_a_follower_is_registered_correclty_in_users_feed registered_feed = user_feed[0] File "/home/luciano/eryx/projects/pc_newsfeed/env/local/lib/python2.7/site-packages/feedly/feeds/base.py", line 261, in getitem start, bound) File "/home/luciano/eryx/projects/pc_newsfeed/env/local/lib/python2.7/site-packages/feedly/feeds/base.py", line 304, in get_activity_slice activities = self.hydrate_activities(activities) File "/home/luciano/eryx/projects/pc_newsfeed/env/local/lib/python2.7/site-packages/feedly/feeds/base.py", line 285, in hydrate_activities return [activity.get_hydrated(activity_data) for activity in activities] File "/home/luciano/eryx/projects/pc_newsfeed/env/local/lib/python2.7/site-packages/feedly/activity.py", line 41, in get_hydrated activity = activities[int(self.serialization_id)] KeyError: 13896234100000000000001001L

I'm just starting with Feedly, but it's difficult to understand what's going on.

Thanks!

— Reply to this email directly or view it on GitHubhttps://github.com/tschellenbach/Feedly/issues/15#issuecomment-32173634 .

lucianolev commented 10 years ago

I did a 'FLUSHALL' in redis-cli and run the test again and it's still faling with the same error (different key though). I'm new to redis, so I'm not sure that that's the right command for flushing the redis storage.

tbarbugli commented 10 years ago

FLUSHALL clear everything stored in the redis server, looking at the error it seems that feedly is not able to find the activity data referenced in the feed.

Is the code in the test everything you run or do you have some setup code / more tests ?

If possible the output of redis-cli MONITOR during the test would help a lot tracking this issue down (redis-cli monitor will output every command issued to the redis machine)

2014/1/13 lucianolev notifications@github.com

I did a 'FLUSHALL' in redis-cli and run the test again and it's still faling with the same error (different key though). I'm new to redis, so I'm not sure that that's the right command for flushing the redis storage.

— Reply to this email directly or view it on GitHubhttps://github.com/tschellenbach/Feedly/issues/15#issuecomment-32175983 .

lucianolev commented 10 years ago

This is the whole test class (changing it a bit from before)

class FeedTest(TestCase):
    fixtures = ['testdata.json']

    def setUp(self):
        TestCase.setUp(self)

        self.bogus_user = get_user_model().objects.get(username='bogus')
        self.bogus_user2 = get_user_model().objects.get(username='bogus2')

    def test_a_like_of_a_guide_from_a_follower_is_registered_correclty_in_users_feed(self):
        a_follower = self.bogus_user
        a_guide = Guide(1)
        a_follower_activity = Activity(a_follower, Like, a_guide)

        the_user = self.bogus_user2
        user_feed = RedisFeed(the_user)
        user_feed.add(a_follower_activity)
        registered_feed = user_feed[0]
        self.assertEquals(a_follower_activity, registered_feed)

(From before, I've change the User mock for a real Django user)

I'm just running that only TestCase with Django test runner (PyCharm runner to be more precise), after the FLUSHALL command

redis monitor output:

1389625588.812962 [0 127.0.0.1:38126] "ZADD" "feed_bogus2" "13896255880000000000001005" "13896255880000000000001005"
1389625588.813516 [0 127.0.0.1:38126] "ZREVRANGE" "feed_bogus2" "0" "0" "withscores"
1389625588.814068 [0 127.0.0.1:38126] "HGET" "global:6" "13896255880000000000001005"

Thanks for your time

lucianolev commented 10 years ago

Just in case, the current traceback:

Error
Traceback (most recent call last):
  File "/home/luciano/eryx/projects/pc_newsfeed/core/tests.py", line 33, in test_a_like_of_a_guide_from_a_follower_is_registered_correclty_in_users_feed
    registered_feed = user_feed[0]
  File "/home/luciano/eryx/projects/pc_newsfeed/env/local/lib/python2.7/site-packages/feedly/feeds/base.py", line 261, in __getitem__
    start, bound)
  File "/home/luciano/eryx/projects/pc_newsfeed/env/local/lib/python2.7/site-packages/feedly/feeds/base.py", line 304, in get_activity_slice
    activities = self.hydrate_activities(activities)
  File "/home/luciano/eryx/projects/pc_newsfeed/env/local/lib/python2.7/site-packages/feedly/feeds/base.py", line 285, in hydrate_activities
    return [activity.get_hydrated(activity_data) for activity in activities]
  File "/home/luciano/eryx/projects/pc_newsfeed/env/local/lib/python2.7/site-packages/feedly/activity.py", line 41, in get_hydrated
    activity = activities[int(self.serialization_id)]
KeyError: 13896255880000000000001005L

tbarbugli commented 10 years ago

Please try to use this code (I added the line with user_feed.insert_activity)

def test_a_like_of_a_guide_from_a_follower_is_registered_correclty_in_users_feed(self):
    a_follower = self.bogus_user
    a_guide = Guide(1)
    a_follower_activity = Activity(a_follower, Like, a_guide)

    the_user = self.bogus_user2
    user_feed = RedisFeed(the_user)
    user_feed.insert_activity(a_follower_activity)
    user_feed.add(a_follower_activity)
    registered_feed = user_feed[0]
    self.assertEquals(a_follower_activity, registered_feed)

Feedly denormalised the data of activities (since 1 activity is in N feeds), when an activity is created you need to add that to the activity storage (with insert_activity) before you push it to a feed (which only stores the reference to the activity)

This job is normally done by a feed manager (eg. Feedly.add_user_activity in feedly.manager.base)

I suggest you to have a look at our example app for a simple and yet complete implementation of feedly (https://github.com/tbarbugli/feedly_pin)

2014/1/13 lucianolev notifications@github.com

Just in case, the current traceback:

Error Traceback (most recent call last): File "/home/luciano/eryx/projects/pc_newsfeed/core/tests.py", line 33, in test_a_like_of_a_guide_from_a_follower_is_registered_correclty_in_users_feed registered_feed = user_feed[0] File "/home/luciano/eryx/projects/pc_newsfeed/env/local/lib/python2.7/site-packages/feedly/feeds/base.py", line 261, in getitem start, bound) File "/home/luciano/eryx/projects/pc_newsfeed/env/local/lib/python2.7/site-packages/feedly/feeds/base.py", line 304, in get_activity_slice activities = self.hydrate_activities(activities) File "/home/luciano/eryx/projects/pc_newsfeed/env/local/lib/python2.7/site-packages/feedly/feeds/base.py", line 285, in hydrate_activities return [activity.get_hydrated(activity_data) for activity in activities] File "/home/luciano/eryx/projects/pc_newsfeed/env/local/lib/python2.7/site-packages/feedly/activity.py", line 41, in get_hydrated activity = activities[int(self.serialization_id)] KeyError: 13896255880000000000001005L

— Reply to this email directly or view it on GitHubhttps://github.com/tschellenbach/Feedly/issues/15#issuecomment-32177195 .

lucianolev commented 10 years ago

Thanks, it's working now!

I'm already looking at feedly_pin for an example :) but I wanted to do a simple test case without the feed manager, to understand how the app works. It seems unintuitive to me that I've to do an insert_activity before add, maybe you should consider leaving that responsibiliy to the add method. I'm just starting knowing this, so maybe you have a good motivation for doing it this way though.

Thanks again for your fast support