Wrong insertion into followers feed?

[andrenro]

Hi! I manage to get the get_user_feed() work properly, but when I try to fetch followers feeds, something strange happens. I think it might be an issue connected to fanout/insertion into feedly: Ive used the pin_feedly example as a starting point.

Here is some code: models.py:

Test implementation of feedly:

class WishFeed(RedisFeed): key_format = 'feed:normal:%(user_id)s'

class UserWishFeed(WishFeed): key_format = 'feed:user:%(user_id)s'

class UserWishFeedly(Feedly): feed_classes = dict( normal=WishFeed, )

user_feed_class = UserWishFeed

def add_wish(self,user,activity):
    # add user activity adds it to the user feed, and starts the fanout
    self.add_user_activity(user.id, activity)

def get_user_follower_ids(self, user_id):
    from email_auth.models import GiftItEndUser
    friends = Friend.objects.filter(Q(from_user=user_id) | Q(to_user=user_id))
    f_ids = []
    if friends.exists():
        for f in friends:
            if f.to_user.pk != user_id:
                f_ids.append(f.to_user.pk)
            if f.from_user.pk != user_id:
                f_ids.append(f.from_user.pk)
    ids = GiftItEndUser.objects.filter(id__in=f_ids).values_list('id',flat=True)
    return {FanoutPriority.HIGH:ids}

views.py @csrf_exempt @api_view(['GET']) @login_required def friends_wish_feed(request, _args, *_kwargs): feed = feedly.get_feeds(request.user.id)['normal'] act_list = [] json_data = [] for acts in feed[:25]: act_list.append(acts) json_data = json.dumps(act_list) return Response(json_data)

this is how I insert into the feeds: views.py ... activity = Activity(wishlist.user,WishVerb,in_prod.id) feed = UserWishFeed(wishlist.user.id) feed.add(activity) feedly.add_wish(wishlist.user, activity) ...

from redis-cli after insertion i will get this: 127.0.0.1:6379> keys * 1) "global:3" 2) "global:4" 3) "feed:user:7" 4) "global:7" 5) "global:9"

if i query 3) : 127.0.0.1:6379> zrange "feed:user:7" 0 1 1) "13873587370000000000003005" 2) "13873587410000000000001005"

127.0.0.1:6379> hgetall "global:3" 1) "13873587470000000000006005" 2) "7,5,6,0,1387358747,"

Seems like the insertion went fine, except that it did not define the correct key_format? How can I fix this?

Why does it say "global:" instead of "user:normal:" like I defined in WishFeed(RedisFeed) ?

The function friends_wish_feed() returns [] every time..

Help appreciated!! :)

[tschellenbach]

Hi !

The global name is used for the storing the activity. (mapping the id to the activity) (For which we should probably really find a way to make it smaller) (@tbarbugli, ideas on this?)

So its normal for data to end up in global. From your setup i would expect: 1 in feed:user 1 in global number of followers in the feed:normal

The feedly setup looks ok. I see two possible reasons why there is nothing in feed:normal

A.) the list of follower ids returned is empty B.) the celery tasks generated by create fanout tasks don't get processed

(You could try enabling CELERY_ALWAYS_EAGER = True CELERY_EAGER_PROPAGATES_EXCEPTIONS = True) Dont use these settings in production though!

What sort of application are you building?

Best of luck!

[andrenro]

Hi!:)

Im building a social gift-planner/wishlist-app! Im trying to integrate feedly so that I can make REST-based feeds on the form "User1 added product1 to wishlist1" :) So what i would need is that other users (User1's friends) can receive this kind of feed. I would guess possibility B) , but I've tried with those settings, nothing happened..Any other possible solutions?

Current settings.py: FEEDLY_NYDUS_CONFIG = { 'CONNECTIONS': { 'redis': { 'engine': 'nydus.db.backends.redis.Redis', 'router': 'nydus.db.routers.redis.PrefixPartitionRouter', 'hosts': { 0: {'prefix': 'default', 'db': 0, 'host': '127.0.0.1', 'port': 6379}, } }, } }

BROKER_URL = 'redis://127.0.0.1:6379/0' CELERY_RESULT_BACKEND = 'redis://localhost:6379/0' CELERY_ALWAYS_EAGER = True CELERY_EAGER_PROPAGATES_EXCEPTIONS = True

I will check get_user_follower_ids() for emptiness (I am pretty certain that the friend-relationships exists in the database though). I would need something that works in production also :)

[tschellenbach]

Sounds cool,

Celery is a great for running async tasks, but it can be a bit of a learning curve. The docs are here: http://www.celeryproject.org/

Try to verify that the code reaches UserWishFeedly.fanout Could you dump the local variables you get when going into UserWishFeedly.fanout?

[andrenro]

I think there might be a problem with my redis setup as well, I've installed rabbitMQ and Redis but it seems that the settings for redis/celery does nothing . If i stop the rabbitMQ (rabbitmqctl stop) i will get an "error [111] connection refused" when I try to add something to the feeds. My impression was that Redis was now my backend, and that I do not need to have rabbitMQ server running? Im confused.. Celery uses EITHER rabbitMQ or Redis as its task broker, right?

On Wed, Dec 18, 2013 at 12:33 PM, Thierry Schellenbach < notifications@github.com> wrote:

Sounds cool,

Celery is a great for running async tasks, but it can be a bit of a learning curve. The docs are here: http://www.celeryproject.org/

Try to verify that the code reaches UserWishFeedly.fanout Could you dump the local variables you get when going into UserWishFeedly.fanout?

— Reply to this email directly or view it on GitHubhttps://github.com/tschellenbach/Feedly/issues/16#issuecomment-30834733 .

Mvh Andreas Røed

[tbarbugli]

when CELERY_ALWAYS_EAGER is set to True tasks are executed in the same thread and celery does not connect to the task broker. I think that your celery settings are ignored / unused; you can find docs about this here https://docs.celeryproject.org/en/latest/django/first-steps-with-django.html

2013/12/18 Andreas Røed notifications@github.com

I think there might be a problem with my redis setup as well, I've installed rabbitMQ and Redis but it seems that the settings for redis/celery does nothing . If i stop the rabbitMQ (rabbitmqctl stop) i will get an "error [111] connection refused" when I try to add something to the feeds. My impression was that Redis was now my backend, and that I do not need to have rabbitMQ server running? Im confused.. Celery uses EITHER rabbitMQ or Redis as its task broker, right?

On Wed, Dec 18, 2013 at 12:33 PM, Thierry Schellenbach < notifications@github.com> wrote:

Sounds cool,

Celery is a great for running async tasks, but it can be a bit of a learning curve. The docs are here: http://www.celeryproject.org/

Try to verify that the code reaches UserWishFeedly.fanout Could you dump the local variables you get when going into UserWishFeedly.fanout?

— Reply to this email directly or view it on GitHub< https://github.com/tschellenbach/Feedly/issues/16#issuecomment-30834733> .

Mvh Andreas Røed

— Reply to this email directly or view it on GitHubhttps://github.com/tschellenbach/Feedly/issues/16#issuecomment-30837484 .

[andrenro]

So I got it working correctly when using CELERY_ALWAYS_EAGER = True ! :) Only thing left is to make the broker settings work correctly! I don't get any errors when CELERY_ALWAYS_EAGER = False, it just doesn't fetch the newest feeds properly..

On Wed, Dec 18, 2013 at 1:49 PM, Tommaso Barbugli notifications@github.comwrote:

when CELERY_ALWAYS_EAGER is set to True tasks are executed in the same thread and celery does not connect to the task broker. I think that your celery settings are ignored / unused; you can find docs about this here

https://docs.celeryproject.org/en/latest/django/first-steps-with-django.html

2013/12/18 Andreas Røed notifications@github.com

I think there might be a problem with my redis setup as well, I've installed rabbitMQ and Redis but it seems that the settings for redis/celery does nothing . If i stop the rabbitMQ (rabbitmqctl stop) i will get an "error [111] connection refused" when I try to add something to the feeds. My impression was that Redis was now my backend, and that I do not need to have rabbitMQ server running? Im confused.. Celery uses EITHER rabbitMQ or Redis as its task broker, right?

On Wed, Dec 18, 2013 at 12:33 PM, Thierry Schellenbach < notifications@github.com> wrote:

Sounds cool,

Celery is a great for running async tasks, but it can be a bit of a learning curve. The docs are here: http://www.celeryproject.org/

Try to verify that the code reaches UserWishFeedly.fanout Could you dump the local variables you get when going into UserWishFeedly.fanout?

— Reply to this email directly or view it on GitHub< https://github.com/tschellenbach/Feedly/issues/16#issuecomment-30834733>

.

Mvh Andreas Røed

— Reply to this email directly or view it on GitHub< https://github.com/tschellenbach/Feedly/issues/16#issuecomment-30837484> .

— Reply to this email directly or view it on GitHubhttps://github.com/tschellenbach/Feedly/issues/16#issuecomment-30838903 .

Mvh Andreas Røed

[tbarbugli]

Thats because tasks are waiting in the broker to be consumed by a celery worker. You need to start a celery worker for that ;) On 18 Dec 2013 14:24, "Andreas Røed" notifications@github.com wrote:

So I got it working correctly when using CELERY_ALWAYS_EAGER = True ! :) Only thing left is to make the broker settings work correctly! I don't get any errors when CELERY_ALWAYS_EAGER = False, it just doesn't fetch the newest feeds properly..

On Wed, Dec 18, 2013 at 1:49 PM, Tommaso Barbugli notifications@github.comwrote:

when CELERY_ALWAYS_EAGER is set to True tasks are executed in the same thread and celery does not connect to the task broker. I think that your celery settings are ignored / unused; you can find docs about this here

https://docs.celeryproject.org/en/latest/django/first-steps-with-django.html

2013/12/18 Andreas Røed notifications@github.com

I think there might be a problem with my redis setup as well, I've installed rabbitMQ and Redis but it seems that the settings for redis/celery does nothing . If i stop the rabbitMQ (rabbitmqctl stop) i will get an "error [111] connection refused" when I try to add something to the feeds. My impression was that Redis was now my backend, and that I do not need to have rabbitMQ server running? Im confused.. Celery uses EITHER rabbitMQ or Redis as its task broker, right?

On Wed, Dec 18, 2013 at 12:33 PM, Thierry Schellenbach < notifications@github.com> wrote:

Sounds cool,

Celery is a great for running async tasks, but it can be a bit of a learning curve. The docs are here: http://www.celeryproject.org/

Try to verify that the code reaches UserWishFeedly.fanout Could you dump the local variables you get when going into UserWishFeedly.fanout?

— Reply to this email directly or view it on GitHub<

https://github.com/tschellenbach/Feedly/issues/16#issuecomment-30834733>

.

Mvh Andreas Røed

— Reply to this email directly or view it on GitHub< https://github.com/tschellenbach/Feedly/issues/16#issuecomment-30837484>

.

— Reply to this email directly or view it on GitHub< https://github.com/tschellenbach/Feedly/issues/16#issuecomment-30838903> .

Mvh Andreas Røed

— Reply to this email directly or view it on GitHubhttps://github.com/tschellenbach/Feedly/issues/16#issuecomment-30840972 .

[thedrow]

So this should be closed?