Top hashtags and top mentions charts

[mef]

Display a set of metrics / charts related to the tweets:

• [x] top hashtags
• [x] top mentions

Remaining from #85

[PanosSynetos]

Hey @mef . I've started with the implementation. Here is what i'm doing

• Whenever a new tweet arrives on the server, im making an array of hashtags/mentions
• Inside the model, I'm looping through all tweets and creating two arrays that hold all the hashtags/mentions
• I reduce them {hashtag: counts}
• Sort the object based on counts

This is happening in the beggining. My idea was to send the sorted objects to the client and on a new tweet, update that array. I dont know if this is efficient.

a: 5
b: 4
c: 3

Lets say a new tag b arrives. Loop through all hashtags, find property b and ++ the counter, resort the array objects . If we don't find the property b just push it to the end.

Do you think that this is efficient to be done on the client side? Should we do the calculation on the server, everytime there is a new tweet? What do you think?

[mef]

Here are some suggestions:

• call the variable entitiesStats or entitiesCount, instea of topStats
• rename the event eventTopStats accordingly.
• store the counts in an object, using the entity as a key, and count as value, instead of an array of arrays:

entitiesCount.hashtags = {
    belgium: 55
    , toto: 32
    ...
}

• In a separate variable, keep the top n entities, and the lowest count present in this ranking:

topEntities = {
    hashtags: [
        // key,value again, only for the top n.
    ]
    , lowestHashtagCountCount: 10
    ...
}

• When a new tweet is received, increment its entities inside entitiesCount variable. Only for these entities included in the new tweet, check whether their count has gotten higher than the known lowest count. If so, update topEntities entities array accordingly.
• proceed as done in tweetStats regarding the client side / server side repartition. You can even go further and create a metrics controller, to be used at both sides to update the stats.
• do not forget to reactivate updateTweetStats call, for some reason it has been commented out in the previous commit.

Hope this helps,

[PanosSynetos]

Thanks a lot. The reason for the array of arrays, was that I couldn't sort the object. Do you have any ideas for that?

[mef]

You are going to have a very long tail of hashtags that are here only once. This is why you want to avoid to have to iterate through this array, and also you don't want to sort it.

• reduce the array once to get the top n entities, and store the result inside the topEntities.hashtags array (which is sorted).
• then whenever there is a new tweet, you don't need to iterate through the counts. Just increment the count of the tweet's entities, compare these counts to what's inside topEntities to know whether this entity should be added to the topEntities (replacing another one)
• also think of updating the entities counts when the old tweets are removed...

[PanosSynetos]

I'll give it a try, thanks

[mef]

You don't need to sort the object. Inside a reduce function, just identify whether the element's count is higher than previous elements, and keep the top n records.

On Tue, Sep 20, 2016 at 2:19 PM, 11 Digit Labs notifications@github.com wrote:

Thanks a lot. The reason for the array of arrays, was that I couldn't sort the object. Do you have any ideas for that?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Mango-information-systems/twitto_be/issues/100#issuecomment-248298925, or mute the thread https://github.com/notifications/unsubscribe-auth/AA4fAzk2tFDdUAwXzQ45MiZWmap63WIzks5qr916gaJpZM4J_X55 .

[mef]

Does not seem to be working with an empty database:

/home/c-user/development/twitto_be/model/tweets.js:197
        return {'top': arr, 'all': obj, 'lowestCount': arr[slice-1].value}
                                                                   ^

TypeError: Cannot read property 'value' of undefined
    at orderSliceObject (/home/c-user/development/twitto_be/model/tweets.js:197:62)
    at calculateEntitiesStats (/home/c-user/development/twitto_be/model/tweets.js:136:14)
    at new Tweets (/home/c-user/development/twitto_be/model/tweets.js:37:2)
    at new Storage (/home/c-user/development/twitto_be/model/index.js:24:16)
    at Object.<anonymous> (/home/c-user/development/twitto_be/app.js:18:16)
    at Module._compile (module.js:409:26)
    at Object.Module._extensions..js (module.js:416:10)
    at Module.load (module.js:343:32)
    at Function.Module._load (module.js:300:12)
    at Function.Module.runMain (module.js:441:10)
[nodemon] app crashed - waiting for file changes before starting...

[PanosSynetos]

Just like @mef suggested, no arrays, no sorting and slicing \m/ (still WIP!)

[mef]

2.2.0-pre has just been deployed in the production server, data will be usable in 24h :smile: .

[mef]

@11digitlabs I think I've fixed the various small things that remained, please have a look.

Apart from that, I thought the refactoring removing the sort of all hashtags, using reduce to pick the top 10, had been done, but it looks like it's not in place. Has it been done in some other branch? Or perhaps I misunderstood?

[PanosSynetos]

Nope, no misunderstanding. Seems like I didnt do it, if you want, please do the reduce, you JS master :) Will check the rest later tonight

[mef]

What do you think of this alternative presentation?

Trending hashtags / mentions is more relevant to users, so pushing it to the top seems like a good move to me:

[PanosSynetos]

Makes more sense, plus its like grouped by color :) Grey ones on top Green ones at the bottom

Should we make the hastags/mentions clickable too?

[PanosSynetos]

A bug I just noticed, couldnt resist to download the latest version, im a junkie ;)​

http://take.ms/xaJBL

[mef]

A bug I just noticed, couldnt resist to download the latest version, im a junkie ;)​

Yes, reported in #101

[mef]

@11digitlabs the features looks ready for deployment to me. What do you think?

[PanosSynetos]

Alright, lets go for it. Seems to be working/looking fine