bad performance for top trends

Mango-information-systems / twitto_be

twitto_be is a real-time tweets analytics dashboard

https://twitto.be

Other

12 stars 2 forks source link

bad performance for top trends #117

Closed mef closed 7 years ago

mef commented 7 years ago

top hashtags / mentions take several seconds to get computed, before each visitor's page load.

refactoring is needed to fix this design flaw.

top trends should be fully computed at server start
top trends should be efficiently updated when a new tweet is extracted. This means that rather than re-computing all the data, only the relevant variables need to be changed. The whole collections of hashtags and mentions does not need to be recalculated / resorted all the time.

mef commented 7 years ago

cc @11digitlabs

PanosSynetos commented 7 years ago

The way I see it is for the server to calculate / sort and emit an event with only the necessary data everytime we have a new tweet.

Otherwise, we cannot do it on the client without the full set of data. Reasoning why:

Server calculates the top 10 on start
Emits just the top 10
Lets say all of the trends have a count of 100
New tweet comes in with hashtag and its not in the list of top 10
How does the client know that this new hashtag has count of 101 in order to include it in the top 10 and recalculate, or count 99 and discard it?

Comments, thoughts are welcome ;)

mef commented 7 years ago

After measuring the performance of current implementation, the poor performance is not caused by heavy computation on client, neither on server side, but by the volume of data transmitted to the client. For Belgium, objects with approx 5000 keys for hashtags, and 5000 keys for mentions is sent to the client.

On the client, it takes the following time to update the data:

incrementing stats: 0.98ms
sorting : 1.92ms
slicing : 0.83ms

As @11digitlabs suggested, stats should be updated on the server and only the top trends data sent to the client.