tags / mentions graph + realtime effect

mef commented 5 years ago

Experiment with the implementation of the following solution:

Replace the top tags and mentions bar charts by a force-directed graph
include the graph visual (image) to the tweetbot contents
display the realtime tweets (e.g. keep the last n tweets received in screen)

mef commented 5 years ago

hashtags + mentions graph

Design notes - work in progress.

graphology may be used to manage the graph data.

ETL changes (tweetStream)

When receiving a tweet from the API:

store node with mergeNode
store edge with either addEdge, or updateEdgeAttribute to increment weight - depending on whether the edge already exists (reference).

subgraph generation

The subgraph to be sent to the client must be limited to the n top nodes (perhaps using their HITS authority measure). This subgraph does not need to be maintained in real time, perhaps one update per minute is relevant.

Todo: validate that HITS can be executed with a decent performance on the full graph. Alternative: use node degree...

Update 2020-01-07: top n nodes are determined using degree measure. HITS is not needed, since the graph is undirected.

data expiration

Nodes must have a last update timestamp attribute, to be updated each time mergeNode is called. This attribute is to be used in order to clear records older than 24h. Good news is that dropNode also removes "all its attached edges from the graph", so it won't be necessary to iterate to the edges, in addition to the nodes.

Update 2020-01-07: last update timestamp has been removed from node attributes. Cleanup is done based on the timestamp stored in the tweets array, this allows to decrement edges weights in addition to updating node counts.

mef commented 4 years ago

Remains to do for the graph display:

[x] check that FA2 layout is working as expected
[x] run FA2 on the whole graph, rather than on the subgraph
[x] make sure that FA2 positions are not reset at each iteration (might be solved by previous item)
[x] debounce filterGraph function (c.f. just-debounce)
[x] compute modularity
[x] color graph according to modularity class
[x] make it possible to pan and zoom on the graph
[x] update layout: 4/5 column for the graph, 1/5 column for the other charts
[x] ~make it possible to conditionally activate console.time (should be only active for perf debug logging).~
[x] check app behaviour under heavy load
[x] disconnect from socket data feed whenever the tab is not active (using page visibility api and a socket.io room)
[x] display a legend for the graph colors
[x] add a 'pause' feature during which graph is updated so that no label overlaps
[ ] add an 'export graph' feature
[ ] add a button to switch tracked topic
[ ] add an option to show the graph full screen (c.f. code sample below, and css config)

<button onclick="document.querySelector('#wikiArticle').mozRequestFullScreen()">Toggle</button>

mef commented 4 years ago

implemented in v4.0.0

Mango-information-systems / twitto_be