Graph-like network visualization

[KaranMuthusamy]

Hi, Is it possible to do dynamic network visualisation(graph, like Gephi), pulling data from ES, at least in the future version? or any idea of adding this graph visualisation? If not can someone suggest what is the best tool to do network visualisation pulling data directly from ES without any GraphDB? and I not sure this Git project will help(https://github.com/andyburgin/es2gefx), since there is no update in the past 2 years. Regards

[rashidkpc]

Can you give some examples? Maybe some screenshots? I'm inclined to say this is not really possible as elasticsearch is not a graphDB

[KaranMuthusamy]

yes, you are right, ES is not graphDB and here are few links about network visualisation..

https://www.youtube.com/watch?v=6rURezjoEDo https://www.youtube.com/watch?v=kbLFMObmLNQ https://www.youtube.com/watch?v=yZ0G9jljCto https://www.youtube.com/watch?v=7LMnpM0p4cM

[rashidkpc]

It really would be helpful for you to describe your case in detail and provide screenshots of what you'd like to accomplish in addition to linking to youtube. The more information you can insert in the ticket the easier it is for us to understand your case.

I'd recommend describing what "Dynamic network visualization" is and what problems it could solve. You should also attach an elasticsearch query that could generate the data you're looking to visualize.

[KaranMuthusamy]

ok, Here is the use case, We have mobile app clickstream data on ES and each and every events(clicks) are collected on the Elasticsearch. Now we want to see only few events in the form of state diagram. There are more events that we do not care.

The specific events are definite state such as login, play, purchase and logout with timestamp. In the above 4 events(state) for a particular user how many times he logged and played and logged out, on this example below one user after logged in he played 7 times and logged out 5 times and 2 times purchased after playing and so on., based on the time scale in the bottom based on the timestamp of those events.

Also I want to see this diagram for all the users at the same time. I hope this helps, Please let me know if you have more questions. Thanks

[rashidkpc]

Yeah, I can't think of a way to express this as an elasticsearch aggregation

[rcrezende]

I can see two graphical models, but both would require special field encoding.

presentation=Sankey http://en.wikipedia.org/wiki/Sankey_diagram: Field type="enumerated edge" Sample values= "0,A", "1,A,B", "1,A,C","2,B,D".

presentation=State Diagram http://en.wikipedia.org/wiki/State_diagram Field Type="non-enumerated edge" Sample values: "A", "A,B", "A,C", "B,D"

A simple aggr using count would rank top edges and so would be up to the front-end decode the field type and render the graph accordingly.

I feel that is conceptually similar to the pair (type=geohash, presentation=tiles).

[markharwood]

Update: I'm working on the official elasticsearch graph plugin that aims to identify strongly connected terms in docs. I've been testing on a lot of datasets and the closest to your example is click data. Below is a prototype visualization that shows the connections between search terms and clicked products, for queries containing the word "mixer".

Each "click" document in my index has to contain the terms that need to be connected e.g. query->clickedProduct. These in aggregate are used to strengthen the connections between query/product pairings. In your case the docs would need to contain currentAction and previousAction in order to draw the graph you outline. Each term or term pairing can of course be queried simply from your client to provide additional aggregations such as date histograms, gender breakdowns, max-time-spent-between-actions etc etc When we release the graph API (scheduled for 2.2) we hope to include some elements of UI support (cleaner than my prototype screenshot!)

[elvarb]

The Kibana3 Packetbeats dashboard had one new panel that could be used as a base for this

https://www.elastic.co/guide/en/beats/packetbeat/current/images/topology_map.png

Would love a Sankey visualization, would fit so well with buckets

[kevross33]

Hi,

What is the status of official elasticsearch graph plugin? I have an interest in it as I can see it potentially being useful in a security analysis. Also a graph plugin in kibana even where you could specify data links would be useful i.e if I had some data such as web traffic logs from Bro-IDS in elasticsearch being able to then search for certain information and then create the links based on chosen fields i.e website, URI and referer making it possible to create something like the following for web exploit analysis:

Google Search > Website (and also graphing the rest of this website) > redirect > exploit kit landing, exploit, payload.

Obviously this sort of thing could be useful in lots of areas but given I use this for security logs it would be very useful to apply analysis to these kinds of cases to derive more intelligence and links.

[pykler]

@markharwood when will you release this awesome plugin #elasticon

[markharwood]

Shooting for the 2.3 release of elastic stack.

[rodrigovanini]

@kevross33 : I might be wrong but I am pretty sure there was a sample usecase for security analysis in the (awesome) presentation @markharwood gave at ElasticON last week

[synhershko]

@markharwood where can we find / how can we play with your work on the new graph API?

[markharwood]

@synhershko I'm packaging things like docs up now ready for a 2.3 release. There was a whole session at elasticon on the new graph capabilities. The full video from that isn't up yet but there was a taster in the Keynote presentation around the 1:08 mark here: https://www.elastic.co/elasticon/conf/2016/sf/opening-keynote

[speedplane]

Very interested to see the new graph api in action. My use-case: I have millions of text documents in ES that cite each other, and I would like to visually explore this citation network at scale.

[thomasneirynck]

this 3rd party plugin might be useful: https://github.com/dlumbrer/kbn_network

cc @ppisljar

[timductive]

Closing this because it's not planned to be resolved in the foreseeable future. It will be tracked in our Icebox and will be re-opened if our priorities change. Feel free to re-open if you think it should be melted sooner.