Kibana 3 and Graphite Integration

[vamsikish]

Hi,

1. I would like to know if Kibana 3 allows us to query/display/graph data in Graphite (whisper/ceres). According to graphite docs, it is possible to retrieve data in JSON format [ http://graphite.readthedocs.org/en/latest/render_api.html ].
2. Is it possible to use Kibana 3 if the data is stored in other database other than Elastic search? I would also like to know usecases where the scripted dashboards would be useful and do we have detailed documentation.

Thanks in advance

[ruckc]

So, i've been trying to figure out the use case for this. Kibana is better at looking at sets of records with multiple fields. Graphite stores numerical values over time. To me, these seem like the data sets stored in ES vs Graphite would be incompatible except on Temporal values. Can you provide a mockup of what you would expect the combination to do?

Curtis Ruck Anytime: 210-857-1126

On Fri, Nov 22, 2013 at 7:05 AM, vamsikish notifications@github.com wrote:

Hi, I would like to know if Kibana 3 allows us to query/display/graph data in Graphite (whisper/ceres). According to graphite docs, it is possible to retrieve data in JSON format [ http://graphite.readthedocs.org/en/latest/render_api.html ]. Also, Is it possible to use Kibana 3 if the data is stored in other database other than Elastic search? I would also like to know usecases where the scripted dashboards would be useful and do we have detailed documentation on this,

— Reply to this email directly or view it on GitHubhttps://github.com/elasticsearch/kibana/issues/720 .

[stuart-warren]

I could imagine graphing some specified graphite/opentsdb metrics and overlaying some (filtered/queried) vertical lines from elasticsearch for events.

That way you could potentially see a spike in a metric and an event, eg: processor load increasing shortly after a log message saying deploying new version of software, etc.

[vamsikish]

Hi, thanks for the reply, As Stuart mentioned, the idea is to drill down to the source of problem where I would like to view both the logs and statistics in a single UI. Currently, we view the performance statistics on graphite dashboard and logs on kibana3. Ideally we can write our custom UI querying both the Elastic search and graphite, and request the information in JSON format and chart/graph it. However, current Kibana 3 (milestone 4 or pre released milestone 5) already is an aesthetic dashboard, and supports maps/charts/graphs. What i currently miss with kiabana 3 is the ability to query/display data in graphite. Are there any guidelines on how to work around with kibana 3? I would also like to understand how to use the scripted dashboards feature in kibana 3 (there was an example logstash.js where the output is a dashboard)

[mcallaway]

I'm also interested in using Kibana 3 to view graphite data. But I'll go farther. I don't like how graphite forces me to define hierarchy of data points foo.bar.baz.metric = 1234. I want to define a metric and a value together with arbitrary tags: metric = { value: 1234, tag1: foo, tag2: bar, tag3: baz }. I'd expect to store the values/metrics in elastic search and build the time series graphs via Kibana 3. In this way, "Show me the value over time for metrics matching this arbitrary search pattern." In that way I don't have to pre-define a hierarchy of metrics, I can collect them with whatever I know about them, and then build the graphs later, based on what I query.

On Fri, Nov 22, 2013 at 10:56 PM, vamsikish notifications@github.comwrote:

Hi, thanks for the reply, As Stuart mentioned, the idea is to drill down to the source of problem where I would like to view both the logs and statistics in a single UI. Currently, we view the performance statistics on graphite dashboard and logs on kibana3. Ideally we can write our custom UI querying both the Elastic search and graphite, and request the information in JSON format and chart/graph it. However, current Kibana 3 (milestone 4 or pre released milestone 5) already is an aesthetic dashboard, and supports maps/charts/graphs. What i currently miss with kiabana 3 is the ability to query/display data in graphite. Are there any guidelines on how to work around with kibana 3? I would also like to understand how to use the scripted dashboards feature in kibana 3 (there was an example logstash.js where the output is a dashboard)

— Reply to this email directly or view it on GitHubhttps://github.com/elasticsearch/kibana/issues/720#issuecomment-29125900 .

[ruckc]

So, that can already be done with elasticsearch. A histogram with a query on a tag, and sum the values or count the unique events. The issue is that Graphite's data format is extremely compact for key/value compared to elasticsearch, and if you had 500 independent metrics for a server, 100 servers, and your value generation rate is every 60 seconds you end up with about 70 million records per day. In graphite this would be stored in 50,000 independent whisper datafiles, and query/graph would be a linear operation. In elasticsearch, its a log n operation due to the index.

Curtis Ruck Anytime: 210-857-1126

On Sat, Nov 23, 2013 at 4:18 PM, Matt Callaway notifications@github.comwrote:

I'm also interested in using Kibana 3 to view graphite data. But I'll go farther. I don't like how graphite forces me to define hierarchy of data points foo.bar.baz.metric = 1234. I want to define a metric and a value together with arbitrary tags: metric = { value: 1234, tag1: foo, tag2: bar, tag3: baz }. I'd expect to store the values/metrics in elastic search and build the time series graphs via Kibana 3. In this way, "Show me the value over time for metrics matching this arbitrary search pattern." In that way I don't have to pre-define a hierarchy of metrics, I can collect them with whatever I know about them, and then build the graphs later, based on what I query.

On Fri, Nov 22, 2013 at 10:56 PM, vamsikish notifications@github.comwrote:

Hi, thanks for the reply, As Stuart mentioned, the idea is to drill down to the source of problem where I would like to view both the logs and statistics in a single UI. Currently, we view the performance statistics on graphite dashboard and logs on kibana3. Ideally we can write our custom UI querying both the Elastic search and graphite, and request the information in JSON format and chart/graph it. However, current Kibana 3 (milestone 4 or pre released milestone 5) already is an aesthetic dashboard, and supports maps/charts/graphs. What i currently miss with kiabana 3 is the ability to query/display data in graphite. Are there any guidelines on how to work around with kibana 3? I would also like to understand how to use the scripted dashboards feature in kibana 3 (there was an example logstash.js where the output is a dashboard)

— Reply to this email directly or view it on GitHub< https://github.com/elasticsearch/kibana/issues/720#issuecomment-29125900> .

— Reply to this email directly or view it on GitHubhttps://github.com/elasticsearch/kibana/issues/720#issuecomment-29141807 .

[vamsikish]

Is it possible to use Kibana 3 if the data is stored in other database other than Elastic search? I would also like to know usecases on usage of scripted dashboards feature with some documentation. Is it possible to reuse Kibana3 to make a custom UI? Do you suggest, we store metrics in elastic search directly compared to storing it in Graphite/openTSDB (Does this solution allow scalability)? I am referring to Log management system described here , In this architecture, both kibana and graphite are used, I would like to know how could we integrate two different UIs? Thanks in advance.

[daithi-coombes]

I'm also interested in documentation on scripted dashboards etc, so far all I can find are these two methods: https://github.com/elasticsearch/kibana/blob/v3.0.0milestone4/src/app/dashboards/logstash.js https://github.com/elasticsearch/kibana/blob/v3.0.0milestone4/src/app/dashboards/logstash.json

but no further information

[Dieterbe]

have a look at http://vimeo.github.io/graph-explorer/ . some of its concepts are similar to kibana's (structures your metrics, aggregation methods, pattern search), but made as a dashboard for graphite.

it uses https://github.com/vimeo/timeserieswidget/ for rendering graphs and supports drawing events from ES as an overlay on the graphs. (although admittedly the rendering and positioning of the ES events is broken right now)

(disclaimer: i'm the main developer behind projects)

[avnivamsikrishna]

Hi, I have actually checked graph explorer (GE), It uses ES to just store tags, and enable faster searching (log complexity instead of linear). To be honest, i did not like GE in terms of visual impact though its currently a good graphite dashboard in terms of features, I am looking at https://github.com/kenhub/giraffe, as all i need is an aesthetic dashboard which is easily configurable and something that looks/feels like kibana3.

Out of context, regarding using the scripted dashboard feature in Kibana3, could you please share any tutorial where you demo usage of scripted dashboards (possibly as a video in Youtube/Dailymotion), currently am storing Dashboard JSON to ES. This becomes cumbersome especially if we have to create a dashboard for every instance.

[rashidkpc]

There is another graphite ticket here: #37