search

vsergeyev / loudml-grafana-app

Visualization panel and datasource for Grafana to connect with Loud ML AI solution for ICT and IoT automation
https://vsergeyev.github.io/loudml-grafana-app/
MIT License
48 stars 20 forks source link

Elasticsearch Datasource for loudml-grafana #1

Open fadjar340 opened 4 years ago

fadjar340 commented 4 years ago

Dear Sir,

Is it possible to use elasticsearch as datasource for loudml? Is there some example for this?

Regards, Fadjar340

vsergeyev commented 4 years ago

Good day @fadjar340 ,

LoudML server supports Elasticsearch (self hosted and AWS) as a datasource https://github.com/regel/loudml/blob/develop/loudml/elastic.py

Let me add a parser into "Create Baseline" button to handle Elasticsearch on Grafana side.

Will keep you posted.

With best regards, Volodymyr

fadjar340 commented 4 years ago

You rock @vsergeyev !!!

Fadjar340

vsergeyev commented 4 years ago

You welcome @fadjar340 !

if it possible - my you add a note about index structure/fields you use, and query in Grafana datasource.

fadjar340 commented 4 years ago

I just use the standard index structure from elasticsearch, the query will follow the index structure. It's possible to use all the index structure because a lots of metric, log and audit system that can be use and forecast. There several standard indices, such as filebeat, metricbeat, auditbeat, apm, heartbeat, winlogbeat, elastic use some common structure to build this indices, so I'll try to get the standard index structure in my system, then I'll send here... Those i mentioned is the common index that have by elasticsearch.

I'll send when it's ready...

fadjar340 commented 4 years ago

Dear @vsergeyev ,

I attached the standard index structure of the elasticsearch. elasticsearch_index_structure.tar.gz

vsergeyev commented 4 years ago

@fadjar340 cool, thank you for this data!

fadjar340 commented 4 years ago

You're welcome @vsergeyev

vsergeyev commented 4 years ago

@fadjar340 I try to follow this index structure. Looks good so far. If it is a chance you may share with me also 1-2 queries you use in Grafana panel. It may speedup my work.

Thank you in advance, V.

vsergeyev commented 4 years ago

Committed a first version of Elasticsearch query parser in 48f274b1615115363b6b1da1d7cc0e660c0b0f26

fadjar340 commented 4 years ago

Dear @vsergeyev ,

Below the sample query for elasticsearch from grafana:

request:Object url:"api/datasources/proxy/10/_msearch" method:"POST" data:"{"search_type":"query_then_fetch","ignore_unavailable":true,"index":"logstash-*"} {"size":0,"query":{"bool":{"filter":[{"range":{"@timestamp":{"gte":1586624400000,"lte":1586706446964,"format":"epoch_millis"}}},{"query_string":{"analyze_wildcard":true,"query":"serialnum: (\"18b16902b65c\")"}}]}},"aggs":{"5":{"terms":{"field":"sonicwall.wan.ip","size":500,"order":{"_count":"desc"},"min_doc_count":1},"aggs":{"2":{"date_histogram":{"interval":"30s","field":"@timestamp","min_doc_count":0,"extended_bounds":{"min":1586624400000,"max":1586706446964},"format":"epoch_millis"},"aggs":{"1":{"avg":{"field":"sonicwall.source.bytes","script":{"inline":"_value*8"}}}}}}}}} "

request:Object url:"api/datasources/proxy/21/_msearch?max_concurrent_shard_requests=5" method:"POST" data:"{"search_type":"query_then_fetch","ignore_unavailable":true,"index":"heartbeat-*"} {"size":0,"query":{"bool":{"filter":[{"range":{"@timestamp":{"gte":1586685034661,"lte":1586706634661,"format":"epoch_millis"}}},{"query_string":{"analyze_wildcard":true,"query":"monitor.name: food AND url.domain: 10.124.138.1"}}]}},"aggs":{"2":{"date_histogram":{"interval":"10s","field":"@timestamp","min_doc_count":0,"extended_bounds":{"min":1586685034661,"max":1586706634661},"format":"epoch_millis"},"aggs":{"1":{"max":{"field":"rtt.us"}}}}}} {"search_type":"query_then_fetch","ignore_unavailable":true,"index":"heartbeat-*"} {"size":0,"query":{"bool":{"filter":[{"range":{"@timestamp":{"gte":1586685034661,"lte":1586706634661,"format":"epoch_millis"}}},{"query_string":{"analyze_wildcard":true,"query":"monitor.name: food AND url.domain: 10.124.138.1"}}]}},"aggs":{"2":{"date_histogram":{"interval":"10s","field":"@timestamp","min_doc_count":0,"extended_bounds":{"min":1586685034661,"max":1586706634661},"format":"epoch_millis"},"aggs":{"1":{"avg":{"field":"rtt.us"}}}}}} {"search_type":"query_then_fetch","ignore_unavailable":true,"index":"heartbeat-*"} {"size":0,"query":{"bool":{"filter":[{"range":{"@timestamp":{"gte":1586685034661,"lte":1586706634661,"format":"epoch_millis"}}},{"query_string":{"analyze_wildcard":true,"query":"monitor.name: food AND url.domain: 10.124.138.1"}}]}},"aggs":{"2":{"date_histogram":{"interval":"10s","field":"@timestamp","min_doc_count":0,"extended_bounds":{"min":1586685034661,"max":1586706634661},"format":"epoch_millis"},"aggs":{"1":{"min":{"field":"rtt.us"}}}}}} "