Open fadjar340 opened 4 years ago
Good day @fadjar340 ,
LoudML server supports Elasticsearch (self hosted and AWS) as a datasource https://github.com/regel/loudml/blob/develop/loudml/elastic.py
Let me add a parser into "Create Baseline" button to handle Elasticsearch on Grafana side.
Will keep you posted.
With best regards, Volodymyr
You rock @vsergeyev !!!
Fadjar340
You welcome @fadjar340 !
if it possible - my you add a note about index structure/fields you use, and query in Grafana datasource.
I just use the standard index structure from elasticsearch, the query will follow the index structure. It's possible to use all the index structure because a lots of metric, log and audit system that can be use and forecast. There several standard indices, such as filebeat, metricbeat, auditbeat, apm, heartbeat, winlogbeat, elastic use some common structure to build this indices, so I'll try to get the standard index structure in my system, then I'll send here... Those i mentioned is the common index that have by elasticsearch.
I'll send when it's ready...
Dear @vsergeyev ,
I attached the standard index structure of the elasticsearch. elasticsearch_index_structure.tar.gz
@fadjar340 cool, thank you for this data!
You're welcome @vsergeyev
@fadjar340 I try to follow this index structure. Looks good so far. If it is a chance you may share with me also 1-2 queries you use in Grafana panel. It may speedup my work.
Thank you in advance, V.
Committed a first version of Elasticsearch query parser in 48f274b1615115363b6b1da1d7cc0e660c0b0f26
Dear @vsergeyev ,
Below the sample query for elasticsearch from grafana:
request:Object
url:"api/datasources/proxy/10/_msearch"
method:"POST"
data:"{"search_type":"query_then_fetch","ignore_unavailable":true,"index":"logstash-*"} {"size":0,"query":{"bool":{"filter":[{"range":{"@timestamp":{"gte":1586624400000,"lte":1586706446964,"format":"epoch_millis"}}},{"query_string":{"analyze_wildcard":true,"query":"serialnum: (\"18b16902b65c\")"}}]}},"aggs":{"5":{"terms":{"field":"sonicwall.wan.ip","size":500,"order":{"_count":"desc"},"min_doc_count":1},"aggs":{"2":{"date_histogram":{"interval":"30s","field":"@timestamp","min_doc_count":0,"extended_bounds":{"min":1586624400000,"max":1586706446964},"format":"epoch_millis"},"aggs":{"1":{"avg":{"field":"sonicwall.source.bytes","script":{"inline":"_value*8"}}}}}}}}} "
request:Object
url:"api/datasources/proxy/21/_msearch?max_concurrent_shard_requests=5"
method:"POST"
data:"{"search_type":"query_then_fetch","ignore_unavailable":true,"index":"heartbeat-*"} {"size":0,"query":{"bool":{"filter":[{"range":{"@timestamp":{"gte":1586685034661,"lte":1586706634661,"format":"epoch_millis"}}},{"query_string":{"analyze_wildcard":true,"query":"monitor.name: food AND url.domain: 10.124.138.1"}}]}},"aggs":{"2":{"date_histogram":{"interval":"10s","field":"@timestamp","min_doc_count":0,"extended_bounds":{"min":1586685034661,"max":1586706634661},"format":"epoch_millis"},"aggs":{"1":{"max":{"field":"rtt.us"}}}}}} {"search_type":"query_then_fetch","ignore_unavailable":true,"index":"heartbeat-*"} {"size":0,"query":{"bool":{"filter":[{"range":{"@timestamp":{"gte":1586685034661,"lte":1586706634661,"format":"epoch_millis"}}},{"query_string":{"analyze_wildcard":true,"query":"monitor.name: food AND url.domain: 10.124.138.1"}}]}},"aggs":{"2":{"date_histogram":{"interval":"10s","field":"@timestamp","min_doc_count":0,"extended_bounds":{"min":1586685034661,"max":1586706634661},"format":"epoch_millis"},"aggs":{"1":{"avg":{"field":"rtt.us"}}}}}} {"search_type":"query_then_fetch","ignore_unavailable":true,"index":"heartbeat-*"} {"size":0,"query":{"bool":{"filter":[{"range":{"@timestamp":{"gte":1586685034661,"lte":1586706634661,"format":"epoch_millis"}}},{"query_string":{"analyze_wildcard":true,"query":"monitor.name: food AND url.domain: 10.124.138.1"}}]}},"aggs":{"2":{"date_histogram":{"interval":"10s","field":"@timestamp","min_doc_count":0,"extended_bounds":{"min":1586685034661,"max":1586706634661},"format":"epoch_millis"},"aggs":{"1":{"min":{"field":"rtt.us"}}}}}} "
Dear Sir,
Is it possible to use elasticsearch as datasource for loudml? Is there some example for this?
Regards, Fadjar340