ElasticSearch crashes when running rule

[TheAnimal12345]

Hello,

Elasticsearch keeps crashing whenever I start up Elastalert. I have a feeling it has something to do with one of the rules I made. This is the rule:

name: Metricbeat CPU Spike Rule
type: any

#es_host: localhost
#es_port: 9200

index: metricbeat-*

buffer_time:
  hours: 1

metric_agg_key: system.cpu.user.pct
metric_agg_type: avg
query_key: beat.hostname
doc_type: metricsets

bucket_interval:
  minutes: 5

sync_bucket_interval: true
#allow_buffer_time_overlap: true
#use_run_every_query_size: true

min_threshold: 0.01
max_threshold: 0.1

filter:
- term:
    metricset.name: cpu

# (Required)
# The alert is use when a match is found
alert:
- "debug"
- "slack"

slack:

Any help is appreciated. I left out the slack part.

[Qmando]

Please give the output of the crash. You should get a traceback or message.

edit: oh, Elasticsearch itself is crashing?

[TheAnimal12345]

@Qmando Yes, the actual service crashes. I always have to manually start it up again.

[Qmando]

Hmm, you'll definitely need to check the Elasticsearch logs, they should have some useful information. I've managed to crash Elasticsearch a few times when we ask it to perform an aggregation it can't fit into memory, no idea if that's the issue here. You can use --es_debug_trace trace.log to get the raw queries being made to elasicsearch, that might help you debug.

[dungkma]

@TheAnimal12345 i have same issues. How did you fix it ?

[dungkma]

@Qmando When i run elastalert rule, the elasticsearch service was dead (failed). My version ELK is 6.2.2. Please help me fix it. Thanks you!

[Qmando]

Did you check the Elasticsearch logs?

[TheAnimal12345]

@dungkma What worked for me was deleting the ElastAlert index and creating a new one. Mine was somehow corrupt. The same thing happened with Kibana when I was using it. Kibana would not load and I tried everything to fix it and no luck. Finally deleted the Kibana index and created a new one and everything started running smoothly.

[dungkma]

@Qmando Elasticsearch logs: [2018-08-22T00:01:04,228][DEBUG][o.e.a.s.TransportSearchAction] [Elastic-01] [elastalert_status][1], node[77Jo8GGyQWS0X9yHq9QjRA], [P], s[STARTED], a[id=-IbQ5EiJTVm4-kWP8rq-sQ]: Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[elastalert_status], indicesOptions=IndicesOptions[id=38, ignore_unavailable=false, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false], types=[elastalert], routing='null', preference='null', requestCache=null, scroll=null, maxConcurrentShardRequests=20, batchedReduceSize=512, preFilterShardSize=128, source={"size":10000,"query":{"query_string":{"query":"aggregate_id:-0HKXGUBpdqEbiOevJ70","fields":[],"type":"best_fields","default_operator":"or","max_determinized_states":10000,"enable_position_increments":true,"fuzziness":"AUTO","fuzzy_prefix_length":0,"fuzzy_max_expansions":50,"phrase_slop":0,"escape":false,"auto_generate_synonyms_phrase_query":true,"fuzzy_transpositions":true,"boost":1.0}},"sort":[{"@timestamp":{"order":"asc"}}]}}]