Elasticsearch failed service when running Elastalert multiple rule

dungkma commented 6 years ago

When i ran multiple elastalert rule, the elasticsearch service was failed. My version of elasticsearch is 6.2.2 I have about 30 rules elastalert. This is my config elastalert config.yaml:

rules_folder: rules
run_every:
  minutes: 15
buffer_time:
  minutes: 30
es_host: 10.1.6.195
es_port: 9200
es_username: mirage
es_password: mirage
writeback_index: elastalert_status
alert_time_limit:
  days: 2
from_addr: monitor2@dkdte.com.vn
smtp_host: mail.dkdte.com.vn
email_reply_to: dkdte@dkdte.com.vn
slack_webhook_url: https://hooks.slack.com/services/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
slack_proxy: 10.1.7.55:8080

This is my elasticsearch logs when run elastalert rule:

[2018-08-22T00:01:04,228][DEBUG][o.e.a.s.TransportSearchAction] [Elastic-01] [elastalert_status][1], node[77Jo8GGyQWS0X9yHq9QjRA], [P], s[STARTED], a[id=-IbQ5EiJTVm4-kWP8rq-sQ]: Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[elastalert_status], indicesOptions=IndicesOptions[id=38, ignore_unavailable=false, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false], types=[elastalert], routing='null', preference='null', requestCache=null, scroll=null, maxConcurrentShardRequests=20, batchedReduceSize=512, preFilterShardSize=128, source={"size":10000,"query":{"query_string":{"query":"aggregate_id:-0HKXGUBpdqEbiOevJ70","fields":[],"type":"best_fields","default_operator":"or","max_determinized_states":10000,"enable_position_increments":true,"fuzziness":"AUTO","fuzzy_prefix_length":0,"fuzzy_max_expansions":50,"phrase_slop":0,"escape":false,"auto_generate_synonyms_phrase_query":true,"fuzzy_transpositions":true,"boost":1.0}},"sort":[{"@timestamp":{"order":"asc"}}]}}]

Qmando commented 6 years ago

That's the only log you've got? Nothing above DEBUG level? In #1838 it was reporter that deleting the elastaler_status index and recreating it fixed the issue, have you tried that?

dungkma commented 6 years ago

@Qmando Thanks. i deleted all index elastalert* and recreating but it was error and i still updating elastalert new version via pip. My Elasticsearch service was still dead.

Yelp / elastalert

Elasticsearch failed service when running Elastalert multiple rule #1878