Elasticsearch Versions supported

[unaisarakkal]

Wondering if elastalert supports elasticsearch version 2.0.0

ref: https://elastalert.readthedocs.org/en/latest/running_elastalert.html#requirements

[Qmando]

I have not done extensive testing with 2.0 but I've fixed several bugs which other people have brought up. As far as I can tell, most features work fine with 2.0. I will change the listed requirements when I do a little more thorough testing.

[bradwilliams-nm]

I am also curious if ElastAlert will work for 2.x. I tried creating the elastalert indexes against a 2.x cluster and received a RequestError from the elasticsearch api.

# elastalert-create-index
New index name? (Default elastalert_status)
Name of existing index to copy? (Default None)
Traceback (most recent call last):
  File "/usr/bin/elastalert-create-index", line 9, in <module>
    load_entry_point('elastalert==0.0.71', 'console_scripts', 'elastalert-create-index')()
  File "/usr/lib/python2.7/site-packages/elastalert-0.0.71-py2.7.egg/elastalert/create_index.py", line 94, in main
    es.indices.create(index)
  File "build/bdist.linux-x86_64/egg/elasticsearch/client/utils.py", line 69, in _wrapped
  File "build/bdist.linux-x86_64/egg/elasticsearch/client/indices.py", line 105, in create
  File "build/bdist.linux-x86_64/egg/elasticsearch/transport.py", line 329, in perform_request
  File "build/bdist.linux-x86_64/egg/elasticsearch/connection/http_urllib3.py", line 106, in perform_request
  File "build/bdist.linux-x86_64/egg/elasticsearch/connection/base.py", line 105, in _raise_error
elasticsearch.exceptions.RequestError

I was able to successfully to test a rule using elastalert-test-rule, but when I ran elastalert itself it also failed.

# elastalert-test-rule rules/sample_rule.yaml
Successfully loaded Event spike

INFO:urllib3.connectionpool:Starting new HTTP connection (1): elasticsearch-logging
Got 1259308 hits from the last 1 day

Available terms in first hit:
    log
    @timestamp
    container_id
    source
    tag
    container_name

INFO:urllib3.connectionpool:Starting new HTTP connection (1): elasticsearch-logging
INFO:elastalert:Queried rule Event spike from 2016-01-04 17:51 UTC to 2016-01-04 18:36 UTC: 10000 hits

# python elastalert/elastalert.py --debug --verbose
INFO:elastalert:Starting up
WARNING:elasticsearch:GET /elastalert_status/elastalert/_search?size=1000 [status:400 request:0.017s]
WARNING:elasticsearch:GET /elastalert_status/elastalert_status/_search?_source_include=endtime%2Crule_name&size=1 [status:400 request:0.017s]
ERROR:root:Traceback (most recent call last):
  File "elastalert/elastalert.py", line 696, in run_all_rules
    num_matches = self.run_rule(rule, endtime, self.starttime)
  File "elastalert/elastalert.py", line 482, in run_rule
    self.set_starttime(rule, endtime)
  File "elastalert/elastalert.py", line 430, in set_starttime
    last_run_end = self.get_starttime(rule)
  File "elastalert/elastalert.py", line 419, in get_starttime
    self.handle_error('Error querying for last run: %s' % (e), {'rule': rule['name']})
  File "build/bdist.linux-x86_64/egg/elasticsearch/exceptions.py", line 55, in __str__
    cause = ', %r' % self.info['error']['root_cause'][0]['reason']
TypeError: string indices must be integers

ERROR:root:Uncaught exception running rule Event spike: string indices must be integers
INFO:elastalert:Skipping writing to ES: {'message': 'Uncaught exception running rule Event spike: string indices must be integers', 'traceback': ['Traceback (most recent call last):', '  File "elastalert/elastalert.py", line 696, in run_all_rules', '    num_matches = self.run_rule(rule, endtime, self.starttime)', '  File "elastalert/elastalert.py", line 482, in run_rule', '    self.set_starttime(rule, endtime)', '  File "elastalert/elastalert.py", line 430, in set_starttime', '    last_run_end = self.get_starttime(rule)', '  File "elastalert/elastalert.py", line 419, in get_starttime', "    self.handle_error('Error querying for last run: %s' % (e), {'rule': rule['name']})", '  File "build/bdist.linux-x86_64/egg/elasticsearch/exceptions.py", line 55, in __str__', "    cause = ', %r' % self.info['error']['root_cause'][0]['reason']", 'TypeError: string indices must be integers'], 'data': {'rule': 'Event spike'}}

[jraby]

For the record, we're running with elasticsearch 2.1 for our backends. You have to upgrade the elasticsearch python module for it to work. We're running with elasticsearch==2.1.0 currently and it works fine for our use cases, but we don't have spike rules at the moment.

There was also some changes that were required to the creadex_index.py script. Did you try with master or 0.71 ?

[bradwilliams-nm]

I pulled from master just this morning, but did not update to use the 2.x elasticsearch python modules. I will give that a shot. Thanks!

[bradwilliams-nm]

Upgrading the elasticsearch module to 2.1.0 worked. Thanks again.

[alexandreabeh]

For anyone who runs into similar problems:

I am running elasticsearch 1.7.2 and the 2.2 elasticsearch python module was giving me the same 'TypeError: string indices must be integers' error. Changing to the 2.1.0 elasticsearch python module worked.

[dmilanp]

Same error with Elasticsearch 1.7.4. Fixed using elasticsearch==2.1.0 as well for the python module.