Integration with elasticsearch v5

[rauizab]

Hi

I know that the new version v5 of elasticsearch has just being released, but do you know when it will be available the integration of elastalert with the new version? At the moment I am getting:

INFO:elastalert:Starting up WARNING:elasticsearch:POST http://elasticsearch:9200/elastalert_status/elastalert/_search?size=1000 [status:400 request:0.013s] WARNING:elasticsearch:POST http://elasticsearch:9200/elastalert_status/elastalert_status/_search?_source_include=endtime%2Crule_name&size=1 [status:400 request:0.002s] ERROR:root:Error querying for last run: TransportError(400, {u'line': 1, u'root_cause': [{u'reason': u'Unknown key for a START_OBJECT in [filter].', u'type': u'parsing_exception', u'line': 1, u'col': 12}], u'type': u'parsing_exception', u'reason': u'Unknown key for a START_OBJECT in [filter].', u'col': 12}) WARNING:elasticsearch:POST http://elasticsearch:9200/elastalert_status/elastalert_error?op_type=create [status:400 request:0.003s] ERROR:root:Error writing alert info to elasticsearch: TransportError(400, {u'root_cause': [{u'reason': u'Validation Failed: 1: an id must be provided if version type or value are set;', u'type': u'action_request_validation_exception'}], u'type': u'action_request_validation_exception', u'reason': u'Validation Failed: 1: an id must be provided if version type or value are set;'}) Traceback (most recent call last): File "/opt/elastalert/elastalert/elastalert.py", line 1033, in writeback doc_type=doc_type, body=body) File "/usr/lib/python2.7/site-packages/elasticsearch/client/utils.py", line 68, in _wrapped return func(*args, params=params, kwargs) File "/usr/lib/python2.7/site-packages/elasticsearch/client/init.py", line 227, in create return self.index(index, doc_type, body, id=id, params=params, op_type='create') File "/usr/lib/python2.7/site-packages/elasticsearch/client/utils.py", line 68, in _wrapped return func(*args, params=params, kwargs) File "/usr/lib/python2.7/site-packages/elasticsearch/client/init.py", line 257, in index _make_path(index, doc_type, id), params=params, body=body) File "/usr/lib/python2.7/site-packages/elasticsearch/transport.py", line 301, in perform_request status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout) File "/usr/lib/python2.7/site-packages/elasticsearch/connection/http_requests.py", line 72, in perform_request self._raise_error(response.status_code, raw_data) File "/usr/lib/python2.7/site-packages/elasticsearch/connection/base.py", line 102, in _raise_error raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info) RequestError: TransportError(400, {u'root_cause': [{u'reason': u'Validation Failed: 1: an id must be provided if version type or value are set;', u'type': u'action_request_validation_exception'}], u'type': u'action_request_validation_exception', u'reason': u'Validation Failed: 1: an id must be provided if version type or value are set;'}) WARNING:elasticsearch:GET http://elasticsearch:9200/payment-*/_search?_source_include=eventtime%2C%2A%2CuserId&ignore_unavailable=true&size=10000 [status:400 request:0.003s] ERROR:root:Error running query: TransportError(400, {u'line': 1, u'root_cause': [{u'reason': u'no [query] registered for [filtered]', u'type': u'parsing_exception', u'line': 1, u'col': 67}], u'type': u'parsing_exception', u'reason': u'no [query] registered for [filtered]', u'col': 67}) WARNING:elasticsearch:POST http://elasticsearch:9200/elastalert_status/elastalert_error?op_type=create [status:400 request:0.003s] ERROR:root:Error writing alert info to elasticsearch: TransportError(400, {u'root_cause': [{u'reason': u'Validation Failed: 1: an id must be provided if version type or value are set;', u'type': u'action_request_validation_exception'}], u'type': u'action_request_validation_exception', u'reason': u'Validation Failed: 1: an id must be provided if version type or value are set;'}) Traceback (most recent call last): File "/opt/elastalert/elastalert/elastalert.py", line 1033, in writeback doc_type=doc_type, body=body) File "/usr/lib/python2.7/site-packages/elasticsearch/client/utils.py", line 68, in _wrapped return func(*args, params=params, kwargs) File "/usr/lib/python2.7/site-packages/elasticsearch/client/init.py", line 227, in create return self.index(index, doc_type, body, id=id, params=params, op_type='create') File "/usr/lib/python2.7/site-packages/elasticsearch/client/utils.py", line 68, in _wrapped return func(*args, params=params, kwargs) File "/usr/lib/python2.7/site-packages/elasticsearch/client/init.py", line 257, in index _make_path(index, doc_type, id), params=params, body=body) File "/usr/lib/python2.7/site-packages/elasticsearch/transport.py", line 301, in perform_request status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout) File "/usr/lib/python2.7/site-packages/elasticsearch/connection/http_requests.py", line 72, in perform_request self._raise_error(response.status_code, raw_data) File "/usr/lib/python2.7/site-packages/elasticsearch/connection/base.py", line 102, in _raise_error raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info) RequestError: TransportError(400, {u'root_cause': [{u'reason': u'Validation Failed: 1: an id must be provided if version type or value are set;', u'type': u'action_request_validation_exception'}], u'type': u'action_request_validation_exception', u'reason': u'Validation Failed: 1: an id must be provided if version type or value are set;'}) INFO:elastalert:Ran User1111MakePayment from 2016-11-02 08:20 UTC to 2016-11-02 13:20 UTC: 0 query hits, 0 matches, 0 alerts sent

Thanks for this great tool!!!!

[penekk]

ES 5 has been around for a while already, its the GA version that just came out, any ETA on the support would be lovely :+1:

[watollop]

+1

[lrolsen]

+1

[vijayarulmuthu]

+1

On Sat, Nov 5, 2016 at 5:14 AM -0700, "lrolsen" notifications@github.com<mailto:notifications@github.com> wrote:

+1

You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/Yelp/elastalert/issues/790#issuecomment-258607858, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AH7j4V-D_DPsJ-KmIqww-sOMgZ5FaPG-ks5q7HMggaJpZM4KnOvN.

[sabueso]

@watollop say me to put +1 here

[pasqualguerrero]

+1

[tynor88]

+1

[bennneuh]

+1

[nick-george]

+1 Duplicate of #510

[AndrewPix]

+1

[Alexwii]

+1

[stumyp]

Is it known what is broken and how can we help ?

[bkeifer]

IRC pointed me to https://github.com/Yelp/elastalert/blob/master/elastalert/elastalert.py#L159

[doublesea]

+1

[doublesea]

Is it known what is broken and how can we help ?

as i known, it is because "filter" is not available any more in ES 5.0.

There are many "filter" query in EA which will get the 400 HTTP response for the incorrect query clause.

[kojisaiki]

+1

[branchnetconsulting]

+1

[stumyp]

It looks like all needed is to rewrite this query: https://github.com/Yelp/elastalert/blob/master/elastalert/elastalert.py#L159 (thanks @bkeifer )

Documentation says that change should be simple: https://www.elastic.co/guide/en/elasticsearch/reference/5.0/query-dsl-filtered-query.html

I'll try to find some time, my python is a bit rusty :)

[timwsuqld]

My reading of that doc suggests that the change should be:

diff --git a/elastalert/elastalert.py b/elastalert/elastalert.py
index 2cbd553..a13e739 100644
--- a/elastalert/elastalert.py
+++ b/elastalert/elastalert.py
@@ -156,7 +156,7 @@ class ElastAlerter():
         if starttime and endtime:
             es_filters['filter']['bool']['must'].insert(0, {'range': {timestamp_field: {'gt': starttime,
                                                                                         'lte': endtime}}})
-        query = {'query': {'filtered': es_filters}}
+        query = {'query': {'bool': es_filters}}
         if sort:
             query['sort'] = [{timestamp_field: {'order': 'desc' if desc else 'asc'}}]
         return query

I'll see if I can make it work in our ES5 environment

[stumyp]

@timwsuqld: plus if statement to detect ES version

[timwsuqld]

@stumyp Good point. Any idea if we already detect ES version anywhere?

[stumyp]

@timwsuqld : So far I saw only this: https://github.com/Yelp/elastalert/commit/63584350031599a59a90816c85fb2de7ed5a1bd0

Not really a version detection :)

[timwsuqld]

I'm also not sure if the elasticsearch library (https://pypi.python.org/pypi/elasticsearch/5.0.1) is backwards compatible. My understanding is that it should be, so we can use version 5 of the library with older ES clusters.

[stumyp]

From what they say on the link you gave: major version of library must match ES version. Means, elastalert should split versions too or just warn users for incompatibility ?

[Qmando]

I believe this patch must also be applied, because id must be explicitly set to None.

diff --git a/elastalert/elastalert.py b/elastalert/elastalert.py
index 2cbd553..fd49223 100644
--- a/elastalert/elastalert.py
+++ b/elastalert/elastalert.py
@@ -841,7 +841,8 @@ class ElastAlerter():

         res = es.create(index='kibana-int',
                         doc_type='temp',
-                        body=db_body)
+                        body=db_body,
+                        id=None)

         # Return dashboard URL
         kibana_url = rule.get('kibana_url')
@@ -1015,7 +1016,7 @@ class ElastAlerter():
         if self.writeback_es:
             try:
                 res = self.writeback_es.create(index=self.writeback_index,
-                                               doc_type=doc_type, body=body)
+                                               doc_type=doc_type, body=body, id=None)
                 return res
             except ElasticsearchException as e:
                 logging.exception("Error writing alert info to Elasticsearch: %s" % (e))

[timwsuqld]

For what it's worth, I appear to have no problems using the elasticsearch 2.4.0 library, and the index creation worked without a problem, as do my filters with my patch. I'm sure there are incompatibilities between the 2.4.0 library and ES 5, but maybe the things that we need won't actually hit those incompatibilities?

[Qmando]

The filters work fine? The current format is

query:
  filtered:
    filter:
      bool:
        must: [filters from rule here]

The docs say that filtered has been deprecated. I'm not really an expert on the query DSL though.

Another breaking change: No more fields https://github.com/Yelp/elastalert/blob/master/elastalert/elastalert.py#L243 That should be stored_fields instead.

search_type=count is removed too https://github.com/Yelp/elastalert/blob/master/elastalert/elastalert.py#L318 Instead, you have to add size: 0

These are just what stood out when scanning the breaking changes page

[stumyp]

@Qmando : I believe it should look like

query:
  bool:
    must: [filters from rule here]

[timwsuqld]

@Qmando the filters work fine with my above patch, and the ES 2.4.0 library against a ES 5.0.0 server The filter comes out something like:

query:
  bool:
    filter:
      bool:
        must: [filters from rule here]

It looks odd with a bool then filter then bool, it just happens we are using the bool filter. @stumyp I'm not sure if your shorter query would work, it's not what I understood the docs to mean.

[timwsuqld]

I've started a branch (https://github.com/suqld/elastalert/tree/support_es5) that we can work on. I'll try and find ways to make it crash (based on the breaking changes), then commit fixes

[stumyp]

you're right @timwsuqld , my query would work, but a bit differently:

from documentation:

*filter*
The clause (query) must appear in matching documents. 
However unlike must the score of the query will be ignored.

[timwsuqld]

@Qmando Regarding the ID, it's not as easy as id=None :( https://github.com/elastic/elasticsearch-py/issues/474#issuecomment-256903012

Looks like we need to change the call from create to index

[amirkool]

+1

[timwsuqld]

Given the number of changes, I'm wondering the best way to handle the ES5 changes. We could have a config option (or autodetect ES version) and then in all the places the query needs to be changed, have an if statement. Alternatively, we maintain 2 branches.

Suggestions?

[stumyp]

I'm not sure how much work (and time answering people in issues) will it be to maintain everything in one place with detection/configuration option.

If it is easier to keep separate branch/tag for ES5 - I'm fine with that.

[k2xl]

+1

[bdh1011]

+1

[Fuhunter]

+1 need this for my Bachelorthesis :P

[KenanAlptekin]

+1 would love this for our new ES5 stack.

[timwsuqld]

Please don't +1 this, use the thumbs up on the issue. The branch at https://github.com/suqld/elastalert/tree/support_es5 is currently working for me in production against ES5. I'm thinking it would be good if @stumyp could create an es5 branch that I can submit a merge request against, so we can get my changes into the elastalert repo.

[Fuhunter]

@timwsuqld Your branch seems to be working fine! Except mass amount of errors while installation process. Btw skipped pip install -r requirements.txt in my docker container, but no problems occurred

[stumyp]

@timwsuqld I'm external collaborator, same as you, don't have any control over this repo. I think @Qmando can do it.

[Qmando]

Hey guys. I took @timwsuqld's changes and added code to grab the elasticsearch version.

Please pull the support_es5 branch and test it for me!!

$ git fetch origin
$ git checkout origin/support_es5
$ pip install elasticsearch>3.0

https://github.com/Yelp/elastalert/pull/820/files

[timwsuqld]

@Qmando Thanks for doing that! I'll try and test that in the next few days. I knew it should be easy to work out the version, just hadn't had time to dig that deep!

[malterb]

@Qmando with support_es5 I still get ERROR:root:Error running query: TransportError(400, u'parsing_exception', u'no [query] registered for [query]')

[doublesea]

I encountered the same problem with elmalto, while I tried to resolved it. It was because the rule.yaml file need to be modified ,too. I changed it form

filter:
- query:
    query_string:
      query: "Extends:0x60"

to

filter:
  - query_string: 
      query: "Extends:0x60"

Then it works.

It seems that the sample need to modified in http://elastalert.readthedocs.io/en/latest/recipes/writing_filters.html#writingfilters

[i-sam]

+1

[Hronom]

Guys what the progress on this? We are extremely need this...

[Qmando]

@Hronom The branch is out. I would like more feedback as I don't have an ES5 test environment right now. There are some things that need to be done, like use_terms_query doesn't work.

[Hronom]

@Qmando So I can use branch support_es5?