Errors in elastalert-server (bitsensor/elastalert)

[mukeshchouhan]

I am getting below error in ElastAlert. I am using latest ElastAlert Image My Elastic search Version is 6.2.3 I have around 48 Rules Configured in ElastAlert

07:06:02.208Z ERROR elastalert-server:
    ProcessController:  WARNING:elasticsearch:POST http://escoordinator.*******.com:9200/elastalert_status_status/elastalert_status [status:429 request:0.007s]

07:06:02.218Z ERROR elastalert-server:
    ProcessController:  ERROR:root:Error writing alert info to Elasticsearch: TransportError(429, u'es_rejected_execution_exception', u'rejected execution of org.elasticsearch.transport.TransportService$7@61f5f91d on EsThreadPoolExecutor[name = server.********.com/bulk, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@1f0b4b9a[Running, pool size = 56, active threads = 56, queued tasks = 294, completed tasks = 2310423249]]')
    Traceback (most recent call last):
      File "/opt/elastalert/elastalert/elastalert.py", line 1518, in writeback
        doc_type=doc_type, body=body)
      File "/usr/lib/python2.7/site-packages/elasticsearch-6.3.1-py2.7.egg/elasticsearch/client/utils.py", line 76, in _wrapped
        return func(*args, params=params, **kwargs)
      File "/usr/lib/python2.7/site-packages/elasticsearch-6.3.1-py2.7.egg/elasticsearch/client/__init__.py", line 319, in index
        _make_path(index, doc_type, id), params=params, body=body)
      File "/usr/lib/python2.7/site-packages/elasticsearch-6.3.1-py2.7.egg/elasticsearch/transport.py", line 318, in perform_request
        status, headers_response, data = connection.perform_request(method, url, params, body, headers=headers, ignore=ignore, timeout=timeout)
      File "/usr/lib/python2.7/site-packages/elasticsearch-6.3.1-py2.7.egg/elasticsearch/connection/http_requests.py", line 90, in perform_request
        self._raise_error(response.status_code, raw_data)
      File "/usr/lib/python2.7/site-packages/elasticsearch-6.3.1-py2.7.egg/elasticsearch/connection/base.py", line 125, in _raise_error
        raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
    TransportError: TransportError(429, u'es_rejected_execution_exception', u'rejected execution of org.elasticsearch.transport.TransportService$7@61f5f91d on EsThreadPoolExecutor[name = server.********.com/bulk, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@1f0b4b9a[Running, pool size = 56, active threads = 56, queued tasks = 294, completed tasks = 2310423249]]')

07:06:02.219Z ERROR elastalert-server:
    ProcessController:  WARNING:root:Querying from 2018-12-27 06:59 UTC to 2018-12-27 07:06 UTC took longer than 0:00:05!

07:06:02.980Z ERROR elastalert-server:
    ProcessController:  WARNING:root:Querying from 2018-12-27 06:59 UTC to 2018-12-27 07:06 UTC took longer than 0:00:05!

[abhishekjiitr]

you are using the bitsensor-fork of elastalert, and I too encountered these errors on using it. The Yelp Official Elastalert is working fine though.

[mukeshchouhan]

Thanks @abhishekjiitr for the reply.

Did you find any workaround to fix it? I preferred bitsensor-fork of elastalert as it was easy to setup and comes with kibana plugin.

[abhishekjiitr]

Couldn't find any fix for it. It worked fine though, but kept on displaying the ERROR messages. I was just experimenting and now using the official ElastAlert. You can close this issue here, and open it on the Bitsensor fork Git Repo if you want.

[hazelcakli]

Hey @mukeshchouhan , Did you resolve this problem at least workaround on your environments? I got same problem. Especially, an alert's filter query took long time. The other rules working correct and doesn't have time problem.

[mukeshchouhan]

@hazelcakli 07:06:02.218Z ERROR elastalert-server: ProcessController: ERROR:root:Error writing alert info to Elasticsearch: TransportError(429, u'es_rejected_execution_exception', u'rejected execution of org.elasticsearch.transport.TransportService$7@61f5f91d on EsThreadPoolExecutor[name = server.********.com/bulk, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@1f0b4b9a[Running, pool size = 56, active threads = 56, queued tasks = 294, completed tasks = 2310423249]]')

this error was resolved by setting up below in elastalert.yaml file es_send_get_body_as: POST

I am still getting below errors but my alerts are coming fine. so that can be ignored.

07:06:02.219Z ERROR elastalert-server: ProcessController: WARNING:root:Querying from 2018-12-27 06:59 UTC to 2018-12-27 07:06 UTC took longer than 0:00:05!