Openshift Elasticsearch & Elastalert

[Hyper200]

Hi there,

Thanks to the following repo we have been able to get an elastalert instance running on RHEL openshift.

https://github.com/kilimandjango/openshift-elastalert

It does appear to work but gets the following error messages:

INFO:elastalert:Sleeping for 59.966621 seconds INFO:elastalert:Queried rule OutOfMemoryError from 2018-05-24 13:15 UTC to 2018-05-24 13:26 UTC: 0 / 0 hits WARNING:elasticsearch:POST https://logging-es:9200/elastalert_status/elastalert_status [status:403 request:0.003s] ERROR:root:Error writing alert info to Elasticsearch: TransportError(403, u'security_exception', u'no permissions for indices:data/write/index') Traceback (most recent call last): File "/opt/app-root/lib/python2.7/site-packages/elastalert/elastalert.py", line 1438, in writeback doc_type=doc_type, body=body) File "/opt/app-root/lib/python2.7/site-packages/elasticsearch/client/utils.py", line 76, in _wrapped return func(*args, params=params, **kwargs) File "/opt/app-root/lib/python2.7/site-packages/elasticsearch/client/init.py", line 319, in index _make_path(index, doc_type, id), params=params, body=body) File "/opt/app-root/lib/python2.7/site-packages/elasticsearch/transport.py", line 314, in perform_request status, headers_response, data = connection.perform_request(method, url, params, body, headers=headers, ignore=ignore, timeout=timeout) File "/opt/app-root/lib/python2.7/site-packages/elasticsearch/connection/http_requests.py", line 90, in perform_request self._raise_error(response.status_code, raw_data) File "/opt/app-root/lib/python2.7/site-packages/elasticsearch/connection/base.py", line 125, in _raise_error raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info) AuthorizationException: TransportError(403, u'security_exception', u'no permissions for indices:data/write/index') INFO:elastalert:Ran OutOfMemoryError from 2018-05-24 13:15 UTC to 2018-05-24 13:26 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent INFO:elastalert:Sleeping for 59.973543 seconds

Is there configuration for elasticsearch that requires updated to permit elastalert to be able to write to the indices?

Cheers

[Qmando]

I don't really know anything about openshift or why elasticsearch doesn't accept writes. Being able to write is required for a number of features, though it's basic functionally may continue working.

You could create a new Elasticsearch cluster locally to allow elastalert to write to, I guess?

[rotem443]

Just throwing up an idea, maybe u don't have the permission to create an index ? Because the first step when the pod is running is to log for the elastalert logs index.

[Hyper200]

I suspect your are right @rotem443 not sure how to permit writes to the existing ES in openshift. I have asked Redhat this.

@Qmando thanks, not sure that we can as elstalert would then need to support connections to two ES instances?

[Qmando]

It does support that. Just put es_host/port in the global config.yaml pointing to one, and another es_host/port in each rule yaml.

[Hyper200]

Hi there,

I endedup having a chat to Openshift support and it turns out the client cert, key only has permissions to read.

Redhad suggested using the admin key/cert for the eslastic search for this component and that worked.