NVISOsecurity / ee-outliers

Open-source framework to detect outliers in Elasticsearch events
https://blog.nviso.eu
GNU General Public License v3.0
205 stars 34 forks source link

Handle ES conflict errors causing outliers to stop/restart #422

Open jvanwilder opened 5 years ago

jvanwilder commented 5 years ago
<SNIP> POST http://esnode1:9200/logstash-eagleeye-*/_update_by_query?refresh=true&wait_for_completion=true [status:409 request:0.511s]
Traceback (most recent call last):
  File "outliers.py", line 391, in <module>
    run_outliers()
  File "outliers.py", line 56, in run_outliers
    run_daemon_mode()
  File "outliers.py", line 179, in run_daemon_mode
    es.remove_all_outliers()
  File "/app/helpers/es.py", line 310, in remove_all_outliers
    self.conn.update_by_query(index=idx, body=query, refresh=True, wait_for_completion=True)
  File "/usr/local/lib/python3.6/site-packages/elasticsearch/client/utils.py", line 84, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/elasticsearch/client/__init__.py", line 975, in update_by_query
    body=body,
  File "/usr/local/lib/python3.6/site-packages/elasticsearch/transport.py", line 353, in perform_request
    timeout=timeout,
  File "/usr/local/lib/python3.6/site-packages/elasticsearch/connection/http_urllib3.py", line 236, in perform_request
    self._raise_error(response.status, raw_data)
  File "/usr/local/lib/python3.6/site-packages/elasticsearch/connection/base.py", line 162, in _raise_error
    status_code, error_message, additional_info
elasticsearch.exceptions.ConflictError: ConflictError(409 <SNIP>

Messages included in the conflict error are like the following:

{"type":"version_conflict_engine_exception",<SNIP> version conflict, required seqNo [<number>], primary term [<number>]. current document has seqNo [<number>] and primary term [<number>]" <SNIP>
daanraman commented 4 years ago

Relevant to this:

https://stackoverflow.com/questions/56840637/version-conflict-when-using-the-delete-method-of-elasticsearch-dsl