outflanknl / RedELK

Red Team's SIEM - tool for Red Teams used for tracking and alarming about Blue Team activities as well as better usability in long term operations.
BSD 3-Clause "New" or "Revised" License
2.35k stars 371 forks source link

ES document conflicterrors #288

Open MarcOverIP opened 1 year ago

MarcOverIP commented 1 year ago

Latest install - with a lot of test data - gives me numerous errors like the one below. The only common denominator seems that these concern documents in the redirtraffic index and/or are related to enrich_iplists from helpers.py.

2022-12-05 13:13:50,864 - ERROR - alarm - daemon.py - run_enrichments -- Error running enrichment enrich_iplists: ConflictError(409, 'version_conflict_engine_exception', '[-D_ZzYQBhF0UMvQj_Y6m]: version conflict, required seqNo [4813570], primary term [1]. current document has seqNo [7071780] and primary term [2]') | StackTrace: Traceback (most recent call last):
  File "/usr/share/redelk/bin/daemon.py", line 86, in run_enrichments
    set_tags(enrich_dict[enrich_module]["info"]["submodule"], [hit])
  File "/usr/share/redelk/bin/modules/helpers.py", line 118, in set_tags
    es.update(index=doc["_index"], id=doc["_id"], body={"doc": doc["_source"]})
  File "/usr/local/lib/python3.6/dist-packages/elasticsearch/client/utils.py", line 347, in _wrapped
    return func(*args, params=params, headers=headers, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/elasticsearch/client/__init__.py", line 2102, in update
    "POST", path, params=params, headers=headers, body=body
  File "/usr/local/lib/python3.6/dist-packages/elasticsearch/transport.py", line 466, in perform_request
    raise e
  File "/usr/local/lib/python3.6/dist-packages/elasticsearch/transport.py", line 434, in perform_request
    timeout=timeout,
  File "/usr/local/lib/python3.6/dist-packages/elasticsearch/connection/http_urllib3.py", line 291, in perform_request
    self._raise_error(response.status, raw_data)
  File "/usr/local/lib/python3.6/dist-packages/elasticsearch/connection/base.py", line 329, in _raise_error
    status_code, error_message, additional_info
elasticsearch.exceptions.ConflictError: ConflictError(409, 'version_conflict_engine_exception', '[-D_ZzYQBhF0UMvQj_Y6m]: version conflict, required seqNo [4813570], primary term [1]. current document has seqNo [7071780] and primary term [2]')

2022-12-05 13:13:50,865 - ERROR - alarm - daemon.py - run_enrichments -- ConflictError(409, 'version_conflict_engine_exception', '[-D_ZzYQBhF0UMvQj_Y6m]: version conflict, required seqNo [4813570], primary term [1]. current document has seqNo [7071780] and primary term [2]')
Traceback (most recent call last):
  File "/usr/share/redelk/bin/daemon.py", line 86, in run_enrichments
    set_tags(enrich_dict[enrich_module]["info"]["submodule"], [hit])
  File "/usr/share/redelk/bin/modules/helpers.py", line 118, in set_tags
    es.update(index=doc["_index"], id=doc["_id"], body={"doc": doc["_source"]})
  File "/usr/local/lib/python3.6/dist-packages/elasticsearch/client/utils.py", line 347, in _wrapped
    return func(*args, params=params, headers=headers, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/elasticsearch/client/__init__.py", line 2102, in update
    "POST", path, params=params, headers=headers, body=body
  File "/usr/local/lib/python3.6/dist-packages/elasticsearch/transport.py", line 466, in perform_request
    raise e
  File "/usr/local/lib/python3.6/dist-packages/elasticsearch/transport.py", line 434, in perform_request
    timeout=timeout,
  File "/usr/local/lib/python3.6/dist-packages/elasticsearch/connection/http_urllib3.py", line 291, in perform_request
    self._raise_error(response.status, raw_data)
  File "/usr/local/lib/python3.6/dist-packages/elasticsearch/connection/base.py", line 329, in _raise_error
    status_code, error_message, additional_info
elasticsearch.exceptions.ConflictError: ConflictError(409, 'version_conflict_engine_exception', '[-D_ZzYQBhF0UMvQj_Y6m]: version conflict, required seqNo [4813570], primary term [1]. current document has seqNo [7071780] and primary term [2]')
2022-12-05 13:13:50,865 - DEBUG - helpers - helpers.py - module_did_run -- Module did run: enrich:enrich_iplists [error] Error running enrichment enrich_iplists: ConflictError(409, 'version_conflict_engine_exception', '[-D_ZzYQBhF0UMvQj_Y6m]: version conflict, required seqNo [4813570], primary term [1]. current document has seqNo [7071780] and primary term [2]') | StackTrace: Traceback (most recent call last):
  File "/usr/share/redelk/bin/daemon.py", line 86, in run_enrichments
    set_tags(enrich_dict[enrich_module]["info"]["submodule"], [hit])
  File "/usr/share/redelk/bin/modules/helpers.py", line 118, in set_tags
    es.update(index=doc["_index"], id=doc["_id"], body={"doc": doc["_source"]})
  File "/usr/local/lib/python3.6/dist-packages/elasticsearch/client/utils.py", line 347, in _wrapped
    return func(*args, params=params, headers=headers, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/elasticsearch/client/__init__.py", line 2102, in update
    "POST", path, params=params, headers=headers, body=body
  File "/usr/local/lib/python3.6/dist-packages/elasticsearch/transport.py", line 466, in perform_request
    raise e
  File "/usr/local/lib/python3.6/dist-packages/elasticsearch/transport.py", line 434, in perform_request
    timeout=timeout,
  File "/usr/local/lib/python3.6/dist-packages/elasticsearch/connection/http_urllib3.py", line 291, in perform_request
    self._raise_error(response.status, raw_data)
  File "/usr/local/lib/python3.6/dist-packages/elasticsearch/connection/base.py", line 329, in _raise_error
    status_code, error_message, additional_info
elasticsearch.exceptions.ConflictError: ConflictError(409, 'version_conflict_engine_exception', '[-D_ZzYQBhF0UMvQj_Y6m]: version conflict, required seqNo [4813570], primary term [1]. current document has seqNo [7071780] and primary term [2]')

2022-12-05 13:13:50,901 - INFO - helpers - helpers.py - module_should_run -- All checks ok for module [enrich_greynoise]. Module should run.