OpenSearch "version_conflict_engine_exception", document already exists

[taskula]

Hi,

First of all, thank you for the amazing project.

I have Evebox version 0.18.0 connected to OpenSearch 2.10.2. Evebox-agents are sending Suricata events to OpenSearch and the events can be viewed on my Evebox server. My problem is that on our Suricata/evebox-agent hosts, syslog grows daily by a few GB with the following errors:

Mar 31 00:00:04 host evebox[2543582]: #033[2m2024-03-31 00:00:04#033[0m #033[31mERROR#033[0m #033[2mevebox::elastic::importer#033[0m#033[2m:#033[0m Elasticsearch reported errors during commit:                       {"took":2,"errors":true,"items":[{"create":{"_index":"suricata_host-2024.03.26","_id":"01HSWBYXAT3DHCG69QK573BKHH","status":409,"error":{"type":"version_conflict_engine_exception","reason":"[01HSWBYXAT3DHCG69QK573BKHH]: version conflict, document already exists (current version [1])","index":"suricata_host-2024.03.26","shard":"0","index_uuid":"hvM4oIPCQ-i06sWuBtXozA"}}},{"create":{"_index":"suricata_host-2024.03.26","_id":"01HSWBYXATEYCJSJDGKN2QRRFS","status":409,"error":{"type":"version_conflict_engine_exception","reason":"[01HSWBYXATEYCJSJDGKN2QRRFS]: version conflict, document already exists (current version [1])","index":"suricata_host-2024.03.26","shard":"0","index_uuid":"hvM4oIPCQ-i06sWuBtXozA"}}},{"create":{"_index":"suricata_host-2024.03.26","_id":"01HSWBYY0RVSDWP93NG0KD5EBC","status":409,"error":{"type":"version_conflict_engine_exception...
Mar 31 00:00:04 host evebox[2543582]: #033[2m2024-03-31 00:00:04#033[0m #033[31mERROR#033[0m #033[2mevebox::eve::processor#033[0m#033[2m:#033[0m Failed to commit events (will try again): elasticsearch commit error: {"took":2,"errors":true,"items":[{"create":{"_index":"suricata_host-2024.03.26","_id":"01HSWBYXAT3DHCG69QK573BKHH","status":409,"error":{"type":"version_conflict_engine_exception","reason":"[01HSWBYXAT3DHCG69QK573BKHH]: version conflict, document already exists (current version [1])","index":"suricata_host-2024.03.26","shard":"0","index_uuid":"hvM4oIPCQ-i06sWuBtXozA"}}},{"create":{"_index":"suricata_host-2024.03.26","_id":"01HSWBYXATEYCJSJDGKN2QRRFS","status":409,"error":{"type":"version_conflict_engine_exception","reason":"[01HSWBYXATEYCJSJDGKN2QRRFS]: version conflict, document already exists (current version [1])","index":"suricata_host-2024.03.26","shard":"0","index_uuid":"hvM4oIPCQ-i06sWuBtXozA"}}},{"create":{"_index":"suricata_host-2024.03.26","_id":"01HSWBYY0RVSDWP93NG0KD5EBC","status":409,"error":{"type":"version_conflict_engine_exception...

I have cut the logs by "..." but the same error message just keeps repeating (with different "_id" value), and the above two entries are repeated every second in syslog/journal.

agent.yaml

elasticsearch:
  enabled: true
  index: suricata_host
  password: verysecret
  url: https://opensearch.example.org
  username: me
input:
  paths:
  - /var/log/suricata/eve.json
  - /var/log/suricata/eve.*.json

Am I missing something?

[jasonish]

Does this happen after a fresh restart of the agent? I ask, because I do see a way the agent could get into this state, for example if it submitted some events and some failed for some reason, it will try to resubmit all again. However, it should start clean, or at least given a different error, then enter this error state.

With Elasticsearch I've only ever seen all fail or none, so this is a case I'll have to look at more closely.

I did some testing with Opensearch 2.11.x a while back but am currently not running it. I'll see about flipping back to it again for a while.

[taskula]

I have not seen any more errors today after restarting the agent. That's odd because I was sure I tried restarting it before creating this issue. However as of now evebox-agent is working normally and no more error messages are present. I will report back if I am able to reproduce the issue. For now, thanks a ton for the quick response!