[BUG] Document Level Monitor failing with ConcurrentModificationException

[jkrishnarao2003]

What is the bug? I setup a Document Level Monitor in OpenSearch 2.10.0. It is failing to run with error ConcurrentModificationException.

How can one reproduce the bug? Monitor Config

{ "name": "Monitor1", "type": "monitor", "monitor_type": "doc_level_monitor", "enabled": true, "schedule": { "period": { "unit": "MINUTES", "interval": 1 } }, "inputs": [ { "doc_level_input": { "description": "", "indices": [ "rlr" ], "queries": [ { "id": "e1a6c189-aece-47d3-8fe9-97cfc4b12d68", "name": "query1", "query": "message:\"java.lang.OutOfMemoryError\"", "tags": [ "ERROR" ] } ] } } ], "triggers": [], "ui_metadata": { "schedule": { "timezone": null, "frequency": "interval", "period": { "unit": "MINUTES", "interval": 1 }, "daily": 0, "weekly": { "tue": false, "wed": false, "thur": false, "sat": false, "fri": false, "mon": false, "sun": false }, "monthly": { "type": "day", "day": 1 }, "cronExpression": "0 /1 *" }, "monitor_type": "doc_level_monitor", "doc_level_input": { "queries": [ { "id": "e1a6c189-aece-47d3-8fe9-97cfc4b12d68", "queryName": "query1", "field": "message", "operator": "is", "query": "java.lang.OutOfMemoryError", "tags": [ "ERROR" ] } ] }, "search": { "searchType": "graph" } } }

Exception trace when starting the Monitor

[2024-05-27T22:35:10,135][ERROR][o.o.a.DocumentLevelMonitorRunner] [omhqp120d5.uprr.com] Failed running Document-level-monitor Monitor1 java.util.ConcurrentModificationException: null at java.util.LinkedHashMap$LinkedHashIterator.nextNode(LinkedHashMap.java:756) ~[?:?] at java.util.LinkedHashMap$LinkedKeyIterator.next(LinkedHashMap.java:778) ~[?:?] at org.opensearch.alerting.DocumentLevelMonitorRunner.runMonitor(DocumentLevelMonitorRunner.kt:141) [opensearch-alerting-2.10.0.0.jar:2.10.0.0] at org.opensearch.alerting.DocumentLevelMonitorRunner$runMonitor$1.invokeSuspend(DocumentLevelMonitorRunner.kt) [opensearch-alerting-2.10.0.0.jar:2.10.0.0] at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) [kotlin-stdlib-1.8.21.jar:1.8.21-release-380(1.8.21)] at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:233) [kotlinx-coroutines-core-1.1.1.jar:?] at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:594) [kotlinx-coroutines-core-1.1.1.jar:?] at kotlinx.coroutines.scheduling.CoroutineScheduler.access$runSafely(CoroutineScheduler.kt:60) [kotlinx-coroutines-core-1.1.1.jar:?] at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:742) [kotlinx-coroutines-core-1.1.1.jar:?]

What is the expected behavior? Is it a bug in OpenSearch or something wrong with my monitor config?

What is your host/environment?

• OS: CentOS
• Version 7
• Plugins: OpenSearch 2.10.0 opensearch-alerting plugin

[dblock]

Catch All Triage - 1 2 3 4 5 6