[BUG] Notification Channels list shows objects multiple times - breaking notifications

[tomuk5]

In the Monitor creation or modification screen, the notification channels dropdown menu shows the list of configured email and email group channels multiple times over. Only one of the duplicated choices results in the monitor working, selection of a non-working channel results in the error displayed to the user: For input string: "NOT_FOUND"

in the oplog on the opensearch node, the below error is seen:

[2023-11-07T14:39:36,075][INFO ][o.o.n.i.ConfigIndexingActions] [graylog-ingest1] notifications:NotificationConfig-info ns7q4IoBZokGLCNvK7Zm
[2023-11-07T14:39:36,082][INFO ][o.o.n.s.SendMessageActionHelper] [graylog-ingest1] notifications:getSingleConfig-get ns7q4IoBZokGLCNvK7Zm
[2023-11-07T14:39:36,086][INFO ][o.o.c.n.m.NotificationConfig] [graylog-ingest1] non-allowed config type for Status
[2023-11-07T14:39:36,087][WARN ][o.o.n.s.SendMessageActionHelper] [graylog-ingest1] Cannot send message to destination for config id :ns7q4IoBZokGLCNvK7Zm
[2023-11-07T14:39:36,087][INFO ][o.o.c.n.m.NotificationConfig] [graylog-ingest1] non-allowed config type for Status
[2023-11-07T14:39:36,089][WARN ][o.o.n.a.PluginBaseAction ] [graylog-ingest1] notifications:IllegalArgumentException:
java.lang.NumberFormatException: For input string: "NOT_FOUND"
        at java.lang.NumberFormatException.forInputString(NumberFormatException.java:67) ~[?:?]
        at java.lang.Integer.parseInt(Integer.java:668) ~[?:?]
        at java.lang.Integer.parseInt(Integer.java:786) ~[?:?]
        at org.opensearch.notifications.send.SendMessageActionHelper.executeRequest(SendMessageActionHelper.kt:103) ~[?:?]
        at org.opensearch.notifications.send.SendMessageActionHelper$executeRequest$1.invokeSuspend(SendMessageActionHelper.kt) ~[?:?]
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) [kotlin-stdlib-1.8.21.jar:1.8.21-release-380(1.8.21)]
        at kotlinx.coroutines.internal.ScopeCoroutine.afterResume(Scopes.kt:32) [kotlinx-coroutines-core-jvm-1.4.3.jar:?]
        at kotlinx.coroutines.AbstractCoroutine.resumeWith(AbstractCoroutine.kt:113) [kotlinx-coroutines-core-jvm-1.4.3.jar:?]
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46) [kotlin-stdlib-1.8.21.jar:1.8.21-release-380(1.8.21)]
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106) [kotlinx-coroutines-core-jvm-1.4.3.jar:?]
        at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571) [kotlinx-coroutines-core-jvm-1.4.3.jar:?]
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750) [kotlinx-coroutines-core-jvm-1.4.3.jar:?]
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678) [kotlinx-coroutines-core-jvm-1.4.3.jar:?]
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665) [kotlinx-coroutines-core-jvm-1.4.3.jar:?]

querying GET _plugins/_notifications/configs shows that there are 3x email groups configured, but the dropdown list shows 9x (3x duplicated lots of 3x)

How can one reproduce the bug? Steps to reproduce the behavior:

1. Go to Alerting>Monitors
2. Edit or create a monitor
3. Scroll Down to actions and then notifications
4. click on the Channel dropdown list
5. entries for channels (email and email groups) are displayed multiple times in the list

What is the expected behavior? The Notification emails/email groups be shown correctly

What is your host/environment?

• OS: Debian 11
• Version 2.11

Do you have any screenshots?

[pbn4]

I also face this issue, @amsiglan I see you self assigned yourself not so long ago, do you plan to work on this?