search

opensearch-project / security-analytics

Security Analytics enables users for detecting security threats on their security event log data. It will also allow them to modify/tailor the pre-packaged solution.
Apache License 2.0
72 stars 74 forks source link

[BUG] Alerting error: "Filter by user backend roles is enabled with security disabled." #878

Open wbeckler opened 1 year ago

wbeckler commented 1 year ago

A user reported the following via the forum:

I have recently upgraded the opensearch to 2.6 version and Alerting plugin is throwing exception Below is the exception showing when I open the monitor or the alerting page

image

I am not able to figure out the issue and the security admin script is also working and the cluster is security enabled

Do you have any additional context? Here's an error log:

[2023-04-05T08:21:40,974][ERROR][o.o.a.u.AlertingException] [olselkdev-manager-1] Alerting error: OpenSearchStatusException[Filter by user backend roles is enabled with security disabled.]
[2023-04-05T08:21:40,975][ERROR][o.o.a.t.TransportGetAlertsAction] [olselkdev-manager-1] Failed to get alerts
org.opensearch.alerting.util.AlertingException: Filter by user backend roles is enabled with security disabled.
at org.opensearch.alerting.util.AlertingException$Companion.wrap(AlertingException.kt:70) ~[?:?]
at org.opensearch.alerting.transport.SecureTransportAction$DefaultImpls.validateUserBackendRoles(SecureTransportAction.kt:82) ~[?:?]
at org.opensearch.alerting.transport.TransportGetMonitorAction.validateUserBackendRoles(TransportGetMonitorAction.kt:36) ~[?:?]
at org.opensearch.alerting.transport.TransportGetMonitorAction.doExecute(TransportGetMonitorAction.kt:61) ~[?:?]
at org.opensearch.alerting.transport.TransportGetMonitorAction.doExecute(TransportGetMonitorAction.kt:36) ~[?:?]
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) ~[opensearch-2.6.0.jar:2.6.0]
at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:118) ~[?:?]
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) ~[opensearch-2.6.0.jar:2.6.0]
at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:264) ~[?:?]
at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:149) ~[?:?]
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) ~[opensearch-2.6.0.jar:2.6.0]
at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:78) ~[?:?]
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) ~[opensearch-2.6.0.jar:2.6.0]
at org.opensearch.action.support.TransportAction.execute(TransportAction.java:188) ~[opensearch-2.6.0.jar:2.6.0]
at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107) ~[opensearch-2.6.0.jar:2.6.0]
at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110) ~[opensearch-2.6.0.jar:2.6.0]
at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97) ~[opensearch-2.6.0.jar:2.6.0]
at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:465) ~[opensearch-2.6.0.jar:2.6.0]
at org.opensearch.alerting.transport.TransportGetAlertsAction$resolveAlertsIndexName$2$getMonitorResponse$1.invoke(TransportGetAlertsAction.kt:165) ~[opensearch-alerting-2.6.0.0.jar:2.6.0.0]
at org.opensearch.alerting.transport.TransportGetAlertsAction$resolveAlertsIndexName$2$getMonitorResponse$1.invoke(TransportGetAlertsAction.kt:164) ~[opensearch-alerting-2.6.0.0.jar:2.6.0.0]
at org.opensearch.alerting.opensearchapi.OpenSearchExtensionsKt.suspendUntil(OpenSearchExtensions.kt:153) ~[alerting-core-2.6.0.0.jar:?]
at org.opensearch.alerting.transport.TransportGetAlertsAction$resolveAlertsIndexName$2.invokeSuspend(TransportGetAlertsAction.kt:164) ~[opensearch-alerting-2.6.0.0.jar:2.6.0.0]
at org.opensearch.alerting.transport.TransportGetAlertsAction$resolveAlertsIndexName$2.invoke(TransportGetAlertsAction.kt) ~[opensearch-alerting-2.6.0.0.jar:2.6.0.0]
at org.opensearch.alerting.transport.TransportGetAlertsAction$resolveAlertsIndexName$2.invoke(TransportGetAlertsAction.kt) ~[opensearch-alerting-2.6.0.0.jar:2.6.0.0]
at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturn(Undispatched.kt:91) ~[kotlinx-coroutines-core-1.1.1.jar:?]
at kotlinx.coroutines.BuildersKt__Builders_commonKt.withContext(Builders.common.kt:146) ~[kotlinx-coroutines-core-1.1.1.jar:?]
at kotlinx.coroutines.BuildersKt.withContext(Unknown Source) ~[kotlinx-coroutines-core-1.1.1.jar:?]
at org.opensearch.alerting.transport.TransportGetAlertsAction.resolveAlertsIndexName(TransportGetAlertsAction.kt:156) ~[opensearch-alerting-2.6.0.0.jar:2.6.0.0]
at org.opensearch.alerting.transport.TransportGetAlertsAction$doExecute$1$1.invokeSuspend(TransportGetAlertsAction.kt:132) [opensearch-alerting-2.6.0.0.jar:2.6.0.0]
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) [kotlin-stdlib-1.6.10.jar:1.6.10-release-923(1.6.10)]
at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:233) [kotlinx-coroutines-core-1.1.1.jar:?]
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:594) [kotlinx-coroutines-core-1.1.1.jar:?]
at kotlinx.coroutines.scheduling.CoroutineScheduler.access$runSafely(CoroutineScheduler.kt:60) [kotlinx-coroutines-core-1.1.1.jar:?]
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:742) [kotlinx-coroutines-core-1.1.1.jar:?]
Caused by: java.lang.Exception: org.opensearch.OpenSearchStatusException: Filter by user backend roles is enabled with security disabled.
ravikiranvuppu commented 1 year ago

thanks @wbeckler . Hello team - appreciate any help on the issue. We will also test on latest 2.8 release and confirm if this still continues to be a bug.

lezzago commented 1 year ago

@wbeckler @ravikiranvuppu Is the issue still there if you call the get alerts api? I believe this issue should be fixed in 2.7 with this PR

ravikiranvuppu commented 1 year ago

Hi @lezzago .. looks like get alerts api works fine. however that exception and error is still thrown if you set "plugins.alerting.filter_by_backend_roles" : "true" . This seems to be same behavior with 2.5 up till 2.8 latest release.

image image

but as you said it does look like it's throwing this error when you click on monitor or alerts ui and is trying to get / load the related alerts.

ravikiranvuppu commented 1 year ago

Hello @lezzago @wbeckler - wondering if you have had any luck with bug above ? This is preventing us from using the feature and extending this to product teams .. is there a solution that we could try ?

ravikiranvuppu commented 1 year ago

Hi @lezzago - any chance that this bug could get addressed in upcoming 2.9 release ?

techrna commented 1 year ago

Hi @lezzago I am also facing same problem when we enable "plugins.alerting.filter_by_backend_roles": "true" Please check the debug logs these might help.

2023-07-10T11:52:37,023][DEBUG][o.o.a.t.SecureTransportAction] [opensearch-cluster_manager] User and roles string from thread context: survival|alerting_full_access_tenant|own_index,alerting_full_access_tenant|survival_demo [2023-07-10T11:52:37,024][DEBUG][o.o.a.t.SecureTransportAction] [opensearch-cluster_manager] User and roles string from thread context: null

2023-07-10T11:52:37,023][DEBUG][o.o.a.t.SecureTransportAction] [opensearch-cluster_manager] User and roles string from thread context: survival|alerting_full_access_tenant|own_index,alerting_full_access_tenant|survival_demo
[2023-07-10T11:52:37,024][DEBUG][o.o.a.t.SecureTransportAction] [opensearch-cluster_manager] User and roles string from thread context: null
[2023-07-10T11:52:37,024][ERROR][o.o.a.u.AlertingException] [opensearch-cluster_manager] Alerting error: OpenSearchStatusException[Filter by user backend roles is enabled with security disabled.]
[2023-07-10T11:52:37,024][ERROR][o.o.a.t.TransportGetAlertsAction] [opensearch-cluster_manager] Failed to get alerts
org.opensearch.alerting.util.AlertingException: Filter by user backend roles is enabled with security disabled.
        at org.opensearch.alerting.util.AlertingException$Companion.wrap(AlertingException.kt:70) ~[?:?]
        at org.opensearch.alerting.transport.SecureTransportAction$DefaultImpls.validateUserBackendRoles(SecureTransportAction.kt:82) ~[?:?]
        at org.opensearch.alerting.transport.TransportGetMonitorAction.validateUserBackendRoles(TransportGetMonitorAction.kt:36) ~[?:?]
        at org.opensearch.alerting.transport.TransportGetMonitorAction.doExecute(TransportGetMonitorAction.kt:64) ~[?:?]
        at org.opensearch.alerting.transport.TransportGetMonitorAction.doExecute(TransportGetMonitorAction.kt:36) ~[?:?]
        at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) ~[opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:118) ~[?:?]
        at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) ~[opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:264) ~[?:?]
        at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:149) ~[?:?]
        at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) ~[opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:78) ~[?:?]
        at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) ~[opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.action.support.TransportAction.execute(TransportAction.java:188) ~[opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107) ~[opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110) ~[opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97) ~[opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:472) ~[opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.alerting.transport.TransportGetAlertsAction$resolveAlertsIndexName$2$getMonitorResponse$1.invoke(TransportGetAlertsAction.kt:170) ~[opensearch-alerting-2.7.0.0.jar:2.7.0.0]
        at org.opensearch.alerting.transport.TransportGetAlertsAction$resolveAlertsIndexName$2$getMonitorResponse$1.invoke(TransportGetAlertsAction.kt:169) ~[opensearch-alerting-2.7.0.0.jar:2.7.0.0]
        at org.opensearch.alerting.opensearchapi.OpenSearchExtensionsKt.suspendUntil(OpenSearchExtensions.kt:152) ~[alerting-core-2.7.0.0.jar:?]
        at org.opensearch.alerting.transport.TransportGetAlertsAction$resolveAlertsIndexName$2.invokeSuspend(TransportGetAlertsAction.kt:169) ~[opensearch-alerting-2.7.0.0.jar:2.7.0.0]
        at org.opensearch.alerting.transport.TransportGetAlertsAction$resolveAlertsIndexName$2.invoke(TransportGetAlertsAction.kt) ~[opensearch-alerting-2.7.0.0.jar:2.7.0.0]
        at org.opensearch.alerting.transport.TransportGetAlertsAction$resolveAlertsIndexName$2.invoke(TransportGetAlertsAction.kt) ~[opensearch-alerting-2.7.0.0.jar:2.7.0.0]
        at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturn(Undispatched.kt:91) ~[kotlinx-coroutines-core-1.1.1.jar:?]
        at kotlinx.coroutines.BuildersKt__Builders_commonKt.withContext(Builders.common.kt:146) ~[kotlinx-coroutines-core-1.1.1.jar:?]
        at kotlinx.coroutines.BuildersKt.withContext(Unknown Source) ~[kotlinx-coroutines-core-1.1.1.jar:?]
        at org.opensearch.alerting.transport.TransportGetAlertsAction.resolveAlertsIndexName(TransportGetAlertsAction.kt:161) ~[opensearch-alerting-2.7.0.0.jar:2.7.0.0]
        at org.opensearch.alerting.transport.TransportGetAlertsAction$doExecute$1$1.invokeSuspend(TransportGetAlertsAction.kt:137) [opensearch-alerting-2.7.0.0.jar:2.7.0.0]
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) [kotlin-stdlib-1.6.10.jar:1.6.10-release-923(1.6.10)]
        at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:233) [kotlinx-coroutines-core-1.1.1.jar:?]
        at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:594) [kotlinx-coroutines-core-1.1.1.jar:?]
        at kotlinx.coroutines.scheduling.CoroutineScheduler.access$runSafely(CoroutineScheduler.kt:60) [kotlinx-coroutines-core-1.1.1.jar:?]
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:742) [kotlinx-coroutines-core-1.1.1.jar:?]
Caused by: java.lang.Exception: org.opensearch.OpenSearchStatusException: Filter by user backend roles is enabled with security disabled.
        ... 34 more
[2023-07-10T11:52:37,026][DEBUG][r.suppressed             ] [opensearch-cluster_manager] path: /_plugins/_alerting/monitors/alerts, params: {severityLevel=ALL, sortString=start_time, startIndex=0, searchString=, monitorId=u7WQP4kBKP5QTD2qlAZp, size=800, sortOrder=asc, alertState=ALL}
org.opensearch.alerting.util.AlertingException: Filter by user backend roles is enabled with security disabled.
        at org.opensearch.alerting.util.AlertingException$Companion.wrap(AlertingException.kt:70) ~[?:?]
        at org.opensearch.alerting.transport.SecureTransportAction$DefaultImpls.validateUserBackendRoles(SecureTransportAction.kt:82) ~[?:?]
        at org.opensearch.alerting.transport.TransportGetMonitorAction.validateUserBackendRoles(TransportGetMonitorAction.kt:36) ~[?:?]
        at org.opensearch.alerting.transport.TransportGetMonitorAction.doExecute(TransportGetMonitorAction.kt:64) ~[?:?]
        at org.opensearch.alerting.transport.TransportGetMonitorAction.doExecute(TransportGetMonitorAction.kt:36) ~[?:?]
        at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) ~[opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:118) ~[?:?]
        at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) ~[opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:264) ~[?:?]
        at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:149) ~[?:?]
        at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) ~[opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:78) ~[?:?]
        at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) ~[opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.action.support.TransportAction.execute(TransportAction.java:188) ~[opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107) ~[opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110) ~[opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97) ~[opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:472) ~[opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.alerting.transport.TransportGetAlertsAction$resolveAlertsIndexName$2$getMonitorResponse$1.invoke(TransportGetAlertsAction.kt:170) ~[opensearch-alerting-2.7.0.0.jar:2.7.0.0]
        at org.opensearch.alerting.transport.TransportGetAlertsAction$resolveAlertsIndexName$2$getMonitorResponse$1.invoke(TransportGetAlertsAction.kt:169) ~[opensearch-alerting-2.7.0.0.jar:2.7.0.0]
        at org.opensearch.alerting.opensearchapi.OpenSearchExtensionsKt.suspendUntil(OpenSearchExtensions.kt:152) ~[alerting-core-2.7.0.0.jar:?]
        at org.opensearch.alerting.transport.TransportGetAlertsAction$resolveAlertsIndexName$2.invokeSuspend(TransportGetAlertsAction.kt:169) ~[opensearch-alerting-2.7.0.0.jar:2.7.0.0]
        at org.opensearch.alerting.transport.TransportGetAlertsAction$resolveAlertsIndexName$2.invoke(TransportGetAlertsAction.kt) ~[opensearch-alerting-2.7.0.0.jar:2.7.0.0]
        at org.opensearch.alerting.transport.TransportGetAlertsAction$resolveAlertsIndexName$2.invoke(TransportGetAlertsAction.kt) ~[opensearch-alerting-2.7.0.0.jar:2.7.0.0]
        at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturn(Undispatched.kt:91) ~[kotlinx-coroutines-core-1.1.1.jar:?]
        at kotlinx.coroutines.BuildersKt__Builders_commonKt.withContext(Builders.common.kt:146) ~[kotlinx-coroutines-core-1.1.1.jar:?]
        at kotlinx.coroutines.BuildersKt.withContext(Unknown Source) ~[kotlinx-coroutines-core-1.1.1.jar:?]
        at org.opensearch.alerting.transport.TransportGetAlertsAction.resolveAlertsIndexName(TransportGetAlertsAction.kt:161) ~[opensearch-alerting-2.7.0.0.jar:2.7.0.0]
        at org.opensearch.alerting.transport.TransportGetAlertsAction$doExecute$1$1.invokeSuspend(TransportGetAlertsAction.kt:137) [opensearch-alerting-2.7.0.0.jar:2.7.0.0]
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) [kotlin-stdlib-1.6.10.jar:1.6.10-release-923(1.6.10)]
        at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:233) [kotlinx-coroutines-core-1.1.1.jar:?]
        at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:594) [kotlinx-coroutines-core-1.1.1.jar:?]
        at kotlinx.coroutines.scheduling.CoroutineScheduler.access$runSafely(CoroutineScheduler.kt:60) [kotlinx-coroutines-core-1.1.1.jar:?]
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:742) [kotlinx-coroutines-core-1.1.1.jar:?]
Caused by: java.lang.Exception: org.opensearch.OpenSearchStatusException: Filter by user backend roles is enabled with security disabled.
        ... 34 more
lezzago commented 1 year ago

@techrna Thanks for those logs. Interesting to see the users and roles string from the thread context is null. The thread context must be getting reset.

lezzago commented 1 year ago

Seems like the issue is caused by this line of code.

This is due to the context being stashed and its calling another transport call that sees there is no thread context, which causes the error.

lezzago commented 1 year ago

I have a PR to fix this issue. This fix should be in the 2.9 release.

ravikiranvuppu commented 1 year ago

thank you @lezzago . Will look forward for 2.9 release with fix.

tallyoh commented 1 year ago

Did this fix make the 2.9 release? I'm still experiencing the issue described above. Thank you.

lezzago commented 1 year ago

Yes this fix made it in the 2.9 release. Can you share the stack trace of the error from your logs?

tallyoh commented 1 year ago

Thank you for the prompt response. I hope this is whatyou are asking for. If not please let me know.

[edited] to include the alerting exception. I had pasted the wrong block originally.

`org.opensearch.alerting.util.AlertingException: Filter by user backend roles is enabled with security disabled. at org.opensearch.alerting.util.AlertingException$Companion.wrap(AlertingException.kt:70) ~[?:?] at org.opensearch.alerting.transport.SecureTransportAction$DefaultImpls.validateUserBackendRoles(SecureTransportAction.kt:82) ~[?:?] at org.opensearch.alerting.transport.TransportIndexMonitorAction.validateUserBackendRoles(TransportIndexMonitorAction.kt:88) ~[?:?] at org.opensearch.alerting.transport.TransportIndexMonitorAction.doExecute(TransportIndexMonitorAction.kt:135) ~[?:?] at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) ~[opensearch-2.9.0.jar:2.9.0] at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:118) ~[?:?] at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) ~[opensearch-2.9.0.jar:2.9.0] at org.opensearch.indexmanagement.controlcenter.notification.filter.IndexOperationActionFilter.apply(IndexOperationActionFilter.kt:39) ~[?:?] at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) ~[opensearch-2.9.0.jar:2.9.0] at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:78) ~[?:?] at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) ~[opensearch-2.9.0.jar:2.9.0] at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:320) ~[?:?] at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:165) ~[?:?] at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) ~[opensearch-2.9.0.jar:2.9.0] at org.opensearch.action.support.TransportAction.execute(TransportAction.java:188) ~[opensearch-2.9.0.jar:2.9.0] at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107) ~[opensearch-2.9.0.jar:2.9.0] at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110) ~[opensearch-2.9.0.jar:2.9.0] at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97) ~[opensearch-2.9.0.jar:2.9.0] at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:476) ~[opensearch-2.9.0.jar:2.9.0] at org.opensearch.commons.alerting.AlertingPluginInterface.indexMonitor(AlertingPluginInterface.kt:55) ~[?:?] at org.opensearch.securityanalytics.transport.TransportIndexDetectorAction.createMonitorFromQueries(TransportIndexDetectorAction.java:302) ~[?:?] at org.opensearch.securityanalytics.transport.TransportIndexDetectorAction$AsyncIndexDetectorsAction$10.onResponse(TransportIndexDetectorAction.java:1203) ~[?:?] at org.opensearch.securityanalytics.transport.TransportIndexDetectorAction$AsyncIndexDetectorsAction$10.onResponse(TransportIndexDetectorAction.java:1180) ~[?:?] at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:113) ~[opensearch-2.9.0.jar:2.9.0] at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:107) ~[opensearch-2.9.0.jar:2.9.0] at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionListener.onResponse(PerformanceAnalyzerActionListener.java:55) ~[?:?] at org.opensearch.action.search.TransportSearchAction.lambda$executeRequest$0(TransportSearchAction.java:398) ~[opensearch-2.9.0.jar:2.9.0] at org.opensearch.action.ActionListener$1.onResponse(ActionListener.java:80) ~[opensearch-2.9.0.jar:2.9.0] at org.opensearch.action.ActionListener$5.onResponse(ActionListener.java:266) ~[opensearch-2.9.0.jar:2.9.0] at org.opensearch.action.search.AbstractSearchAsyncAction.sendSearchResponse(AbstractSearchAsyncAction.java:659) ~[opensearch-2.9.0.jar:2.9.0] at org.opensearch.action.search.ExpandSearchPhase.run(ExpandSearchPhase.java:132) ~[opensearch-2.9.0.jar:2.9.0] at org.opensearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:428) ~[opensearch-2.9.0.jar:2.9.0] at org.opensearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:422) ~[opensearch-2.9.0.jar:2.9.0] at org.opensearch.action.search.FetchSearchPhase.moveToNextPhase(FetchSearchPhase.java:299) ~[opensearch-2.9.0.jar:2.9.0] at org.opensearch.action.search.FetchSearchPhase.lambda$innerRun$1(FetchSearchPhase.java:139) ~[opensearch-2.9.0.jar:2.9.0] at org.opensearch.action.search.FetchSearchPhase.innerRun(FetchSearchPhase.java:151) ~[opensearch-2.9.0.jar:2.9.0] at org.opensearch.action.search.FetchSearchPhase$1.doRun(FetchSearchPhase.java:123) ~[opensearch-2.9.0.jar:2.9.0] at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.9.0.jar:2.9.0] at org.opensearch.threadpool.TaskAwareRunnable.doRun(TaskAwareRunnable.java:78) ~[opensearch-2.9.0.jar:2.9.0] at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.9.0.jar:2.9.0] at org.opensearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:59) ~[opensearch-2.9.0.jar:2.9.0] at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:908) [opensearch-2.9.0.jar:2.9.0] at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.9.0.jar:2.9.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?] at java.lang.Thread.run(Thread.java:833) [?:?] Caused by: java.lang.Exception: org.opensearch.OpenSearchStatusException: Filter by user backend roles is enabled with security disabled.

[2023-08-29T19:19:54,356][INFO ][o.o.s.t.SecureTransportAction] [oslab-data-01 ] User and roles string from thread context: charles@|siemlab_admin|own_index,all_access|elkan [2023-08-29T19:19:56,575][INFO ][o.o.s.t.TransportIndexDetectorAction] [oslab-data-01 ] Updated .opensearch-sap-detectors-config with mappings. [2023-08-29T19:19:56,642][ERROR][o.o.a.u.AlertingException] [oslab-data-01 ] Alerting error: OpenSearchStatusException[Filter by user backend roles is enabled with security disabled.] [2023-08-29T19:19:56,645][ERROR][o.o.s.u.SecurityAnalyticsException] [oslab-data-01 ] Security Analytics error: org.opensearch.alerting.util.AlertingException: Filter by user backend roles is enabled with security disabled. at org.opensearch.alerting.util.AlertingException$Companion.wrap(AlertingException.kt:70) ~[?:?] at org.opensearch.alerting.transport.SecureTransportAction$DefaultImpls.validateUserBackendRoles(SecureTransportAction.kt:82) ~[?:?] at org.opensearch.alerting.transport.TransportIndexMonitorAction.validateUserBackendRoles(TransportIndexMonitorAction.kt:88) ~[?:?] at org.opensearch.alerting.transport.TransportIndexMonitorAction.doExecute(TransportIndexMonitorAction.kt:135) ~[?:?] at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) ~[opensearch-2.9.0.jar:2.9.0] at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:118) ~[?:?] at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) ~[opensearch-2.9.0.jar:2.9.0] at org.opensearch.indexmanagement.controlcenter.notification.filter.IndexOperationActionFilter.apply(IndexOperationActionFilter.kt:39) ~[?:?] at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) ~[opensearch-2.9.0.jar:2.9.0] at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:78) ~[?:?] at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) ~[opensearch-2.9.0.jar:2.9.0] at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:320) ~[?:?] at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:165) ~[?:?] at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) ~[opensearch-2.9.0.jar:2.9.0] at org.opensearch.action.support.TransportAction.execute(TransportAction.java:188) ~[opensearch-2.9.0.jar:2.9.0] at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107) ~[opensearch-2.9.0.jar:2.9.0] at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110) ~[opensearch-2.9.0.jar:2.9.0] at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97) ~[opensearch-2.9.0.jar:2.9.0] at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:476) ~[opensearch-2.9.0.jar:2.9.0] at org.opensearch.commons.alerting.AlertingPluginInterface.indexMonitor(AlertingPluginInterface.kt:55) ~[?:?] at org.opensearch.securityanalytics.transport.TransportIndexDetectorAction.executeMonitorActionRequest(TransportIndexDetectorAction.java:692) ~[?:?] at org.opensearch.securityanalytics.transport.TransportIndexDetectorAction.lambda$updateAlertingMonitors$11(TransportIndexDetectorAction.java:459) ~[?:?] at org.opensearch.action.ActionListener$1.onResponse(ActionListener.java:80) ~[opensearch-2.9.0.jar:2.9.0] at org.opensearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:126) ~[opensearch-2.9.0.jar:2.9.0] at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.9.0.jar:2.9.0] at org.opensearch.common.util.concurrent.OpenSearchExecutors$DirectExecutorService.execute(OpenSearchExecutors.java:343) ~[opensearch-2.9.0.jar:2.9.0] at org.opensearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:120) ~[opensearch-2.9.0.jar:2.9.0] at org.opensearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:82) ~[opensearch-2.9.0.jar:2.9.0] at org.opensearch.action.StepListener.whenComplete(StepListener.java:93) ~[opensearch-2.9.0.jar:2.9.0] at org.opensearch.securityanalytics.transport.TransportIndexDetectorAction.updateAlertingMonitors(TransportIndexDetectorAction.java:453) ~[?:?] at org.opensearch.securityanalytics.transport.TransportIndexDetectorAction.updateMonitorFromQueries(TransportIndexDetectorAction.java:423) ~[?:?] at org.opensearch.securityanalytics.transport.TransportIndexDetectorAction$AsyncIndexDetectorsAction$10.onResponse(TransportIndexDetectorAction.java:1205) ~[?:?] at org.opensearch.securityanalytics.transport.TransportIndexDetectorAction$AsyncIndexDetectorsAction$10.onResponse(TransportIndexDetectorAction.java:1180) ~[?:?] at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:113) ~[opensearch-2.9.0.jar:2.9.0] at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:107) ~[opensearch-2.9.0.jar:2.9.0] at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionListener.onResponse(PerformanceAnalyzerActionListener.java:55) ~[?:?] at org.opensearch.action.search.TransportSearchAction.lambda$executeRequest$0(TransportSearchAction.java:398) ~[opensearch-2.9.0.jar:2.9.0] at org.opensearch.action.ActionListener$1.onResponse(ActionListener.java:80) ~[opensearch-2.9.0.jar:2.9.0] at org.opensearch.action.ActionListener$5.onResponse(ActionListener.java:266) ~[opensearch-2.9.0.jar:2.9.0] at org.opensearch.action.search.AbstractSearchAsyncAction.sendSearchResponse(AbstractSearchAsyncAction.java:659) ~[opensearch-2.9.0.jar:2.9.0] at org.opensearch.action.search.ExpandSearchPhase.run(ExpandSearchPhase.java:132) ~[opensearch-2.9.0.jar:2.9.0] at org.opensearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:428) ~[opensearch-2.9.0.jar:2.9.0] at org.opensearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:422) ~[opensearch-2.9.0.jar:2.9.0] at org.opensearch.action.search.FetchSearchPhase.moveToNextPhase(FetchSearchPhase.java:299) ~[opensearch-2.9.0.jar:2.9.0] at org.opensearch.action.search.FetchSearchPhase.lambda$innerRun$1(FetchSearchPhase.java:139) ~[opensearch-2.9.0.jar:2.9.0] at org.opensearch.action.search.FetchSearchPhase.innerRun(FetchSearchPhase.java:151) ~[opensearch-2.9.0.jar:2.9.0] at org.opensearch.action.search.FetchSearchPhase$1.doRun(FetchSearchPhase.java:123) ~[opensearch-2.9.0.jar:2.9.0] at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.9.0.jar:2.9.0] at org.opensearch.threadpool.TaskAwareRunnable.doRun(TaskAwareRunnable.java:78) ~[opensearch-2.9.0.jar:2.9.0] at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.9.0.jar:2.9.0] at org.opensearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:59) ~[opensearch-2.9.0.jar:2.9.0] at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:908) [opensearch-2.9.0.jar:2.9.0] at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.9.0.jar:2.9.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?] at java.lang.Thread.run(Thread.java:833) [?:?] Caused by: java.lang.Exception: org.opensearch.OpenSearchStatusException: Filter by user backend roles is enabled with security disabled. ... 56 more [2023-08-29T19:19:56,651][WARN ][r.suppressed ] [oslab-data-01 ] path: /_plugins/_security_analytics/detectors/Q0kCPooBe5eiFOPCfMTm, params: {detector_id=Q0kCPooBe5eiFOPCfMTm} org.opensearch.securityanalytics.util.SecurityAnalyticsException: Filter by user backend roles is enabled with security disabled. at org.opensearch.securityanalytics.util.SecurityAnalyticsException.wrap(SecurityAnalyticsException.java:51) ~[?:?] at org.opensearch.securityanalytics.transport.TransportIndexDetectorAction$AsyncIndexDetectorsAction.lambda$finishHim$0(TransportIndexDetectorAction.java:1268) ~[?:?] at org.opensearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:73) [opensearch-2.9.0.jar:2.9.0] at org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:88) ~[opensearch-2.9.0.jar:2.9.0] at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:908) [opensearch-2.9.0.jar:2.9.0] at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.9.0.jar:2.9.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?] at java.lang.Thread.run(Thread.java:833) [?:?] Caused by: java.lang.Exception: org.opensearch.alerting.util.AlertingException: Filter by user backend roles is enabled with security disabled.`

techrna commented 1 year ago

@tallyoh @lezzago Note : Previous error stack which i had shared was related to creating normal alerts flow with notification in multi tenancy. We have tested it there no issue in creating the alerts from Alerting menu.

Currently the issue i can see and we are also facing it is when creating alerts from "Security Analytics error"


[security_analytics_exception] Filter by user backend roles is enabled with security disabled.
lezzago commented 1 year ago

@tallyoh @techrna

This seems to be a Security Analytics plugin issue with how they are integrating with the Alerting plugin. Please create a bug issue in the Security-Analytics plugin here and include the stack trace as well as the steps done to get the error.

Sreekanth-hubs commented 1 month ago

https://forum.opensearch.org/t/alerting-backend-roles-not-working-even-after-enabling-the-backend-roles/21353

Can anyone please help me on this above one.