When trying to create a monitor in the coordinating cluster, only the indices located in the coordinating cluster can be selected. The selector does not show the indices of the remote clusters.
Just for testing purposes, we are using the *:* index pattern
When trying to create the monitor in the coordinating cluster ( Go to Alerting > Monitors > Create monitor), the selector does not show the indices of the remote clusters, only the ones located in the coordinating cluster.
We expect that the remote clusters indices can be selected from the coordinating cluster.
As a workaround, we tried creating the monitor using the alerting API in the coordinating cluster. In this way, we were able to add the indices of the remote clusters in the monitor.
However, although the monitor was created, it does not work properly since it says there is an error when collecting inputs:
Full odfe error log
```
[2021-07-21T11:41:53,793][INFO ][c.a.o.a.MonitorRunner ] [] Error collecting inputs for monitor: _BPeyHoB4vRRwQ0XXHel
org.elasticsearch.transport.RemoteTransportException: [error while communicating with remote cluster []]
Caused by: org.elasticsearch.transport.RemoteTransportException: [][:][indices:data/read/search]
Caused by: org.elasticsearch.ElasticsearchSecurityException: no permissions for [indices:data/read/search] and User [name=plugin, backend_roles=[], requestedTenant=null]
at com.amazon.opendistroforelasticsearch.security.filter.OpenDistroSecurityFilter.apply0(OpenDistroSecurityFilter.java:349) ~[?:?]
at com.amazon.opendistroforelasticsearch.security.filter.OpenDistroSecurityFilter.apply(OpenDistroSecurityFilter.java:151) ~[?:?]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177) ~[elasticsearch-7.10.2.jar:7.10.2]
at com.amazon.opendistro.elasticsearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:77) ~[?:?]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:155) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:64) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:60) ~[elasticsearch-7.10.2.jar:7.10.2]
at com.amazon.opendistro.elasticsearch.performanceanalyzer.transport.PerformanceAnalyzerTransportRequestHandler.messageReceived(PerformanceAnalyzerTransportRequestHandler.java:49) ~[?:?]
at com.amazon.opendistroforelasticsearch.security.ssl.transport.OpenDistroSecuritySSLRequestHandler.messageReceivedDecorate(OpenDistroSecuritySSLRequestHandler.java:182) ~[?:?]
at com.amazon.opendistroforelasticsearch.security.transport.OpenDistroSecurityRequestHandler.messageReceivedDecorate(OpenDistroSecurityRequestHandler.java:293) ~[?:?]
at com.amazon.opendistroforelasticsearch.security.ssl.transport.OpenDistroSecuritySSLRequestHandler.messageReceived(OpenDistroSecuritySSLRequestHandler.java:142) ~[?:?]
at com.amazon.opendistroforelasticsearch.security.OpenDistroSecurityPlugin$7$1.messageReceived(OpenDistroSecurityPlugin.java:639) ~[?:?]
at com.amazon.opendistroforelasticsearch.indexmanagement.rollup.interceptor.RollupInterceptor$interceptHandler$1.messageReceived(RollupInterceptor.kt:124) ~[?:?]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:207) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:107) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:89) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:700) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1518) ~[?:?]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1267) ~[?:?]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:832) ~[?:?]
[2021-07-21T11:41:53,816][INFO ][c.a.o.a.MonitorRunner ] [] Error running script for monitor _BPeyHoB4vRRwQ0XXHel, trigger: -hPeyHoB4vRRwQ0XXHeR
org.elasticsearch.script.ScriptException: runtime error
at org.elasticsearch.painless.PainlessScript.convertToScriptException(PainlessScript.java:96) ~[lang-painless-7.10.2.jar:7.10.2]
at org.elasticsearch.painless.PainlessScript$Script.execute(ctx.results[0].hits.total.value > 0:1) ~[?:?]
at com.amazon.opendistroforelasticsearch.alerting.MonitorRunner.runTrigger(MonitorRunner.kt:410) [opendistro-alerting-1.13.1.0.jar:1.13.1.0]
at com.amazon.opendistroforelasticsearch.alerting.MonitorRunner.runMonitor(MonitorRunner.kt:245) [opendistro-alerting-1.13.1.0.jar:1.13.1.0]
at com.amazon.opendistroforelasticsearch.alerting.MonitorRunner$runMonitor$1.invokeSuspend(MonitorRunner.kt) [opendistro-alerting-1.13.1.0.jar:1.13.1.0]
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) [kotlin-stdlib-1.3.72.jar:1.3.72-release-468 (1.3.72)]
at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:233) [kotlinx-coroutines-core-1.1.1.jar:?]
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:594) [kotlinx-coroutines-core-1.1.1.jar:?]
at kotlinx.coroutines.scheduling.CoroutineScheduler.access$runSafely(CoroutineScheduler.kt:60) [kotlinx-coroutines-core-1.1.1.jar:?]
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:742) [kotlinx-coroutines-core-1.1.1.jar:?]
Caused by: java.lang.IndexOutOfBoundsException: Empty list doesn't contain element at index 0.
at kotlin.collections.EmptyList.get(Collections.kt:35) ~[?:?]
at kotlin.collections.EmptyList.get(Collections.kt:23) ~[?:?]
at org.elasticsearch.painless.PainlessScript$Script.execute(ctx.results[0].hits.total.value > 0:12) ~[?:?]
... 8 more
```
We have suppressed the committed information
We would like to highlight the following line:
Caused by: org.elasticsearch.ElasticsearchSecurityException: no permissions for [indices:data/read/search] and User [name=plugin, backend_roles=[], requestedTenant=null]
Hi Opendistro team!
Since Open Distro
1.12
version, we have been facing some cross-cluster search issues (for instance Cannot create Index Patterns of remote clusters - Open Distro 1.12 version). Most of them were fixed in Opendistro1.13.2
version, however, this problem still remains in the alerting module.Describe the bug
Opendistro version:
1.13.2
When trying to create a monitor in the coordinating cluster, only the indices located in the coordinating cluster can be selected. The selector does not show the indices of the remote clusters.
When trying to create the monitor in the coordinating cluster ( Go to Alerting > Monitors > Create monitor), the selector does not show the indices of the remote clusters, only the ones located in the coordinating cluster.
We expect that the remote clusters indices can be selected from the coordinating cluster.
As a workaround, we tried creating the monitor using the alerting API in the coordinating cluster. In this way, we were able to add the indices of the remote clusters in the monitor.
Create monitor request
``` curl -k -uHowever, although the monitor was created, it does not work properly since it says there is an error when collecting inputs:
Full odfe error log
``` [2021-07-21T11:41:53,793][INFO ][c.a.o.a.MonitorRunner ] [We would like to highlight the following line:
We did not create the
plugin
user.Is it a known issue? Is there any workaround?
Best regards!