We're running Elastic detection on a CCS environment. From time to time EQL rules are failing with the following message:
An error occurred during rule execution: message: "search_phase_execution_exception: [null_pointer_exception] Reason: Cannot invoke "org.elasticsearch.index.cache.IndexCache.bitsetFilterCache()" because "this.indexCache" is null"
Based on a discussion with @romseygeek I'm opening a bug here.
stacktrace
```
Failed to execute phase [query], Partial shards failure; shardFailures {[gpINPd3XTnWIFtBTTKhL3g][infosec_greysector:endgame-4.60.0-000223][1]: org.elasticsearch.transport.RemoteTransportException: [grey-sector-es-data-cold-2][10.64.33.7:9300][internal:transport/proxy/indices:data/read/search[phase/query]]
Caused by: org.elasticsearch.transport.RemoteTransportException: [grey-sector-es-data-cold-10][10.64.8.6:9300][indices:data/read/search[phase/query]]
Caused by: java.lang.NullPointerException: Cannot invoke "org.elasticsearch.index.cache.IndexCache.bitsetFilterCache()" because "this.indexCache" is null
at org.elasticsearch.index.IndexService.newSearchExecutionContext(IndexService.java:637)
at org.elasticsearch.search.DefaultSearchContext.(DefaultSearchContext.java:163)
at org.elasticsearch.search.SearchService.createSearchContext(SearchService.java:1033)
at org.elasticsearch.search.SearchService.createContext(SearchService.java:981)
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:630)
at org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:495)
at org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:50)
at org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:47)
at org.elasticsearch.action.ActionRunnable$3.doRun(ActionRunnable.java:72)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:917)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.lang.Thread.run(Thread.java:1589)
}{[Oj8ystMwTg6C2k5pXb6BKA][infosec_greysector:endgame-4.60.0-000223][2]: org.elasticsearch.transport.RemoteTransportException: [grey-sector-es-data-hot-5][10.64.44.6:9300][internal:transport/proxy/indices:data/read/search[phase/query]]
Caused by: org.elasticsearch.transport.RemoteTransportException: [grey-sector-es-data-cold-4][10.64.39.6:9300][indices:data/read/search[phase/query]]
Caused by: java.lang.NullPointerException: Cannot invoke "org.elasticsearch.index.cache.IndexCache.bitsetFilterCache()" because "this.indexCache" is null
at org.elasticsearch.index.IndexService.newSearchExecutionContext(IndexService.java:637)
at org.elasticsearch.search.DefaultSearchContext.(DefaultSearchContext.java:163)
at org.elasticsearch.search.SearchService.createSearchContext(SearchService.java:1033)
at org.elasticsearch.search.SearchService.createContext(SearchService.java:981)
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:630)
at org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:495)
at org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:50)
at org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:47)
at org.elasticsearch.action.ActionRunnable$3.doRun(ActionRunnable.java:72)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:917)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.lang.Thread.run(Thread.java:1589)
}
at org.elasticsearch.server@8.6.1/org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:728)
at org.elasticsearch.server@8.6.1/org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:432)
at org.elasticsearch.server@8.6.1/org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:760)
at org.elasticsearch.server@8.6.1/org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:512)
at org.elasticsearch.server@8.6.1/org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:349)
at org.elasticsearch.server@8.6.1/org.elasticsearch.action.ActionListener$Delegating.onFailure(ActionListener.java:92)
at org.elasticsearch.server@8.6.1/org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:48)
at org.elasticsearch.server@8.6.1/org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:642)
at org.elasticsearch.server@8.6.1/org.elasticsearch.transport.TransportService.handleSendRequestException(TransportService.java:794)
at org.elasticsearch.server@8.6.1/org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:785)
at org.elasticsearch.server@8.6.1/org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:859)
at org.elasticsearch.server@8.6.1/org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:847)
at org.elasticsearch.server@8.6.1/org.elasticsearch.action.search.SearchTransportService.sendExecuteQuery(SearchTransportService.java:241)
at org.elasticsearch.server@8.6.1/org.elasticsearch.action.search.SearchQueryThenFetchAsyncAction.executePhaseOnShard(SearchQueryThenFetchAsyncAction.java:92)
at org.elasticsearch.server@8.6.1/org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$4(AbstractSearchAsyncAction.java:334)
at org.elasticsearch.server@8.6.1/org.elasticsearch.action.search.AbstractSearchAsyncAction$PendingExecutions.tryRun(AbstractSearchAsyncAction.java:858)
at org.elasticsearch.server@8.6.1/org.elasticsearch.action.search.AbstractSearchAsyncAction.performPhaseOnShard(AbstractSearchAsyncAction.java:368)
at org.elasticsearch.server@8.6.1/org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:520)
at org.elasticsearch.server@8.6.1/org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:349)
at org.elasticsearch.server@8.6.1/org.elasticsearch.action.ActionListener$Delegating.onFailure(ActionListener.java:92)
at org.elasticsearch.server@8.6.1/org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:48)
at org.elasticsearch.server@8.6.1/org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:642)
at org.elasticsearch.server@8.6.1/org.elasticsearch.transport.TransportService$UnregisterChildTransportResponseHandler.handleException(TransportService.java:1646)
at org.elasticsearch.server@8.6.1/org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1372)
at org.elasticsearch.server@8.6.1/org.elasticsearch.transport.InboundHandler.doHandleException(InboundHandler.java:410)
at org.elasticsearch.server@8.6.1/org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:397)
at org.elasticsearch.server@8.6.1/org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:388)
at org.elasticsearch.server@8.6.1/org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:141)
at org.elasticsearch.server@8.6.1/org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:95)
at org.elasticsearch.server@8.6.1/org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:808)
at org.elasticsearch.server@8.6.1/org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:149)
at org.elasticsearch.server@8.6.1/org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:121)
at org.elasticsearch.server@8.6.1/org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:86)
at org.elasticsearch.transport.netty4@8.6.1/org.elasticsearch.transport.netty4.Netty4MessageInboundHandler.channelRead(Netty4MessageInboundHandler.java:63)
at io.netty.transport@4.1.84.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.transport@4.1.84.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.transport@4.1.84.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler@4.1.84.Final/io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280)
at io.netty.transport@4.1.84.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
at io.netty.transport@4.1.84.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.transport@4.1.84.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.codec@4.1.84.Final/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
at io.netty.transport@4.1.84.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.transport@4.1.84.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.transport@4.1.84.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler@4.1.84.Final/io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1373)
at io.netty.handler@4.1.84.Final/io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1236)
at io.netty.handler@4.1.84.Final/io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285)
at io.netty.codec@4.1.84.Final/io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:519)
at io.netty.codec@4.1.84.Final/io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:458)
at io.netty.codec@4.1.84.Final/io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:280)
at io.netty.transport@4.1.84.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.transport@4.1.84.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.transport@4.1.84.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.transport@4.1.84.Final/io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.transport@4.1.84.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
at io.netty.transport@4.1.84.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.transport@4.1.84.Final/io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.transport@4.1.84.Final/io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at io.netty.transport@4.1.84.Final/io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
at io.netty.transport@4.1.84.Final/io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689)
at io.netty.transport@4.1.84.Final/io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652)
at io.netty.transport@4.1.84.Final/io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
at io.netty.common@4.1.84.Final/io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.common@4.1.84.Final/io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.base/java.lang.Thread.run(Thread.java:1589)
Caused by: org.elasticsearch.ElasticsearchException$1: Cannot invoke "org.elasticsearch.index.cache.IndexCache.bitsetFilterCache()" because "this.indexCache" is null
at org.elasticsearch.server@8.6.1/org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:640)
at org.elasticsearch.server@8.6.1/org.elasticsearch.action.search.SearchPhaseExecutionException.guessRootCauses(SearchPhaseExecutionException.java:150)
at org.elasticsearch.server@8.6.1/org.elasticsearch.action.search.SearchPhaseExecutionException.getCause(SearchPhaseExecutionException.java:95)
at java.base/java.lang.Throwable.lockedPrintStackTrace(Throwable.java:694)
at java.base/java.lang.Throwable.printStackTrace(Throwable.java:673)
at java.base/java.lang.Throwable.printStackTrace(Throwable.java:749)
at ecs.logging.core@1.2.0/co.elastic.logging.EcsJsonSerializer.formatThrowable(EcsJsonSerializer.java:242)
at ecs.logging.core@1.2.0/co.elastic.logging.EcsJsonSerializer.serializeException(EcsJsonSerializer.java:207)
at log4j2.ecs.layout@1.2.0/co.elastic.logging.log4j2.EcsLayout.toText(EcsLayout.java:140)
at log4j2.ecs.layout@1.2.0/co.elastic.logging.log4j2.EcsLayout.encode(EcsLayout.java:115)
at log4j2.ecs.layout@1.2.0/co.elastic.logging.log4j2.EcsLayout.encode(EcsLayout.java:59)
at org.apache.logging.log4j.core@8.6.1/org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(AbstractOutputStreamAppender.java:215)
at org.apache.logging.log4j.core@8.6.1/org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutputStreamAppender.java:208)
at org.apache.logging.log4j.core@8.6.1/org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputStreamAppender.java:199)
at org.apache.logging.log4j.core@8.6.1/org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:161)
at org.apache.logging.log4j.core@8.6.1/org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:134)
at org.apache.logging.log4j.core@8.6.1/org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:125)
at org.apache.logging.log4j.core@8.6.1/org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:89)
at org.apache.logging.log4j.core@8.6.1/org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:683)
at org.apache.logging.log4j.core@8.6.1/org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:641)
at org.apache.logging.log4j.core@8.6.1/org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:624)
at org.apache.logging.log4j.core@8.6.1/org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:560)
at org.apache.logging.log4j.core@8.6.1/org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:82)
at org.apache.logging.log4j.core@8.6.1/org.apache.logging.log4j.core.Logger.log(Logger.java:162)
at org.apache.logging.log4j@2.19.0/org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2205)
at org.apache.logging.log4j@2.19.0/org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2159)
at org.apache.logging.log4j@2.19.0/org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2142)
at org.apache.logging.log4j@2.19.0/org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:2012)
at org.apache.logging.log4j@2.19.0/org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1868)
at org.apache.logging.log4j@2.19.0/org.apache.logging.log4j.spi.AbstractLogger.warn(AbstractLogger.java:2681)
at org.elasticsearch.server@8.6.1/org.elasticsearch.rest.RestResponse.(RestResponse.java:118)
at org.elasticsearch.server@8.6.1/org.elasticsearch.rest.RestResponse.(RestResponse.java:102)
at org.elasticsearch.xpack.eql.plugin.RestEqlSearchAction$1.onFailure(RestEqlSearchAction.java:94)
at org.elasticsearch.server@8.6.1/org.elasticsearch.rest.action.RestCancellableNodeClient$1.onFailure(RestCancellableNodeClient.java:96)
at org.elasticsearch.server@8.6.1/org.elasticsearch.client.internal.node.NodeClient$SafelyWrappedActionListener.onFailure(NodeClient.java:170)
at org.elasticsearch.server@8.6.1/org.elasticsearch.tasks.TaskManager$1.onFailure(TaskManager.java:217)
at org.elasticsearch.server@8.6.1/org.elasticsearch.action.support.ContextPreservingActionListener.onFailure(ContextPreservingActionListener.java:38)
at org.elasticsearch.server@8.6.1/org.elasticsearch.action.ActionListener$Delegating.onFailure(ActionListener.java:92)
at org.elasticsearch.xpack.ql.plugin.TransportActionUtils.lambda$executeRequestWithRetryAttempt$0(TransportActionUtils.java:71)
at org.elasticsearch.server@8.6.1/org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:170)
at org.elasticsearch.xpack.eql.execution.PlanExecutor.lambda$eql$0(PlanExecutor.java:80)
at org.elasticsearch.server@8.6.1/org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:170)
at org.elasticsearch.server@8.6.1/org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:170)
at org.elasticsearch.server@8.6.1/org.elasticsearch.action.ActionListener$Delegating.onFailure(ActionListener.java:92)
at org.elasticsearch.server@8.6.1/org.elasticsearch.action.ActionListener$Delegating.onFailure(ActionListener.java:92)
at org.elasticsearch.server@8.6.1/org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:170)
at org.elasticsearch.server@8.6.1/org.elasticsearch.client.internal.node.NodeClient$SafelyWrappedActionListener.onFailure(NodeClient.java:170)
at org.elasticsearch.server@8.6.1/org.elasticsearch.tasks.TaskManager$1.onFailure(TaskManager.java:217)
at org.elasticsearch.server@8.6.1/org.elasticsearch.action.support.ContextPreservingActionListener.onFailure(ContextPreservingActionListener.java:38)
at org.elasticsearch.server@8.6.1/org.elasticsearch.action.ActionListener$Delegating.onFailure(ActionListener.java:92)
at org.elasticsearch.server@8.6.1/org.elasticsearch.action.ActionListener$Delegating.onFailure(ActionListener.java:92)
at org.elasticsearch.server@8.6.1/org.elasticsearch.action.ActionListener$RunAfterActionListener.onFailure(ActionListener.java:358)
at org.elasticsearch.server@8.6.1/org.elasticsearch.action.search.AbstractSearchAsyncAction.raisePhaseFailure(AbstractSearchAsyncAction.java:751)
... 66 more
Caused by: java.lang.NullPointerException: Cannot invoke "org.elasticsearch.index.cache.IndexCache.bitsetFilterCache()" because "this.indexCache" is null
at org.elasticsearch.index.IndexService.newSearchExecutionContext(IndexService.java:637)
at org.elasticsearch.search.DefaultSearchContext.(DefaultSearchContext.java:163)
at org.elasticsearch.search.SearchService.createSearchContext(SearchService.java:1033)
at org.elasticsearch.search.SearchService.createContext(SearchService.java:981)
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:630)
at org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:495)
at org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:50)
at org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:47)
at org.elasticsearch.action.ActionRunnable$3.doRun(ActionRunnable.java:72)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:917)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.lang.Thread.run(Thread.java:1589)
```
The cold data tier mentioned is on searchable snapshot.
From logs around the error it looks like the index is closed and then re-opened
Steps to Reproduce
We have several EQL detection running on Elastic Security.
The error does not happen on each rules execution and can happen on any rule.
Elasticsearch Version
8.6.1
Installed Plugins
No response
Java Version
bundled
OS Version
ECK on GKE Container-Optimized OS
Problem Description
We're running Elastic detection on a CCS environment. From time to time EQL rules are failing with the following message:
Based on a discussion with @romseygeek I'm opening a bug here.
stacktrace
``` Failed to execute phase [query], Partial shards failure; shardFailures {[gpINPd3XTnWIFtBTTKhL3g][infosec_greysector:endgame-4.60.0-000223][1]: org.elasticsearch.transport.RemoteTransportException: [grey-sector-es-data-cold-2][10.64.33.7:9300][internal:transport/proxy/indices:data/read/search[phase/query]] Caused by: org.elasticsearch.transport.RemoteTransportException: [grey-sector-es-data-cold-10][10.64.8.6:9300][indices:data/read/search[phase/query]] Caused by: java.lang.NullPointerException: Cannot invoke "org.elasticsearch.index.cache.IndexCache.bitsetFilterCache()" because "this.indexCache" is null at org.elasticsearch.index.IndexService.newSearchExecutionContext(IndexService.java:637) at org.elasticsearch.search.DefaultSearchContext.The cold data tier mentioned is on searchable snapshot.
From logs around the error it looks like the index is closed and then re-opened
Steps to Reproduce
We have several EQL detection running on Elastic Security. The error does not happen on each rules execution and can happen on any rule.
Logs (if relevant)
No response