search

elastic / elasticsearch

Free and Open Source, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
1.09k stars 24.83k forks source link

Open PIT API incorrectly suggests using allow_partial_search_results if a shard is unavailable #103165

Open rseldner opened 11 months ago

rseldner commented 11 months ago

Elasticsearch Version

8.11.1

Installed Plugins

No response

Java Version

bundled

OS Version

Ubuntu 20.04.6 LTS (Elastic Cloud)

Problem Description

Open PIT requires for all shards involved to be available. If a shard is unavailable, the error response will suggest using allow_partial_search_results to bypass the error.

Request: POST kibana_sample_data_flights*/_pit?keep_alive=1m Response:

 "caused_by": {
  "type": "search_phase_execution_exception",
  "reason": "Search rejected due to missing shards [[kibana_sample_data_flights_2][0]]. Consider using `allow_partial_search_results` setting to bypass this error.",
   "phase": "open_search_context",

This is misleading as it is not currently possible/supported to use allow_partial_search_results when opening a PIT.

Related: https://github.com/elastic/elasticsearch/issues/96775

Steps to Reproduce

  1. Cause a primary shard to become unavailable. Example on a 2+ data node cluster:
    • Create an index with 2 primaries shards and set total shards per node to 1: 
      PUT <index>
{
"settings": {
"number_of_shards": 2,
"routing.allocation.total_shards_per_node": 1
}
}
    • Shutdown one of the data nodes
  2. Open a PIT POST <index>/_pit?keep_alive=1m

Logs (if relevant)

Full Errors+Stack Traces:

8.7.1 ``` { "error": { "root_cause": [], "type": "search_phase_execution_exception", "reason": "", "phase": "open_search_context", "grouped": true, "failed_shards": [], "caused_by": { "type": "search_phase_execution_exception", "reason": "Search rejected due to missing shards [[kibana_sample_data_flights_2][0]]. Consider using `allow_partial_search_results` setting to bypass this error.", "phase": "open_search_context", "grouped": true, "failed_shards": [], "stack_trace": """Failed to execute phase [open_search_context], Search rejected due to missing shards [[kibana_sample_data_flights_2][0]]. Consider using `allow_partial_search_results` setting to bypass this error. at org.elasticsearch.server@8.7.1/org.elasticsearch.action.search.AbstractSearchAsyncAction.run(AbstractSearchAsyncAction.java:245) at org.elasticsearch.server@8.7.1/org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:469) at org.elasticsearch.server@8.7.1/org.elasticsearch.action.search.AbstractSearchAsyncAction.start(AbstractSearchAsyncAction.java:217) at org.elasticsearch.server@8.7.1/org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:1041) at org.elasticsearch.server@8.7.1/org.elasticsearch.action.search.TransportSearchAction.lambda$executeRequest$5(TransportSearchAction.java:459) at org.elasticsearch.server@8.7.1/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:167) at org.elasticsearch.server@8.7.1/org.elasticsearch.action.search.TransportSearchAction$CCSActionListener.maybeFinish(TransportSearchAction.java:1380) at org.elasticsearch.server@8.7.1/org.elasticsearch.action.search.TransportSearchAction$CCSActionListener.onResponse(TransportSearchAction.java:1345) at org.elasticsearch.server@8.7.1/org.elasticsearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:43) at org.elasticsearch.server@8.7.1/org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1367) at org.elasticsearch.server@8.7.1/org.elasticsearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:400) at org.elasticsearch.server@8.7.1/org.elasticsearch.transport.InboundHandler.handleResponse(InboundHandler.java:349) at org.elasticsearch.server@8.7.1/org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:144) at org.elasticsearch.server@8.7.1/org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:97) at org.elasticsearch.server@8.7.1/org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:829) at org.elasticsearch.server@8.7.1/org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:150) at org.elasticsearch.server@8.7.1/org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:121) at org.elasticsearch.server@8.7.1/org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:86) at org.elasticsearch.transport.netty4@8.7.1/org.elasticsearch.transport.netty4.Netty4MessageInboundHandler.channelRead(Netty4MessageInboundHandler.java:63) at io.netty.transport@4.1.86.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at io.netty.transport@4.1.86.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.transport@4.1.86.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at io.netty.handler@4.1.86.Final/io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) at io.netty.transport@4.1.86.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at io.netty.transport@4.1.86.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.transport@4.1.86.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at io.netty.codec@4.1.86.Final/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) at io.netty.transport@4.1.86.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at io.netty.transport@4.1.86.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.transport@4.1.86.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at io.netty.handler@4.1.86.Final/io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1373) at io.netty.handler@4.1.86.Final/io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1236) at io.netty.handler@4.1.86.Final/io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285) at io.netty.codec@4.1.86.Final/io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) at io.netty.codec@4.1.86.Final/io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) at io.netty.codec@4.1.86.Final/io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) at io.netty.transport@4.1.86.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at io.netty.transport@4.1.86.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.transport@4.1.86.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at io.netty.transport@4.1.86.Final/io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.transport@4.1.86.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at io.netty.transport@4.1.86.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.transport@4.1.86.Final/io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.transport@4.1.86.Final/io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at io.netty.transport@4.1.86.Final/io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at io.netty.transport@4.1.86.Final/io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) at io.netty.transport@4.1.86.Final/io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) at io.netty.transport@4.1.86.Final/io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at io.netty.common@4.1.86.Final/io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at io.netty.common@4.1.86.Final/io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at java.base/java.lang.Thread.run(Thread.java:1623) """ }, "stack_trace": """Failed to execute phase [open_search_context], at org.elasticsearch.server@8.7.1/org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:728) at org.elasticsearch.server@8.7.1/org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:474) at org.elasticsearch.server@8.7.1/org.elasticsearch.action.search.AbstractSearchAsyncAction.start(AbstractSearchAsyncAction.java:217) at org.elasticsearch.server@8.7.1/org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:1041) at org.elasticsearch.server@8.7.1/org.elasticsearch.action.search.TransportSearchAction.lambda$executeRequest$5(TransportSearchAction.java:459) at org.elasticsearch.server@8.7.1/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:167) at org.elasticsearch.server@8.7.1/org.elasticsearch.action.search.TransportSearchAction$CCSActionListener.maybeFinish(TransportSearchAction.java:1380) at org.elasticsearch.server@8.7.1/org.elasticsearch.action.search.TransportSearchAction$CCSActionListener.onResponse(TransportSearchAction.java:1345) at org.elasticsearch.server@8.7.1/org.elasticsearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:43) at org.elasticsearch.server@8.7.1/org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1367) at org.elasticsearch.server@8.7.1/org.elasticsearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:400) at org.elasticsearch.server@8.7.1/org.elasticsearch.transport.InboundHandler.handleResponse(InboundHandler.java:349) at org.elasticsearch.server@8.7.1/org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:144) at org.elasticsearch.server@8.7.1/org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:97) at org.elasticsearch.server@8.7.1/org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:829) at org.elasticsearch.server@8.7.1/org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:150) at org.elasticsearch.server@8.7.1/org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:121) at org.elasticsearch.server@8.7.1/org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:86) at org.elasticsearch.transport.netty4@8.7.1/org.elasticsearch.transport.netty4.Netty4MessageInboundHandler.channelRead(Netty4MessageInboundHandler.java:63) at io.netty.transport@4.1.86.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at io.netty.transport@4.1.86.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.transport@4.1.86.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at io.netty.handler@4.1.86.Final/io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) at io.netty.transport@4.1.86.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at io.netty.transport@4.1.86.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.transport@4.1.86.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at io.netty.codec@4.1.86.Final/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) at io.netty.transport@4.1.86.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at io.netty.transport@4.1.86.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.transport@4.1.86.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at io.netty.handler@4.1.86.Final/io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1373) at io.netty.handler@4.1.86.Final/io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1236) at io.netty.handler@4.1.86.Final/io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285) at io.netty.codec@4.1.86.Final/io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) at io.netty.codec@4.1.86.Final/io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) at io.netty.codec@4.1.86.Final/io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) at io.netty.transport@4.1.86.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at io.netty.transport@4.1.86.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.transport@4.1.86.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at io.netty.transport@4.1.86.Final/io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.transport@4.1.86.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at io.netty.transport@4.1.86.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.transport@4.1.86.Final/io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.transport@4.1.86.Final/io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at io.netty.transport@4.1.86.Final/io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at io.netty.transport@4.1.86.Final/io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) at io.netty.transport@4.1.86.Final/io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) at io.netty.transport@4.1.86.Final/io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at io.netty.common@4.1.86.Final/io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at io.netty.common@4.1.86.Final/io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at java.base/java.lang.Thread.run(Thread.java:1623) Caused by: Failed to execute phase [open_search_context], Search rejected due to missing shards [[kibana_sample_data_flights_2][0]]. Consider using `allow_partial_search_results` setting to bypass this error. at org.elasticsearch.server@8.7.1/org.elasticsearch.action.search.AbstractSearchAsyncAction.run(AbstractSearchAsyncAction.java:245) at org.elasticsearch.server@8.7.1/org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:469) ... 49 more """ }, "status": 503 } ```
8.11.1 ``` { "error": { "root_cause": [], "type": "search_phase_execution_exception", "reason": "", "phase": "indices:data/read/open_point_in_time", "grouped": true, "failed_shards": [], "caused_by": { "type": "search_phase_execution_exception", "reason": "Search rejected due to missing shards [[kibana_sample_data_flights_2][0]]. Consider using `allow_partial_search_results` setting to bypass this error.", "phase": "indices:data/read/open_point_in_time", "grouped": true, "failed_shards": [], "stack_trace": """Failed to execute phase [indices:data/read/open_point_in_time], Search rejected due to missing shards [[kibana_sample_data_flights_2][0]]. Consider using `allow_partial_search_results` setting to bypass this error. at org.elasticsearch.server@8.11.1/org.elasticsearch.action.search.AbstractSearchAsyncAction.run(AbstractSearchAsyncAction.java:249) at org.elasticsearch.server@8.11.1/org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:473) at org.elasticsearch.server@8.11.1/org.elasticsearch.action.search.AbstractSearchAsyncAction.start(AbstractSearchAsyncAction.java:221) at org.elasticsearch.server@8.11.1/org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:1123) at org.elasticsearch.server@8.11.1/org.elasticsearch.action.search.TransportSearchAction.lambda$executeRequest$8(TransportSearchAction.java:415) at org.elasticsearch.server@8.11.1/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) at org.elasticsearch.server@8.11.1/org.elasticsearch.action.search.TransportSearchAction$CCSActionListener.maybeFinish(TransportSearchAction.java:1489) at org.elasticsearch.server@8.11.1/org.elasticsearch.action.search.TransportSearchAction$CCSActionListener.onFailure(TransportSearchAction.java:1475) at org.elasticsearch.server@8.11.1/org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:54) at org.elasticsearch.server@8.11.1/org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1421) at org.elasticsearch.server@8.11.1/org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1421) at org.elasticsearch.server@8.11.1/org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:1305) at org.elasticsearch.server@8.11.1/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983) at org.elasticsearch.server@8.11.1/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) at java.base/java.lang.Thread.run(Thread.java:1583) """ }, "stack_trace": """Failed to execute phase [indices:data/read/open_point_in_time], at org.elasticsearch.server@8.11.1/org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:731) at org.elasticsearch.server@8.11.1/org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:478) at org.elasticsearch.server@8.11.1/org.elasticsearch.action.search.AbstractSearchAsyncAction.start(AbstractSearchAsyncAction.java:221) at org.elasticsearch.server@8.11.1/org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:1123) at org.elasticsearch.server@8.11.1/org.elasticsearch.action.search.TransportSearchAction.lambda$executeRequest$8(TransportSearchAction.java:415) at org.elasticsearch.server@8.11.1/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) at org.elasticsearch.server@8.11.1/org.elasticsearch.action.search.TransportSearchAction$CCSActionListener.maybeFinish(TransportSearchAction.java:1489) at org.elasticsearch.server@8.11.1/org.elasticsearch.action.search.TransportSearchAction$CCSActionListener.onFailure(TransportSearchAction.java:1475) at org.elasticsearch.server@8.11.1/org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:54) at org.elasticsearch.server@8.11.1/org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1421) at org.elasticsearch.server@8.11.1/org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1421) at org.elasticsearch.server@8.11.1/org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:1305) at org.elasticsearch.server@8.11.1/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983) at org.elasticsearch.server@8.11.1/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) at java.base/java.lang.Thread.run(Thread.java:1583) Caused by: Failed to execute phase [indices:data/read/open_point_in_time], Search rejected due to missing shards [[kibana_sample_data_flights_2][0]]. Consider using `allow_partial_search_results` setting to bypass this error. at org.elasticsearch.server@8.11.1/org.elasticsearch.action.search.AbstractSearchAsyncAction.run(AbstractSearchAsyncAction.java:249) at org.elasticsearch.server@8.11.1/org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:473) ... 15 more """ }, "status": 503 } ```
elasticsearchmachine commented 11 months ago

Pinging @elastic/es-search (Team:Search)

elasticsearchmachine commented 3 months ago

Pinging @elastic/es-search-foundations (Team:Search Foundations)