Sampler+Filters Aggregation: array_index_out_of_bounds_exception when no docs match on a shard

Elasticsearch version: 2.3.3 Plugins installed: [analysis-icu, analysis-kuromoji ,analysis-smartcn, analysis-stempel, head, elasticsearch-inquisitor, statsd, whatson, langdetect, delete-by-query, lang-javascript]

JVM version: OpenJDK Java 8

OS version: Debian

Description of the problem including expected versus actual behavior:

Combination of factors results in the error:

filters aggregation containing a sampler aggregation and a significant terms aggregation (not sure sig terms matters here)
Querying an index (maybe even just a shard) where the top level filter excludes all documents from the shard. In this case looking at 2016q2 index with a filter for the past 30 days.
The filters aggregation must have more than one filter in it. If I remove the "t1" filter below, then it will work correctly and return an empty set of results from the 2016q2 index.

Example query:

GET global-temporal-2-2016q2/post/_search
{
  "from": 0,
  "size": 0,
  "fields": [
    "blog_id"
  ],
  "query": {
    "filtered": {
      "query": {
        "match_all": []
      },
      "filter": {
        "bool": {
          "must": [
            {
              "range": {
                "date_gmt": {
                  "gte": "now-30d"
                }
              }
            }
          ]
        }
      }
    }
  },
  "aggregations": {
    "topics": {
      "filters": {
        "filters": {
          "t0": {
            "or": [
              {
                "term": {
                  "title.default": "travel"
                }
              },
              {
                "term": {
                  "content.default": "travel"
                }
              },
              {
                "term": {
                  "tag.name.default": "travel"
                }
              },
              {
                "term": {
                  "category.name.default": "travel"
                }
              }
            ]
          },
          "t1": {
            "or": [
              {
                "term": {
                  "title.default": "photography"
                }
              },
              {
                "term": {
                  "content.default": "photography"
                }
              },
              {
                "term": {
                  "tag.name.default": "photography"
                }
              },
              {
                "term": {
                  "category.name.default": "photography"
                }
              }
            ]
          }
        }
      },
      "aggs": {
        "sample": {
          "sampler": {
            "shard_size": 500
          },
          "aggs": {
            "blogs": {
              "significant_terms": {
                "field": "blog_id",
                "size": 10,
                "gnd": {}
              }
            }
          }
        }
      }
    }
  },
  "timeout": "2s"
}

What the response looks like:

500 response when querying 2016q2 index
200 response querying 2016q3 index
500 response querying 2016q* indices. And we get partial results.

Full Response when querying 2016q2:

{
   "error": {
      "root_cause": [
         {
            "type": "array_index_out_of_bounds_exception",
            "reason": null
         }
      ],
      "type": "search_phase_execution_exception",
      "reason": "all shards failed",
      "phase": "query",
      "grouped": true,
      "failed_shards": [
         {
            "shard": 0,
            "index": "global-temporal-2-2016q2",
            "node": "-sTCWr_OQzuhyHfgIgyANw",
            "reason": {
               "type": "array_index_out_of_bounds_exception",
               "reason": null
            }
         }
      ]
   },
   "status": 500
}

Do you have the stack trace in the logs by any chance?

Indeed.

nested: ArrayIndexOutOfBoundsException; }
    at org.elasticsearch.action.search.AbstractSearchAsyncAction.onFirstPhaseResult(AbstractSearchAsyncAction.java:206)
    at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:152)
    at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:46)
    at org.elasticsearch.transport.netty.MessageChannelHandler.handleException(MessageChannelHandler.java:212)
    at org.elasticsearch.transport.netty.MessageChannelHandler.handlerResponseError(MessageChannelHandler.java:202)
    at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:136)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
    at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
    at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
    at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
    at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
    at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
    at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
    at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: ; nested: ArrayIndexOutOfBoundsException;
    at org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:386)
    at org.elasticsearch.action.search.SearchPhaseExecutionException.guessRootCauses(SearchPhaseExecutionException.java:152)
    at org.elasticsearch.action.search.SearchPhaseExecutionException.getCause(SearchPhaseExecutionException.java:99)
    at java.lang.Throwable.printStackTrace(Throwable.java:665)
    at java.lang.Throwable.printStackTrace(Throwable.java:721)
    at org.apache.log4j.DefaultThrowableRenderer.render(DefaultThrowableRenderer.java:60)
    at org.apache.log4j.spi.ThrowableInformation.getThrowableStrRep(ThrowableInformation.java:87)
    at org.apache.log4j.spi.LoggingEvent.getThrowableStrRep(LoggingEvent.java:413)
    at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:313)
    at org.apache.log4j.WriterAppender.append(WriterAppender.java:162)
    at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
    at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
    at org.apache.log4j.Category.callAppenders(Category.java:206)
    at org.apache.log4j.Category.forcedLog(Category.java:391)
    at org.apache.log4j.Category.log(Category.java:856)
    at org.elasticsearch.common.logging.log4j.Log4jESLogger.internalWarn(Log4jESLogger.java:135)
    at org.elasticsearch.common.logging.support.AbstractESLogger.warn(AbstractESLogger.java:109)
    at org.elasticsearch.rest.BytesRestResponse.convert(BytesRestResponse.java:134)
    at org.elasticsearch.rest.BytesRestResponse.<init>(BytesRestResponse.java:96)
    at org.elasticsearch.rest.BytesRestResponse.<init>(BytesRestResponse.java:87)
    at org.elasticsearch.rest.action.support.RestActionListener.onFailure(RestActionListener.java:60)
    at org.elasticsearch.action.support.TransportAction$1.onFailure(TransportAction.java:95)
    at org.elasticsearch.action.search.AbstractSearchAsyncAction.raiseEarlyFailure(AbstractSearchAsyncAction.java:294)
    ... 28 more
Caused by: java.lang.ArrayIndexOutOfBoundsException

And you don't have anything below the last line? (Caused by: java.lang.ArrayIndexOutOfBoundsException)

Those are the full stack traces that I see. There are the error messages that mostly consist of the full query I pasted above and a bunch of details about the server that I'd need to redact information to post, but I can dig through them if you think that will help.

No further feedback. Please reopen if you have more information and this is still causing issues for you on recent versions of Elasticsearch

Sorry about that. I tried recreating this error and was still able to. We've rebuilt this index twice since this was opened (still on ES 2.3.3 though), but the error still is persisting. I was able to find a stack trace though:

Caused by: java.lang.ArrayIndexOutOfBoundsException: 1
    at org.elasticsearch.common.util.BigArrays$ObjectArrayWrapper.get(BigArrays.java:347)
    at org.elasticsearch.search.aggregations.bucket.BestDocsDeferringCollector.getDocCount(BestDocsDeferringCollector.java:299)
    at org.elasticsearch.search.aggregations.bucket.sampler.SamplerAggregator.buildAggregation(SamplerAggregator.java:167)
    at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.bucketAggregations(BucketsAggregator.java:116)
    at org.elasticsearch.search.aggregations.bucket.filters.FiltersAggregator.buildAggregation(FiltersAggregator.java:114)
    at org.elasticsearch.search.aggregations.AggregationPhase.execute(AggregationPhase.java:167)
    at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:119)
    at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:366)
    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:378)
    at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:368)
    at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:365)
    at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)
    at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:75)
    at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:300)
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

elastic / elasticsearch

Sampler+Filters Aggregation: array_index_out_of_bounds_exception when no docs match on a shard #19891