Profiler for scroll request throws Exception (The collector can only be set once)

[dantalian-pv]

Elasticsearch version (bin/elasticsearch --version):

<dependency>
  <groupId>org.elasticsearch.client</groupId>
  <artifactId>transport</artifactId>
  <version>5.6.4</version>
</dependency>

Plugins installed: [<search-guard-client-version>5.6.4-23</search-guard-client-version>]

JVM version (java -version): openjdk version "1.8.0_151"

OS version (uname -a if on a Unix-like system): Linux 4.4.92-31-default #1 x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior: When I am trying to use scroll for any search query with enabled profile, scroll request throws the next exception:

! java.lang.IllegalStateException: The collector can only be set once.
! at org.elasticsearch.search.profile.query.QueryProfiler.setCollector(QueryProfiler.java:52)
! at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:419)
! ... 15 common frames omitted
! Causing: org.elasticsearch.search.query.QueryPhaseExecutionException: Query Failed [Failed to execute main query]
! at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:447)
! at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:114)
! at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:327)
! at org.elasticsearch.action.search.SearchTransportService$8.messageReceived(SearchTransportService.java:360)
! at org.elasticsearch.action.search.SearchTransportService$8.messageReceived(SearchTransportService.java:357)
! at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceivedDecorate(SearchGuardSSLRequestHandler.java:178)
! at com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:101)
! at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:92)
! at com.floragunn.searchguard.SearchGuardPlugin$3$1.messageReceived(SearchGuardPlugin.java:376)
! at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
! at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:644)
! at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638)
! at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
! at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
! at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
! at java.lang.Thread.run(Thread.java:748)
! Causing: org.elasticsearch.transport.RemoteTransportException: [server1][ip:9300][indices:data/read/search[phase/query/scroll]]
! Causing: org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
! at org.elasticsearch.action.search.SearchScrollAsyncAction.onShardFailure(SearchScrollAsyncAction.java:269)
! at org.elasticsearch.action.search.SearchScrollAsyncAction$1.onFailure(SearchScrollAsyncAction.java:202)
! at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:51)
! at com.floragunn.searchguard.transport.SearchGuardInterceptor$RestoringTransportResponseHandler.handleException(SearchGuardInterceptor.java:163)
! at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1067)
! at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1171)
! at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1149)
! at org.elasticsearch.transport.TransportService$7.onFailure(TransportService.java:655)
! at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.onFailure(ThreadContext.java:623)
! at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:39)
! at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
! at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
! ... 1 common frames omitted

Steps to reproduce:

0. Any index with 2 shard and 2 replicas and >50 documents with any content.

   1. 
      final SearchRequestBuilder reqBuilder = mElasticClient.prepareSearch(
      "some-index")
      .setPreference("session-1")
      .setSearchType(SearchType.DFS_QUERY_THEN_FETCH)
      .setQuery(QueryBuilders.matchAllQuery())
      .setProfile(true)
      .setScroll(new Scroll(TimeValue.timeValueMinutes(1)))
      .setFrom(0);
      SearchResponse response = reqBuilder.get();

final String scrollId = response.getScrollId();

 2.

mElasticClient.prepareSearchScroll(scrollId) .setScroll(TimeValue.timeValueMinutes(1)) .get();

[martijnvg]

Thanks for reporting @dantalian-pv. This is indeed producing an error.

Is there a particular issue why you're profiling a scroll search? I think that profiling the search makes sense (query and aggregations), but subsequent scroll searches don't make a lot of sense.

Shorter reproduction:

PUT /index/doc/1
{}

PUT /index/doc/2
{}

GET /index/_search?scroll=1m
{
  "profile": true,
  "size": 1
}

GET /_search/scroll/{scroll_id}

[dantalian-pv]

On the Java Client there is no possibility to enable/disable a profiler for scroll, so, I assume, If the profiler is enabled for the qery, then profiler is enabled for this scroll (Java Client level, I think).

[martijnvg]

I see, I think profiling should then just be ignored for search scroll invocations.

[polyfractal]

I started to look into this, and while it's technically possible for the search to be profiled but not the subsequent scrolls... the code is kinda messy. The place where we profile doesn't have access to clearing out the search context, so it needs several messy conditionals to disable profiling on the subsequent scrolls.

I think we should just throw an exception if scroll and profile are enabled on the same search, and document the behavior. It shouldn't be a problem for the user, as they can just execute the same search without scroll (same behavior) if I understand how everything works correctly.

@martijnvg @jpountz thoughts?

[jpountz]

Since this doesn't work today anyway, I'm +1 to disallow profiling scroll requests for now and documenting it. We can always revisit later if we want to.

[elasticmachine]

Pinging @elastic/es-search-aggs

[elasticsearchmachine]

Pinging @elastic/es-search (Team:Search)

[elasticsearchmachine]

Pinging @elastic/es-search-foundations (Team:Search Foundations)