msearch compatibility

[oli-tu]

The former Java High Level client had the nice feature, that single _search requests and the items of _msearch request were compatible, so our application could use the same builder for both, single and bulk searches. Although not in the specification, the msearch request code also supported source_include/excludes and the collapse parameter for msearches.

The new Elastic client, while forcefully implementing the spec, splits single and bulk searches into two worlds, including the deficiencies of the msearch endpoint described here.

While attempting to migrate to the new client two major pain points came up:

• The source_include/excludes would have been extremely helpful in combination with the typed response, as we can limit the returned fields and do not need to ignore them in the Json deserialization.
• The collapse function is essential for some of our use cases, and I have no idea how I should implement msearches without.

I created example queries for the old and new client, comparing single and msearchs. See my TODOs for the missing compatibilites. Is there any implementation planned? Is there any workaround for it?

Old Highlevel client

import org.elasticsearch.action.search.MultiSearchRequest;
import org.elasticsearch.action.search.MultiSearchResponse;
import org.elasticsearch.action.search.SearchRequest;
import org.elasticsearch.action.search.SearchResponse;
import org.elasticsearch.client.RequestOptions;
import org.elasticsearch.client.RestHighLevelClient;
import org.elasticsearch.common.unit.TimeValue;
import org.elasticsearch.index.query.QueryBuilders;
import org.elasticsearch.search.SearchHit;
import org.elasticsearch.search.SearchHits;
import org.elasticsearch.search.builder.SearchSourceBuilder;
import org.elasticsearch.search.collapse.CollapseBuilder;

@SpringBootTest
public class MSearchCollapseTest {

    private static final String FIELD_ID = "userid";    
    private static final String FIELD_USER = "username";

    @Value("${elasticSearch.indices.useractions}")
    private String index;

    @Autowired
    private RestHighLevelClient elasticHighLevelClient;

    @Test
    public void testSingleSearchCollapse() throws IOException {
        final SearchRequest searchRequest = createRequest();

        final SearchResponse searchResponse = elasticHighLevelClient.search(searchRequest, RequestOptions.DEFAULT);

        // ... do some verifications
    }

    @Test
    public void testMSearchCollapse() throws IOException {
        final MultiSearchRequest msearchRequest = new MultiSearchRequest();
        msearchRequest.add(createRequest()); // request compatibility :-)

        final MultiSearchResponse msearchResponse = elasticHighLevelClient.msearch(msearchRequest, RequestOptions.DEFAULT);

        // ... do some verifications
    }

    private SearchRequest createRequest() {
        final SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
        sourceBuilder.query(QueryBuilders.matchQuery(FIELD_USER, "Smith"));
        sourceBuilder.size(10);
        sourceBuilder.collapse(new CollapseBuilder(FIELD_ID));
        sourceBuilder.fetchSource(new String[]{ FIELD_ID, FIELD_USER }, null);
        sourceBuilder.timeout(new TimeValue(10, TimeUnit.SECONDS));

        final SearchRequest searchRequest = new SearchRequest(index);
        searchRequest.source(sourceBuilder);
        return searchRequest;
    }

}

New elasticsearch-java

import com.fasterxml.jackson.annotation.JsonAlias;
import com.fasterxml.jackson.annotation.JsonIgnoreProperties;

import co.elastic.clients.elasticsearch.ElasticsearchClient;
import co.elastic.clients.elasticsearch._types.query_dsl.Query;
import co.elastic.clients.elasticsearch._types.query_dsl.QueryBuilders;
import co.elastic.clients.elasticsearch.core.MsearchRequest;
import co.elastic.clients.elasticsearch.core.MsearchResponse;
import co.elastic.clients.elasticsearch.core.SearchRequest;
import co.elastic.clients.elasticsearch.core.SearchResponse;
import co.elastic.clients.elasticsearch.core.msearch.MultisearchBody;
import co.elastic.clients.elasticsearch.core.msearch.MultisearchHeader;

@SpringBootTest
public class MSearchCollapseTest {

    private static final String FIELD_ID = "userid";    
    private static final String FIELD_USER = "username";

    @Value("${elasticSearch.indices.useractions}")
    private String index;

    @Autowired
    private ElasticsearchClient elasticsearchClient;

    @Test
    public void testSingleSearchCollapse() throws IOException {

        final Query query = QueryBuilders.match()
                .field(FIELD_USER)
                .query(value -> value.stringValue("Smith"))
                .build()._toQuery();

        final SearchRequest searchRequest = new SearchRequest.Builder()
                .index(index)
                .size(10)
                .query(query)
                .collapse(collapse -> collapse.field(FIELD_ID))
                .source(source -> source.filter(filter -> filter.includes(FIELD_ID, FIELD_USER)))
                .timeout("10s")
                .build();

        final SearchResponse<ResultObject> searchResponse = elasticsearchClient.search(searchRequest, ResultObject.class);

        // ... do some verifications
    }

    @Test
    public void testMSearchCollapse() throws IOException {

        final Query query = QueryBuilders.match()
                .field(FIELD_USER)
                .query(value -> value.stringValue("Smith"))
                .build()._toQuery();

        final MultisearchHeader header = new MultisearchHeader.Builder()
                .index(index)
                .build();

        final MultisearchBody body = new MultisearchBody.Builder()
                .query(query)
                .size(10)
                // TODO timeout ? 
                // TODO field collapse ?
                // TODO source ?
                .build();

        final MsearchRequest msearchRequest = new MsearchRequest.Builder().searches(item -> item.header(header).body(body)).build();

        final MsearchResponse<ResultObject> msearchResponse = elasticsearchClient.msearch(msearchRequest, ResultObject.class);

         // ... do some verifications
    }

    @Getter
    @Setter
    @AllArgsConstructor
    @NoArgsConstructor
    @JsonIgnoreProperties(ignoreUnknown = true) // TODO This required as we cannot limit the returned fields in msearch
    private static class ResultObject {

        private String userid;
        private String username;

    }

}

[adrian-arapiles]

Any info about that?

[swallez]

The missing fields were added in https://github.com/elastic/elasticsearch-java/pull/292

Regarding the unification of search and multisearch builders, this is a very valid point. This however requires some changes that will cause a (limited) breaking change because of the class structure change. I've opened https://github.com/elastic/elasticsearch-specification/issues/1726 to address this.

[Erkanerkisi]

@adrian-arapiles how did you proceed with this structure? Did you seperately create search and multisearch requests?

[adrian-arapiles]

@adrian-arapiles how did you proceed with this structure? Did you seperately create search and multisearch requests?

Hi @Erkanerkisi, in my case I was migrating from the old client to this new client, so I still leave old client in methods that uses multisearch request until this ticket has been implemented

I feel a little sad with that because the old client is already deprecated but the new client hasn't yet same features than old client. It's a little confusing for me.

[Erkanerkisi]

@adrian-arapiles yea same situation applies us, we're using search requests and multisearch requests in a way that are replaceable each other. putting search requests in a multisearch structure is crutial from our side. But this way, we have to seperate them.