search

elastic / elasticsearch-java

Official Elasticsearch Java Client
Apache License 2.0
6 stars 242 forks source link

BulkIngester should allow customizations of intermediate requests #703

Open fabriziofortino opened 12 months ago

fabriziofortino commented 12 months ago

Description

Using the HLRC BulkProcessor, it became feasible to intercept the creation of requests by utilizing a BiConsumer that was invoked every time a bulk request was constructed (refer to https://github.com/elastic/elasticsearch/blob/main/server/src/main/java/org/elasticsearch/action/bulk/BulkProcessor.java#L213).

This capability proved particularly advantageous in various situations. For instance, if we consider a scenario where there are 1000 operations divided into 10 bulk requests, I could intercept the final request and modify the request refresh policy (for example, request.setRefreshPolicy(WriteRequest.RefreshPolicy.WAIT_UNTIL);).

Having a similar feature in the BulkIngester would be highly beneficial. One straightforward approach could involve adjusting the visibility of BulkIngester.newRequest() (refer to https://github.com/elastic/elasticsearch-java/blob/main/java-client/src/main/java/co/elastic/clients/elasticsearch/_helpers/bulk/BulkIngester.java#L243)

amitaiporat commented 3 weeks ago

@swallez do you know if this is something that will be addressed in a near term release?