UpdateByQueryResponse throwing timeout

[Praveen82]

Hi All,

I am Using "elasticsearch-action-updatebyquery"

Reference : https://github.com/yakaz/elasticsearch-action-updatebyquery4

API : The following api will do update bulk "Segment ids" to mached documents.

Example : segmentId= 50 needs to update on more then 20+ million documents.

Map scriptParams = new HashMap(); scriptParams.put("segmentexist", segId); scriptParams.put("pgsegmentobject", pgSegmentIds);

UpdateByQueryClient updateByQueryClient = new UpdateByQueryClientWrapper(client);

UpdateByQueryResponse response = updateByQueryClient.prepareUpdateByQuery().setIndices(props.getProperty("index")).setTypes(props.getProperty("type"))
        .setTimeout(TimeValue.timeValueHours(24))
        .setIncludeBulkResponses(BulkResponseOption.ALL)

        .setScript("if (ctx._source.containsKey(\"pgSegmentIds\") ) { if (ctx._source.pgSegmentIds.contains(segmentexist) ) { ctx.op = \"none\" } else { ctx._source.pgSegmentIds += pgsegmentobject} } else { ctx._source.pgSegmentIds = pgsegmentobject }")
        .setScriptParams(scriptParams)

        .setQuery(query)

        .execute()
        .actionGet();

Its failing while update. I see the following exception.

2015-09-12 05:58:10 INFO transport:123 - [Moon Knight] failed to get local cluster state for [#transport#-1][ip-10-186-199-195][inet[localhost/10.31.48.47:9300]], disconnecting... org.elasticsearch.transport.ReceiveTimeoutTransportException: [][inet[localhost/10.31.48.47:9300]][cluster/state] request_id [416] timed out after [5000ms] at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:356) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2015-09-12 05:58:10 INFO transport:123 - [Moon Knight] failed to get local cluster state for [PGMonetize-ES04][bqljhciDQ4-Tr2dRAcbWtw][ip-10-31-48-47][inet[/10.31.48.47:9300]]{master=true}, disconnecting... org.elasticsearch.transport.ReceiveTimeoutTransportException: [PGMonetize-ES04][inet[/10.31.48.47:9300]][cluster/state] request_id [423] timed out after [5001ms] at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:356) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

I have done following setup

1) We are having 5 nodes, with 5 shards. 2) script.disable_dynamic: false action.updatebyquery.bulk_size: 2500

still I get the above exception. Please help.

How to solve this issue and how to improve performance like ( Updating 20+ million record in <10 mins)

[Praveen82]

any update on this?

[Praveen82]

is there any way to update 20+ million documents < 10 minutes?