search

basho / yokozuna

Riak + Solr
245 stars 76 forks source link

Riak KV Handoff Timeouts caused by Yokozuna [JIRA: RIAK-2732] #669

Open jcapricebasho opened 8 years ago

jcapricebasho commented 8 years ago

An open source user reported Riak KV Handoffs repeatedly timing out and restarting when the yz_solr:entropy_data/2 case in yz_entropy:get_entropy_data/2 failed to return in the default 60 seconds set via YZ_SOLR_REQUEST_TIMEOUT.

Each time handoffs would start at around 6MB/s, gradually slowing down to about 100KB/s before timing out at around the same place each time. Before handoff times out, the below message is logged:

2016-08-04 18:10:58.057 [error] <0.1278.0>@yz_entropy:get_entropy_data:74 failed to iterate over entropy data due to request exceeding timeout 60000 for filter params [{continuation,none},{limit,100},{partition,19}]

Turning Search off allows handoff to complete successfully.

mengzyou commented 7 years ago

How about this error, we also encounter logs of such errors: 

2017-07-24 07:41:14.783 [error] <0.12084.307>@yz_entropy:get_entropy_data:74 failed to iterate over entropy data due to request exceeding timeout 60000 for filter params [{continuation,none},{limit,100},{partition,65}]
2017-07-24 07:42:15.838 [error] <0.12084.307>@yz_entropy:get_entropy_data:74 failed to iterate over entropy data due to request exceeding timeout 60000 for filter params [{continuation,none},{limit,100},{partition,65}]
2017-07-24 08:48:37.996 [error] <0.4253.310>@yz_entropy:get_entropy_data:74 failed to iterate over entropy data due to request exceeding timeout 60000 for filter params [{continuation,none},{limit,100},{partition,49}]
2017-07-24 09:44:57.673 [error] <0.17029.311>@yz_entropy:get_entropy_data:74 failed to iterate over entropy data due to request exceeding timeout 60000 for filter params [{continuation,<<"MyAxOCBZMkZ5WDNKMWJnPT0gWTJGeVgzSjFiakl3TVRjdE1EY3RNRFU9IE1URXpOamRrTjJZdE1qQmhNaTAwWkdRMUxXSm1ObU10WldKaE16UTVZakF5TVRVMiBnMklGTUt0RA">>},{limit,100},{partition,18}]
2017-07-24 09:49:55.593 [error] <0.17029.311>@yz_entropy:get_entropy_data:74 failed to iterate over entropy data due to request exceeding timeout 60000 for filter params [{continuation,<<"MyAxOCBZMmhoY21kcGJtZGZjR2xzWlE9PSBZMmhoY21kcGJtZGZjR2xzWlRJd01UY3RNRGN0TWpRPSBaRFZoTkRNNVpUTXRaamhpTUMwME5UbGpMV0ZsWXpNdE5EWXhPVGd5TVRRMlptWm0gZzJJQU4remY">>},{limit,100},{partition,18}]