search

jprante / elasticsearch-knapsack

Knapsack plugin is an import/export tool for Elasticsearch
Apache License 2.0
472 stars 77 forks source link

no_node_available_exception - Version 2.3.1 (push indices from one cluster to another) #101

Open MobileAppVault opened 8 years ago

MobileAppVault commented 8 years ago

Hi,

I've tried to push indices from one cluster to another and got the following error.

I already checked my firewall settings and all firewalls deactivated and I checkt the transport ports and they are also available. The file export is working fine.

Could you please have a look into that issue?

Request http://10.00.00.00:9200/products/_push?&host=10.00.00.13&port=9300&cluster=TEST

Response

{
  "error": {
    "root_cause": [
      {
        "type": "no_node_available_exception",
        "reason": "no cluster nodes available, check settings {client.transport.ignore_cluster_name=true, client.transport.ping_timeout=30s, client.transport.sniff=true, cluster.name=TEST, flush_interval=5s, host=10.00.00.13, max_actions_per_request=1000, max_concurrent_requests=8, path.plugins=.dontexist, port=9300, timeout=30s}"
      }
    ],
    "type": "no_node_available_exception",
    "reason": "no cluster nodes available, check settings {client.transport.ignore_cluster_name=true, client.transport.ping_timeout=30s, client.transport.sniff=true, cluster.name=TEST, flush_interval=5s, host=10.00.00.13, max_actions_per_request=1000, max_concurrent_requests=8, path.plugins=.dontexist, port=9300, timeout=30s}"
  },
  "status": 503
}

Configuration

CLUSTER 1

CLUSTER 2

Elastic Version 2.3.1

Here is the log file:

[2016-04-26 17:42:10,594][INFO ][org.xbib.elasticsearch.helper.client.BaseTransportClient] creating transport client on Windows 7 Java HotSpot(TM) 64-Bit Server VM Oracle Corporation 1.8.0_91-b14 25.91-b14 with effective settings {client.transport.ignore_cluster_name=true, client.transport.ping_timeout=30s, client.transport.sniff=true, cluster.name=TEST, flush_interval=5s, host=10.00.00.13, max_actions_per_request=1000, max_concurrent_requests=8, path.plugins=.dontexist, port=9300, timeout=30s}
[2016-04-26 17:42:10,599][INFO ][plugins                  ] [Sphinx] modules [], plugins [helper], sites []
[2016-04-26 17:42:10,640][INFO ][org.xbib.elasticsearch.helper.client.BaseTransportClient] trying to connect to [10.00.00.13:9300]
[2016-04-26 17:42:10,656][ERROR][KnapsackPushAction       ] no cluster nodes available, check settings {client.transport.ignore_cluster_name=true, client.transport.ping_timeout=30s, client.transport.sniff=true, cluster.name=TEST, flush_interval=5s, host=10.00.00.13, max_actions_per_request=1000, max_concurrent_requests=8, path.plugins=.dontexist, port=9300, timeout=30s}
NoNodeAvailableException[no cluster nodes available, check settings {client.transport.ignore_cluster_name=true, client.transport.ping_timeout=30s, client.transport.sniff=true, cluster.name=TEST, flush_interval=5s, host=10.00.00.13, max_actions_per_request=1000, max_concurrent_requests=8, path.plugins=.dontexist, port=9300, timeout=30s}]
    at org.xbib.elasticsearch.helper.client.BulkTransportClient.init(BulkTransportClient.java:163)
    at org.xbib.elasticsearch.helper.client.ClientBuilder.toBulkTransportClient(ClientBuilder.java:112)
    at org.xbib.elasticsearch.action.knapsack.push.TransportKnapsackPushAction.doExecute(TransportKnapsackPushAction.java:101)
    at org.xbib.elasticsearch.action.knapsack.push.TransportKnapsackPushAction.doExecute(TransportKnapsackPushAction.java:66)
    at org.elasticsearch.action.support.TransportAction.doExecute(TransportAction.java:149)
    at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:137)
    at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:85)
    at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:58)
    at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:359)
    at org.elasticsearch.client.FilterClient.doExecute(FilterClient.java:52)
    at org.elasticsearch.rest.BaseRestHandler$HeadersAndContextCopyClient.doExecute(BaseRestHandler.java:83)
    at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:359)
    at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1226)
    at org.xbib.elasticsearch.rest.action.knapsack.push.RestKnapsackPushAction.handleRequest(RestKnapsackPushAction.java:76)
    at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:54)
    at org.elasticsearch.rest.RestController.executeHandler(RestController.java:205)
    at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:166)
    at org.elasticsearch.http.HttpServer.internalDispatchRequest(HttpServer.java:128)
    at org.elasticsearch.http.HttpServer$Dispatcher.dispatchRequest(HttpServer.java:86)
    at org.elasticsearch.http.netty.NettyHttpServerTransport.dispatchRequest(NettyHttpServerTransport.java:449)
    at org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(HttpRequestHandler.java:61)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.elasticsearch.http.netty.pipelining.HttpPipeliningHandler.messageReceived(HttpPipeliningHandler.java:60)
    at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
    at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459)
    at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)
    at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:75)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
    at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
    at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
    at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
    at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
[2016-04-26 17:42:10,688][WARN ][rest.suppressed          ] /products/_push Params: {cluster=TEST, size=1000, port=9300, host=10.00.00.13, index=products, sort=_doc}
NoNodeAvailableException[no cluster nodes available, check settings {client.transport.ignore_cluster_name=true, client.transport.ping_timeout=30s, client.transport.sniff=true, cluster.name=TEST, flush_interval=5s, host=10.00.00.13, max_actions_per_request=1000, max_concurrent_requests=8, path.plugins=.dontexist, port=9300, timeout=30s}]
    at org.xbib.elasticsearch.helper.client.BulkTransportClient.init(BulkTransportClient.java:163)
    at org.xbib.elasticsearch.helper.client.ClientBuilder.toBulkTransportClient(ClientBuilder.java:112)
    at org.xbib.elasticsearch.action.knapsack.push.TransportKnapsackPushAction.doExecute(TransportKnapsackPushAction.java:101)
    at org.xbib.elasticsearch.action.knapsack.push.TransportKnapsackPushAction.doExecute(TransportKnapsackPushAction.java:66)
    at org.elasticsearch.action.support.TransportAction.doExecute(TransportAction.java:149)
    at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:137)
    at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:85)
    at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:58)
    at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:359)
    at org.elasticsearch.client.FilterClient.doExecute(FilterClient.java:52)
    at org.elasticsearch.rest.BaseRestHandler$HeadersAndContextCopyClient.doExecute(BaseRestHandler.java:83)
    at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:359)
    at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1226)
    at org.xbib.elasticsearch.rest.action.knapsack.push.RestKnapsackPushAction.handleRequest(RestKnapsackPushAction.java:76)
    at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:54)
    at org.elasticsearch.rest.RestController.executeHandler(RestController.java:205)
    at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:166)
    at org.elasticsearch.http.HttpServer.internalDispatchRequest(HttpServer.java:128)
    at org.elasticsearch.http.HttpServer$Dispatcher.dispatchRequest(HttpServer.java:86)
    at org.elasticsearch.http.netty.NettyHttpServerTransport.dispatchRequest(NettyHttpServerTransport.java:449)
    at org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(HttpRequestHandler.java:61)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.elasticsearch.http.netty.pipelining.HttpPipeliningHandler.messageReceived(HttpPipeliningHandler.java:60)
    at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
    at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459)
    at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)
    at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:75)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
    at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
    at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
    at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
    at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
avgl-sstraub commented 8 years ago

same issue still existent in 2.3.3

mahuxiaozi commented 8 years ago

same issue in 2.3.0, in debug model , i found the error below: [2016-08-04 15:45:01,476][DEBUG][client.transport ] [Dionysus] node_sampler_interval[5s] [2016-08-04 15:45:01,493][DEBUG][org.xbib.elasticsearch.common.GcMonitor] enabled [true], interval [1s], gcthreshold [{default=GcThreshold{name='default', warnThreshold=10000, infoThreshold=5000, debugThreshold=2000}, young=GcThreshold{name='young', warnThreshold=1000, infoThreshold=700, debugThreshold=400}, old=GcThreshold{name='old', warnThreshold=10000, infoThreshold=5000, debugThreshold=2000}}] [2016-08-04 15:45:01,503][INFO ][org.xbib.elasticsearch.helper.client.BaseTransportClient] trying to connect to [192.168.10.4:9300] [2016-08-04 15:45:01,504][DEBUG][org.xbib.elasticsearch.helper.client.TransportClient] [Dionysus] adding address [{#transport#-1}{192.168.10.4}{192.168.10.4:9300}] [2016-08-04 15:45:01,504][DEBUG][org.xbib.elasticsearch.helper.client.TransportClient] [Dionysus] failed to connect to node [{#transport#-1}{192.168.10.4}{192.168.10.4:9300}], removed from nodes list ConnectTransportException[[][192.168.10.4:9300] general node connection failure]; nested: IllegalStateException[await() in I/O thread causes a dead lock or sudden performance drop. Use addListener() instead or call await() from a different thread.]; at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:937) at org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:888) at org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:267) at org.xbib.elasticsearch.helper.client.TransportClient.connect(TransportClient.java:348) at org.xbib.elasticsearch.helper.client.TransportClient.addTransportAddresses(TransportClient.java:281) at org.xbib.elasticsearch.helper.client.BaseTransportClient.connect(BaseTransportClient.java:98) at org.xbib.elasticsearch.helper.client.BulkTransportClient.init(BulkTransportClient.java:162) at org.xbib.elasticsearch.helper.client.ClientBuilder.toBulkTransportClient(ClientBuilder.java:112) at org.xbib.elasticsearch.action.knapsack.push.TransportKnapsackPushAction.doExecute(TransportKnapsackPushAction.java:101) at org.xbib.elasticsearch.action.knapsack.push.TransportKnapsackPushAction.doExecute(TransportKnapsackPushAction.java:66) at org.elasticsearch.action.support.TransportAction.doExecute(TransportAction.java:149) at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:137) at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:85) at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:58) at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:359) at org.elasticsearch.client.FilterClient.doExecute(FilterClient.java:52) ....... at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.IllegalStateException: await() in I/O thread causes a dead lock or sudden performance drop. Use addListener() instead or call await_() from a different thread. at org.jboss.netty.channel.DefaultChannelFuture.checkDeadLock(DefaultChannelFuture.java:320) at org.jboss.netty.channel.DefaultChannelFuture.await0(DefaultChannelFuture.java:284) at org.jboss.netty.channel.DefaultChannelFuture.awaitUninterruptibly(DefaultChannelFuture.java:263) at org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:950) at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:916) ... 63 more [2016-08-04 15:45:01,505][ERROR][KnapsackPushAction ] no cluster nodes available, check settings {client.transport.ignore_cluster_name=true, client.transport.ping_timeout=30s, client.transport.sniff=true, cluster.name=liveyc, flush_interval=5s, host=192.168.10.4, max_actions_per_request=1000, max_concurrent_requests=4, path.plugins=.dontexist, port=9300, timeout=30s} NoNodeAvailableException[no cluster nodes available, check settings {client.transport.ignore_cluster_name=true, client.transport.ping_timeout=30s, client.transport.sniff=true, cluster.name=liveyc, flush_interval=5s, host=192.168.10.4, max_actions_per_request=1000, max_concurrent_requests=4, path.plugins=.dontexist, port=9300, timeout=30s}] at org.xbib.elasticsearch.helper.client.BulkTransportClient.init(BulkTransportClient.java:164) at org.xbib.elasticsearch.helper.client.ClientBuilder.toBulkTransportClient(ClientBuilder.java:112) at org.xbib.elasticsearch.action.knapsack.push.TransportKnapsackPushAction.doExecute(TransportKnapsackPushAction.java:101) at org.xbib.elasticsearch.action.knapsack.push.TransportKnapsackPushAction.doExecute(TransportKnapsackPushAction.java:66) at org.elasticsearch.action.support.TransportAction.doExecute(TransportAction.java:149) at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:137) at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:85) at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:58) at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:359)

jprante commented 8 years ago

Thanks. There are limitations in Elasticsearch, I can only think of fixing this by disabling Netty dead lock checking. Will be in next release #110