search

excelsia / vee

VEE Reference Full Node
MIT No Attribution
29 stars 10 forks source link

Blacklisting node on connection issues #2

Open sunnyking opened 6 years ago

sunnyking commented 6 years ago

Blacklisting node for connection error makes network connectivity fragile. Also, dumping exception stack trace for connection errors seems overkill and bloats log.

2018-05-06 16:46:39 DEBUG [ntLoopGroup-3-4] c.w.n.FatalErrorHandler - [0dab83b2 54.152.46.6:6863] Exception caught java.io.IOException: Connection timed out at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:192) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:288) at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1100) at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:372) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:123) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) at java.lang.Thread.run(Thread.java:748) 2018-05-06 16:46:39 INFO [ntLoopGroup-3-4] c.w.network.NetworkServer - [0dab83b2 54.152.46.6:6863] Connection closed, 0 outgoing channel(s) remaining 2018-05-06 16:47:12 DEBUG [ntLoopGroup-3-1] c.w.network.NetworkServer - Connecting to /54.152.46.6:6863 2018-05-06 16:47:12 DEBUG [ntLoopGroup-3-5] c.w.network.NetworkServer - [478c0c14] Connection failed, blacklisting /54.152.46.6:6863 io.netty.channel.AbstractChannel$AnnotatedSocketException: Network is unreachable: /54.152.46.6:6863 at sun.nio.ch.Net.connect0(Native Method) at sun.nio.ch.Net.connect(Net.java:454) at sun.nio.ch.Net.connect(Net.java:446) at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648) at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:83) at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:80) at java.security.AccessController.doPrivileged(Native Method) at io.netty.util.internal.SocketUtils.connect(SocketUtils.java:80) at io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:337) at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:254) at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1266) at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) at io.netty.channel.ChannelOutboundHandlerAdapter.connect(ChannelOutboundHandlerAdapter.java:47) at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) at io.netty.channel.ChannelOutboundHandlerAdapter.connect(ChannelOutboundHandlerAdapter.java:47) at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) at io.netty.channel.ChannelDuplexHandler.connect(ChannelDuplexHandler.java:50) at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) at io.netty.channel.ChannelDuplexHandler.connect(ChannelDuplexHandler.java:50) at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) at io.netty.channel.ChannelDuplexHandler.connect(ChannelDuplexHandler.java:50) at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) at io.netty.channel.ChannelDuplexHandler.connect(ChannelDuplexHandler.java:50) at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) at io.netty.channel.ChannelDuplexHandler.connect(ChannelDuplexHandler.java:50) at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) at io.netty.channel.ChannelDuplexHandler.connect(ChannelDuplexHandler.java:50) at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:512) at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:985) at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:255) at io.netty.bootstrap.Bootstrap$3.run(Bootstrap.java:252) at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:462) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) at java.lang.Thread.run(Thread.java:748) Caused by: java.net.SocketException: Network is unreachable ... 47 common frames omitted

sunnyking commented 6 years ago

On another occurrence, when network connectivity is unstable, node appears unable to reestablish a connection after losing the connection to the seed node. Here it complains about quorum not available:

2018-05-10 22:19:29 DEBUG [miner-pool-75] c.w.mining.Miner - No block generated because Quorum not available (0/1, not forging block with 3N1YJ6RaYDkmh1fiy8ww7qCXDnySqyxceDS, retrying 2018-05-10 22:19:29 DEBUG [miner-pool-75] c.w.mining.Miner$ - CalculatedTS 1525982506038: CalculatedGenerationTS: 1525982507000 2018-05-10 22:19:29 DEBUG [miner-pool-75] c.w.mining.Miner - Next attempt for acc=3N1YJ6RaYDkmh1fiy8ww7qCXDnySqyxceDS in 1001 milliseconds