search

novitski / bitcoinj

Automatically exported from code.google.com/p/bitcoinj
Apache License 2.0
0 stars 0 forks source link

Unexpected re-entrancy via Channels.write() leads to Netty/user lock inversion #381

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago 
Getting this issue regularly (every few hours).

All the connected peers die and get the below in logs (on 616b4a5).

Looks like 2 threads are connected to same socket (66.166.64.125)? And 
deadlocking on the PeerGroup and the socket or some other lock inside netty.

Log:
2013-04-03/21:38:52.871/BST [New I/O worker #12] INFO  Peer$PeerHandler[225] 
[66.166.64.125]:8333 - An existing connection was forcibly closed by the remote 
host
2013-04-03/21:38:52.872/BST [New I/O worker #9] INFO  PeerGroup[983] 
[129.10.125.130]:8333: Peer died
2013-04-03/21:38:52.873/BST [New I/O worker #9] INFO  PeerGroup[985] Download 
peer died. Picking a new one.
2013-04-03/21:38:52.873/BST [New I/O worker #9] INFO  PeerGroup[912] Unsetting 
download peer: [129.10.125.130]:8333
2013-04-03/21:38:52.872/BST [New I/O worker #12] INFO  Peer$PeerHandler[225] 
[66.166.64.125]:8333 - Socket is disconnected
2013-04-03/21:38:52.872/BST [New I/O worker #1] INFO  Peer$PeerHandler[225] 
[70.30.239.152]:8333 - An existing connection was forcibly closed by the remote 
host
2013-04-03/21:38:52.873/BST [New I/O worker #9] INFO  PeerGroup[917] Setting 
download peer: [173.66.0.21]:8333
2013-04-03/21:38:52.874/BST [New I/O worker #1] INFO  Peer$PeerHandler[225] 
[70.30.239.152]:8333 - Socket is disconnected
2013-04-03/21:38:52.879/BST [New I/O worker #9] INFO  Peer$PeerHandler[225] 
[173.66.0.21]:8333 - An existing connection was forcibly closed by the remote 
host
2013-04-03/21:38:52.880/BST [New I/O worker #9] INFO  Peer$PeerHandler[225] 
[173.66.0.21]:8333 - Socket is disconnected
2013-04-03/21:38:52.882/BST [New I/O worker #9] INFO  Peer$PeerHandler[225] 
[173.66.0.21]:8333 - null
2013-04-03/21:38:52.882/BST [New I/O worker #9] INFO  PeerGroup[983] 
[173.66.0.21]:8333: Peer died
2013-04-03/21:38:52.883/BST [New I/O worker #9] INFO  PeerGroup[985] Download 
peer died. Picking a new one.
2013-04-03/21:38:52.883/BST [New I/O worker #9] INFO  PeerGroup[912] Unsetting 
download peer: [173.66.0.21]:8333
2013-04-03/21:38:52.883/BST [New I/O worker #9] INFO  PeerGroup[917] Setting 
download peer: [66.166.64.125]:8333

ThreadDump:

Found one Java-level deadlock:
=============================
"New I/O worker #9":
  waiting to lock monitor 0x00000000091b1be0 (object 0x0000000602dbdaf0, a java.lang.Object),
  which is held by "New I/O worker #12"
"New I/O worker #12":
  waiting for ownable synchronizer 0x0000000600e8fad0, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
  which is held by "New I/O worker #9"

Java stack information for the threads listed above:
===================================================
"New I/O worker #9":
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:374)
    - waiting to lock <0x0000000602dbdaf0> (a java.lang.Object)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:127)
    at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:83)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:775)
    at org.jboss.netty.channel.Channels.write(Channels.java:725)
    at com.google.bitcoin.core.TCPNetworkConnection$NetworkHandler.handleDownstream(TCPNetworkConnection.java:229)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:587)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:780)
    at org.jboss.netty.channel.SimpleChannelHandler.writeRequested(SimpleChannelHandler.java:292)
    at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:254)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:587)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:578)
    at org.jboss.netty.channel.Channels.write(Channels.java:704)
    at org.jboss.netty.channel.Channels.write(Channels.java:671)
    at com.google.bitcoin.core.Peer.sendMessage(Peer.java:1007)
    at com.google.bitcoin.core.Peer.blockChainDownload(Peer.java:1092)
    at com.google.bitcoin.core.Peer.startBlockChainDownload(Peer.java:1117)
    at com.google.bitcoin.core.PeerGroup.startBlockChainDownloadFromPeer(PeerGroup.java:1025)
    at com.google.bitcoin.core.PeerGroup.handlePeerDeath(PeerGroup.java:992)
    at com.google.bitcoin.core.PeerGroup$PeerStartupListener.onPeerDisconnected(PeerGroup.java:145)
    at com.google.bitcoin.core.Peer.notifyDisconnect(Peer.java:199)
    at com.google.bitcoin.core.Peer.access$000(Peer.java:48)
    at com.google.bitcoin.core.Peer$PeerHandler.channelClosed(Peer.java:207)
    at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:106)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787)
    at org.jboss.netty.handler.codec.replay.ReplayingDecoder.cleanup(ReplayingDecoder.java:570)
    at org.jboss.netty.handler.codec.frame.FrameDecoder.channelClosed(FrameDecoder.java:371)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:88)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
    at org.jboss.netty.channel.Channels.fireChannelClosed(Channels.java:468)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:351)
    at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:57)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:775)
    at com.google.bitcoin.core.TCPNetworkConnection$NetworkHandler.handleDownstream(TCPNetworkConnection.java:220)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:587)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:780)
    at org.jboss.netty.channel.SimpleChannelHandler.closeRequested(SimpleChannelHandler.java:334)
    at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:260)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:587)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:578)
    at org.jboss.netty.channel.Channels.close(Channels.java:812)
    at org.jboss.netty.channel.AbstractChannel.close(AbstractChannel.java:197)
    at com.google.bitcoin.core.Peer$PeerHandler.exceptionCaught(Peer.java:230)
    at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:130)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787)
    at org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:377)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
    at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:248)
    - locked <0x0000000604f26fb0> (a java.lang.Object)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:145)
    at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:83)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:775)
    at org.jboss.netty.channel.Channels.write(Channels.java:725)
    at com.google.bitcoin.core.TCPNetworkConnection$NetworkHandler.handleDownstream(TCPNetworkConnection.java:229)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:587)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:780)
    at org.jboss.netty.channel.SimpleChannelHandler.writeRequested(SimpleChannelHandler.java:292)
    at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:254)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:587)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:578)
    at org.jboss.netty.channel.Channels.write(Channels.java:704)
    at org.jboss.netty.channel.Channels.write(Channels.java:671)
    at com.google.bitcoin.core.Peer.sendMessage(Peer.java:1007)
    at com.google.bitcoin.core.Peer.blockChainDownload(Peer.java:1092)
    at com.google.bitcoin.core.Peer.startBlockChainDownload(Peer.java:1117)
    at com.google.bitcoin.core.PeerGroup.startBlockChainDownloadFromPeer(PeerGroup.java:1025)
    at com.google.bitcoin.core.PeerGroup.handlePeerDeath(PeerGroup.java:992)
    at com.google.bitcoin.core.PeerGroup$PeerStartupListener.onPeerDisconnected(PeerGroup.java:145)
    at com.google.bitcoin.core.Peer.notifyDisconnect(Peer.java:199)
    at com.google.bitcoin.core.Peer.access$000(Peer.java:48)
    at com.google.bitcoin.core.Peer$PeerHandler.channelClosed(Peer.java:207)
    at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:106)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787)
    at org.jboss.netty.handler.codec.replay.ReplayingDecoder.cleanup(ReplayingDecoder.java:570)
    at org.jboss.netty.handler.codec.frame.FrameDecoder.channelClosed(FrameDecoder.java:371)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:88)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
    at org.jboss.netty.channel.Channels.fireChannelClosed(Channels.java:468)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:351)
    at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:57)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:775)
    at com.google.bitcoin.core.TCPNetworkConnection$NetworkHandler.handleDownstream(TCPNetworkConnection.java:220)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:587)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:780)
    at org.jboss.netty.channel.SimpleChannelHandler.closeRequested(SimpleChannelHandler.java:334)
    at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:260)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:587)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:578)
    at org.jboss.netty.channel.Channels.close(Channels.java:812)
    at org.jboss.netty.channel.AbstractChannel.close(AbstractChannel.java:197)
    at com.google.bitcoin.core.Peer$PeerHandler.exceptionCaught(Peer.java:230)
    at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:130)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787)
    at org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:377)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
    at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:248)
    - locked <0x0000000600599a80> (a java.lang.Object)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:150)
    at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335)
    at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:366)
    at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:290)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
    at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
    at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
"New I/O worker #12":
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x0000000600e8fad0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
    at java.util.concurrent.locks.LockSupport.park(Unknown Source)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(Unknown Source)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(Unknown Source)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(Unknown Source)
    at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(Unknown Source)
    at java.util.concurrent.locks.ReentrantLock.lock(Unknown Source)
    at com.google.common.util.concurrent.CycleDetectingLockFactory$CycleDetectingReentrantLock.lock(CycleDetectingLockFactory.java:819)
    at com.google.bitcoin.core.PeerGroup.handlePeerDeath(PeerGroup.java:979)
    at com.google.bitcoin.core.PeerGroup$PeerStartupListener.onPeerDisconnected(PeerGroup.java:145)
    at com.google.bitcoin.core.Peer.notifyDisconnect(Peer.java:199)
    at com.google.bitcoin.core.Peer.access$000(Peer.java:48)
    at com.google.bitcoin.core.Peer$PeerHandler.channelClosed(Peer.java:207)
    at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:106)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787)
    at org.jboss.netty.handler.codec.replay.ReplayingDecoder.cleanup(ReplayingDecoder.java:570)
    at org.jboss.netty.handler.codec.frame.FrameDecoder.channelClosed(FrameDecoder.java:371)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:88)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
    at org.jboss.netty.channel.Channels.fireChannelClosed(Channels.java:468)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:351)
    at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:57)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:775)
    at com.google.bitcoin.core.TCPNetworkConnection$NetworkHandler.handleDownstream(TCPNetworkConnection.java:220)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:587)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:780)
    at org.jboss.netty.channel.SimpleChannelHandler.closeRequested(SimpleChannelHandler.java:334)
    at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:260)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:587)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:578)
    at org.jboss.netty.channel.Channels.close(Channels.java:812)
    at org.jboss.netty.channel.AbstractChannel.close(AbstractChannel.java:197)
    at com.google.bitcoin.core.Peer$PeerHandler.exceptionCaught(Peer.java:230)
    at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:130)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787)
    at org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:377)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
    at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:248)
    - locked <0x0000000602dbdaf0> (a java.lang.Object)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:150)
    at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335)
    at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:366)
    at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:290)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
    at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
    at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)

Found 1 deadlock.

Original issue reported on code.google.com by murexcon...@googlemail.com on 3 Apr 2013 at 9:14

GoogleCodeExporter commented 9 years ago 
Eww, these Netty stack traces are just ridiculous. I hope Netty 4 can fix some 
of these problems.

In this case the issue is not only Netty internal locks, but also unexpected 
re-entrancy. Attempting to send a message as part of switching download peer 
triggers a recursive re-entry via Channels.write(). It will take a bit of time 
to study this and figure out how it happened: looks like Channels.write() can 
make almost anything happen!

Original comment by hearn@google.com on 5 Apr 2013 at 9:29

GoogleCodeExporter commented 9 years ago 
I think the problem is that something bad happened that caused the remote peers 
to hang up on us, so when we picked a new download peer and tried to talk to 
it, that led to it also being terminated and we ended up recursively trying to 
close peers/pick new download peers.

One question is why so many peers died simultaneously. But it's academic, this 
situation can happen and needs to be handled appropriately.

Another question is the what the right fix is. Possibly trying to do too much 
work on an IO thread directly isn't the right approach and picking download 
peers is best done by a separate thread. But then again, Channels.write() on 
the closed peer would result in similar sorts of call stacks no matter what.

Original comment by hearn@google.com on 5 Apr 2013 at 10:18

GoogleCodeExporter commented 9 years ago 
netty is gone

Original comment by hearn@google.com on 12 Dec 2013 at 3:05