Graph request locking up in 2.1.0

[mjameson-se]

For reasons unknown, some requests I make for graphs are blocking indefinitely waiting for a response -- generally from the web UI, and usually for large sets of data. If I forcefully refresh the browser, I can attempt to make other requests. Eventually, if enough requests are permanently blocked, the service completely ceases to service requests. Once I hit this state, a restart of the service is required. While blocked, memory and CPU consumption are stable and at normal levels. Below is a stack trace generated by jstack of one of the blocked threads (which has been waiting for more than 10 minutes at this point). I should also note that there were no threads at all in a running state in the thread dump; all blocked either at this point or in executors waiting for tasks, or in epoll.

Thread 2065: (state = BLOCKED)

• java.lang.Object.wait(long) @bci=0 (Interpreted frame)
• java.lang.Object.wait() @bci=2, line=503 (Interpreted frame)
• com.stumbleupon.async.Deferred.doJoin(boolean, long) @bci=77, line=1138 (Interpreted frame)
• com.stumbleupon.async.Deferred.joinUninterruptibly() @bci=3, line=1064 (Interpreted frame)
• net.opentsdb.core.TsdbQuery.run() @bci=4, line=324 (Interpreted frame)
• net.opentsdb.tsd.GraphHandler.doGraph(net.opentsdb.core.TSDB, net.opentsdb.tsd.HttpQuery) @bci=447, line=197 (Interpreted frame)
• net.opentsdb.tsd.GraphHandler.execute(net.opentsdb.core.TSDB, net.opentsdb.tsd.HttpQuery) @bci=84, line=121 (Interpreted frame)
• net.opentsdb.tsd.RpcHandler.handleHttpQuery(net.opentsdb.core.TSDB, org.jboss.netty.channel.Channel, org.jboss.netty.handler.codec.http.HttpRequest) @bci=354, line=273 (Interpreted frame)
• net.opentsdb.tsd.RpcHandler.messageReceived(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.MessageEvent) @bci=56, line=180 (Interpreted frame)
• org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=13, line=70 (Interpreted frame)
• org.jboss.netty.handler.timeout.IdleStateAwareChannelUpstreamHandler.handleUpstream(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=22, line=36 (Interpreted frame)
• org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=9, line=564 (Interpreted frame)
• org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(org.jboss.netty.channel.ChannelEvent) @bci=22, line=791 (Interpreted frame)
• org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.MessageEvent) @bci=19, line=294 (Interpreted frame)
• org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=13, line=70 (Interpreted frame)
• org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=9, line=564 (Interpreted frame)
• org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(org.jboss.netty.channel.ChannelEvent) @bci=22, line=791 (Interpreted frame)
• org.jboss.netty.handler.codec.http.HttpContentEncoder.messageReceived(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.MessageEvent) @bci=85, line=82 (Interpreted frame)
• org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=13, line=88 (Interpreted frame)
• org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=9, line=564 (Interpreted frame)
• org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(org.jboss.netty.channel.ChannelEvent) @bci=22, line=791 (Interpreted frame)
• org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.MessageEvent) @bci=285, line=108 (Interpreted frame)
• org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=13, line=70 (Interpreted frame)
• org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=9, line=564 (Interpreted frame)
• org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(org.jboss.netty.channel.ChannelEvent) @bci=22, line=791 (Interpreted frame)
• org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.MessageEvent) @bci=99, line=145 (Interpreted frame)
• org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=13, line=70 (Interpreted frame)
• org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=9, line=564 (Interpreted frame)
• org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(org.jboss.netty.channel.ChannelEvent) @bci=22, line=791 (Interpreted frame)
• org.jboss.netty.channel.Channels.fireMessageReceived(org.jboss.netty.channel.ChannelHandlerContext, java.lang.Object, java.net.SocketAddress) @bci=16, line=296 (Interpreted frame)
• org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(org.jboss.netty.channel.ChannelHandlerContext, java.net.SocketAddress, java.lang.Object) @bci=114, line=459 (Interpreted frame)
• org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.Channel, org.jboss.netty.buffer.ChannelBuffer, org.jboss.netty.buffer.ChannelBuffer, java.net.SocketAddress) @bci=176, line=536 (Interpreted frame)
• org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.MessageEvent) @bci=95, line=435 (Interpreted frame)
• org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=13, line=70 (Interpreted frame)
• org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=9, line=564 (Interpreted frame)
• org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(org.jboss.netty.channel.ChannelEvent) @bci=22, line=791 (Interpreted frame)
• org.jboss.netty.channel.SimpleChannelHandler.messageReceived(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.MessageEvent) @bci=2, line=142 (Interpreted frame)
• org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=13, line=88 (Interpreted frame)
• net.opentsdb.tsd.ConnectionManager.handleUpstream(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=22, line=87 (Interpreted frame)
• org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=9, line=564 (Interpreted frame)
• org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(org.jboss.netty.channel.ChannelEvent) @bci=55, line=559 (Interpreted frame)
• org.jboss.netty.channel.Channels.fireMessageReceived(org.jboss.netty.channel.Channel, java.lang.Object, java.net.SocketAddress) @bci=16, line=268 (Interpreted frame)
• org.jboss.netty.channel.Channels.fireMessageReceived(org.jboss.netty.channel.Channel, java.lang.Object) @bci=3, line=255 (Interpreted frame)
• org.jboss.netty.channel.socket.nio.NioWorker.read(java.nio.channels.SelectionKey) @bci=179, line=88 (Interpreted frame)
• org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(java.nio.channels.Selector) @bci=70, line=108 (Interpreted frame)
• org.jboss.netty.channel.socket.nio.AbstractNioSelector.run() @bci=372, line=318 (Interpreted frame)
• org.jboss.netty.channel.socket.nio.AbstractNioWorker.run() @bci=1, line=89 (Interpreted frame)
• org.jboss.netty.channel.socket.nio.NioWorker.run() @bci=1, line=178 (Interpreted frame)
• org.jboss.netty.util.ThreadRenamingRunnable.run() @bci=55, line=108 (Interpreted frame)
• org.jboss.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame)
• java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1145 (Interpreted frame)
• java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615 (Interpreted frame)
• java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

[manolama]

Which version of TSD were you using? Also were there any problems with HBase when this happened, e.g. a region server fell over or anything like that?

[mjameson-se]

I don't know the exact version, the title I used for the issue suggests some 2.1.0 build. As for the state of HBase, I wouldn't know where to start diagnosing issues with it. The only thing I can add at this point is that I was running in pseudo-distributed (single node) mode. I have since moved on to using a different tsd product for my purposes.

[manolama]

Thanks for the heads up, I'll check it out.