search

OpenTSDB / opentsdb

A scalable, distributed Time Series Database.
http://opentsdb.net
GNU Lesser General Public License v2.1
4.99k stars 1.25k forks source link

opentsdb query return error #1038

Open xorangelee opened 7 years ago

xorangelee commented 7 years ago

hi，when i use grafana get data from opentsdb it has that error sometimes，sometines its ok。 Can anyone help why this is happening and how to recover from this。

here is return in web page: {"error":{"code":424,"message":"Too many attempts: OpenScannerRequest(scanner=Scanner(table=\"tsdb\", start_key=[1, 0, 0, 1, 89, -127, -9, 0], stop_key=[1, 0, 0, 1, 89, -125, 86, -112], columns={\"t\"}, populate_blockcache=true, max_num_rows=128, max_num_kvs=4096, region=null, filter=KeyRegexpFilter(\"(?s)^.{8}(?:.{6})\Q\u0000\u0000\u0001\E(?:\Q\u0000\u001Aq\E)(?:.{6})\Q\u0000\u0000\u0002\E(?:\Q\u0000\u0019%\E)(?:.{6})\Q\u0000\u0000\u0003\E.{3}(?:.{6})$\", ISO-8859-1), scanner_id=0x0000000000000000))","trace":"net.opentsdb.tsd.BadRequestException: Too many attempts: OpenScannerRequest(scanner=Scanner(table=\"tsdb\", start_key=[1, 0, 0, 1, 89, -127, -9, 0], stop_key=[1, 0, 0, 1, 89, -125, 86, -112], columns={\"t\"}, populate_blockcache=true, max_num_rows=128, max_num_kvs=4096, region=null, filter=KeyRegexpFilter(\"(?s)^.{8}(?:.{6})\Q\u0000\u0000\u0001\E(?:\Q\u0000\u001Aq\E)(?:.{6})\Q\u0000\u0000\u0002\E(?:\Q\u0000\u0019%\E)(?:.{6})\Q\u0000\u0000\u0003\E.{3}(?:.{6})$\", ISO-8859-1), scanner_id=0x0000000000000000))\n\tat net.opentsdb.tsd.QueryRpc$1ErrorCB.call(QueryRpc.java:181) [build/:na]\n\tat net.opentsdb.tsd.QueryRpc$1ErrorCB.call(QueryRpc.java:158) [build/:na]\n\tat com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1313) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.doCall(Deferred.java:1284) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1313) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.doCall(Deferred.java:1284) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.access$300(Deferred.java:430) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred$Continue.call(Deferred.java:1366) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.callback(Deferred.java:1005) [async-1.4.0.jar:na]\n\tat org.hbase.async.HBaseRpc.callback(HBaseRpc.java:712) [asynchbase-1.7.2.jar:na]\n\tat org.hbase.async.RegionClient.decode(RegionClient.java:1533) [asynchbase-1.7.2.jar:na]\n\tat org.hbase.async.RegionClient.decode(RegionClient.java:88) [asynchbase-1.7.2.jar:na]\n\tat org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) [netty-3.9.4.Final.jar:na]\n\tat org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) [netty-3.9.4.Final.jar:na]\n\tat org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [netty-3.9.4.Final.jar:na]\n\tat org.hbase.async.RegionClient.handleUpstream(RegionClient.java:1223) [asynchbase-1.7.2.jar:na]\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.9.4.Final.jar:na]\n\tat org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.9.4.Final.jar:na]\n\tat org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) [netty-3.9.4.Final.jar:na]\n\tat org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [netty-3.9.4.Final.jar:na]\n\tat org.jboss.netty.handler.timeout.IdleStateAwareChannelHandler.handleUpstream(IdleStateAwareChannelHandler.java:36) [netty-3.9.4.Final.jar:na]\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.9.4.Final.jar:na]\n\tat org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.9.4.Final.jar:na]\n\tat org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:294) [netty-3.9.4.Final.jar:na]\n\tat org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [netty-3.9.4.Final.jar:na]\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.9.4.Final.jar:na]\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [netty-3.9.4.Final.jar:na]\n\tat org.hbase.async.HBaseClient$RegionClientPipeline.sendUpstream(HBaseClient.java:3121) [asynchbase-1.7.2.jar:na]\n\tat org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [netty-3.9.4.Final.jar:na]\n\tat org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [netty-3.9.4.Final.jar:na]\n\tat org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [netty-3.9.4.Final.jar:na]\n\tat org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [netty-3.9.4.Final.jar:na]\n\tat org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) [netty-3.9.4.Final.jar:na]\n\tat org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [netty-3.9.4.Final.jar:na]\n\tat org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [netty-3.9.4.Final.jar:na]\n\tat org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [netty-3.9.4.Final.jar:na]\n\tat org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [netty-3.9.4.Final.jar:na]\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_131]\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_131]\n\tat java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]\n"}}

here is log in opentsdb:

2017-08-03 17:28:30,998 ERROR [AsyncHBase I/O Worker #32] QueryRpc: Query exception: com.stumbleupon.async.DeferredGroupException: At least one of the Deferreds failed, first exception: at com.stumbleupon.async.DeferredGroup.done(DeferredGroup.java:169) ~[async-1.4.0.jar:na] at com.stumbleupon.async.DeferredGroup.recordCompletion(DeferredGroup.java:158) ~[async-1.4.0.jar:na] at com.stumbleupon.async.DeferredGroup.access$200(DeferredGroup.java:36) ~[async-1.4.0.jar:na] at com.stumbleupon.async.DeferredGroup$1NotifyOrdered.call(DeferredGroup.java:97) ~[async-1.4.0.jar:na] at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) [async-1.4.0.jar:na] at com.stumbleupon.async.Deferred.addCallbacks(Deferred.java:688) [async-1.4.0.jar:na] at com.stumbleupon.async.Deferred.addBoth(Deferred.java:772) [async-1.4.0.jar:na] at com.stumbleupon.async.DeferredGroup.(DeferredGroup.java:114) ~[async-1.4.0.jar:na] at com.stumbleupon.async.Deferred.groupInOrder(Deferred.java:913) [async-1.4.0.jar:na] at net.opentsdb.tsd.QueryRpc$1BuildCB.call(QueryRpc.java:260) ~[build/:na] at net.opentsdb.tsd.QueryRpc$1BuildCB.call(QueryRpc.java:252) ~[build/:na] at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) [async-1.4.0.jar:na] at com.stumbleupon.async.Deferred.addCallbacks(Deferred.java:688) [async-1.4.0.jar:na] at com.stumbleupon.async.Deferred.addCallback(Deferred.java:724) [async-1.4.0.jar:na] at net.opentsdb.tsd.QueryRpc$1GlobalCB.call(QueryRpc.java:268) ~[build/:na] at net.opentsdb.tsd.QueryRpc$1GlobalCB.call(QueryRpc.java:265) ~[build/:na] at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) [async-1.4.0.jar:na] at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) [async-1.4.0.jar:na] at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1313) [async-1.4.0.jar:na] at com.stumbleupon.async.Deferred.doCall(Deferred.java:1284) [async-1.4.0.jar:na] at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) [async-1.4.0.jar:na] at com.stumbleupon.async.Deferred.access$300(Deferred.java:430) [async-1.4.0.jar:na] at com.stumbleupon.async.Deferred$Continue.call(Deferred.java:1366) [async-1.4.0.jar:na] at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) [async-1.4.0.jar:na] at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) [async-1.4.0.jar:na] at com.stumbleupon.async.Deferred.callback(Deferred.java:1005) [async-1.4.0.jar:na] at org.hbase.async.HBaseRpc.callback(HBaseRpc.java:712) [asynchbase-1.7.2.jar:na] at org.hbase.async.RegionClient.decode(RegionClient.java:1533) [asynchbase-1.7.2.jar:na] at org.hbase.async.RegionClient.decode(RegionClient.java:88) [asynchbase-1.7.2.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) [netty-3.9.4.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) [netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [netty-3.9.4.Final.jar:na] at org.hbase.async.RegionClient.handleUpstream(RegionClient.java:1223) [asynchbase-1.7.2.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) [netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [netty-3.9.4.Final.jar:na] at org.jboss.netty.handler.timeout.IdleStateAwareChannelHandler.handleUpstream(IdleStateAwareChannelHandler.java:36) [netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.9.4.Final.jar:na] at org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:294) [netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [netty-3.9.4.Final.jar:na] at org.hbase.async.HBaseClient$RegionClientPipeline.sendUpstream(HBaseClient.java:3121) [asynchbase-1.7.2.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) [netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [netty-3.9.4.Final.jar:na] at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [netty-3.9.4.Final.jar:na] at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [netty-3.9.4.Final.jar:na] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_131] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131] Caused by: org.hbase.async.NonRecoverableException: Too many attempts: OpenScannerRequest(scanner=Scanner(table="tsdb", start_key=[-89, 0, 0, 1, 89, -128, -91, -128], stop_key=[-89, 0, 0, 1, 89, -126, 5, 16], columns={"t"}, populate_blockcache=true, max_num_rows=128, max_num_kvs=4096, region=null, filter=KeyRegexpFilter("(?s)^.{8}(?:.{6})\Q^@^@^A\E(?:\Q^@^Zq\E)(?:.{6})\Q^@^@^B\E(?:\Q^@^Y%\E)(?:.{6})\Q^@^@^C\E.{3}(?:.{6})$", ISO-8859-1), scanner_id=0x0000000000000000)) at org.hbase.async.HBaseClient.tooManyAttempts(HBaseClient.java:2056) ~[asynchbase-1.7.2.jar:na] at org.hbase.async.HBaseClient.sendRpcToRegion(HBaseClient.java:1920) ~[asynchbase-1.7.2.jar:na] at org.hbase.async.HBaseClient$1RetryRpc.call(HBaseClient.java:1944) ~[asynchbase-1.7.2.jar:na] at org.hbase.async.HBaseClient$1RetryRpc.call(HBaseClient.java:1927) ~[asynchbase-1.7.2.jar:na] at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) [async-1.4.0.jar:na] at com.stumbleupon.async.Deferred.addCallbacks(Deferred.java:688) [async-1.4.0.jar:na] at com.stumbleupon.async.Deferred.addBothDeferring(Deferred.java:786) [async-1.4.0.jar:na] at org.hbase.async.HBaseClient.sendRpcToRegion(HBaseClient.java:1968) ~[asynchbase-1.7.2.jar:na] at org.hbase.async.HBaseClient$1RetryRpc.call(HBaseClient.java:1944) ~[asynchbase-1.7.2.jar:na] at org.hbase.async.HBaseClient$1RetryRpc.call(HBaseClient.java:1927) ~[asynchbase-1.7.2.jar:na] at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) [async-1.4.0.jar:na] at com.stumbleupon.async.Deferred.addCallbacks(Deferred.java:688) [async-1.4.0.jar:na] at com.stumbleupon.async.Deferred.addBothDeferring(Deferred.java:786) [async-1.4.0.jar:na] at org.hbase.async.HBaseClient.sendRpcToRegion(HBaseClient.java:1968) ~[asynchbase-1.7.2.jar:na] at org.hbase.async.HBaseClient$1RetryRpc.call(HBaseClient.java:1944) ~[asynchbase-1.7.2.jar:na] at org.hbase.async.HBaseClient$1RetryRpc.call(HBaseClient.java:1927) ~[asynchbase-1.7.2.jar:na] at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) [async-1.4.0.jar:na] at com.stumbleupon.async.Deferred.addCallbacks(Deferred.java:688) [async-1.4.0.jar:na] at com.stumbleupon.async.Deferred.addBothDeferring(Deferred.java:786) [async-1.4.0.jar:na] at org.hbase.async.HBaseClient.sendRpcToRegion(HBaseClient.java:1968) ~[asynchbase-1.7.2.jar:na] at org.hbase.async.HBaseClient$1RetryRpc.call(HBaseClient.java:1944) ~[asynchbase-1.7.2.jar:na] at org.hbase.async.HBaseClient$1RetryRpc.call(HBaseClient.java:1927) ~[asynchbase-1.7.2.jar:na] at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) [async-1.4.0.jar:na] at com.stumbleupon.async.Deferred.addCallbacks(Deferred.java:688) [async-1.4.0.jar:na] at com.stumbleupon.async.Deferred.addBothDeferring(Deferred.java:786) [async-1.4.0.jar:na] at org.hbase.async.HBaseClient.sendRpcToRegion(HBaseClient.java:1968) ~[asynchbase-1.7.2.jar:na] at org.hbase.async.HBaseClient$1RetryRpc.call(HBaseClient.java:1944) ~[asynchbase-1.7.2.jar:na] at org.hbase.async.HBaseClient$1RetryRpc.call(HBaseClient.java:1927) ~[asynchbase-1.7.2.jar:na] at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) [async-1.4.0.jar:na] at com.stumbleupon.async.Deferred.addCallbacks(Deferred.java:688) [async-1.4.0.jar:na] at com.stumbleupon.async.Deferred.addBothDeferring(Deferred.java:786) [async-1.4.0.jar:na] at org.hbase.async.HBaseClient.sendRpcToRegion(HBaseClient.java:1968) ~[asynchbase-1.7.2.jar:na] at org.hbase.async.HBaseClient$1RetryRpc.call(HBaseClient.java:1944) ~[asynchbase-1.7.2.jar:na] at org.hbase.async.HBaseClient$1RetryRpc.call(HBaseClient.java:1927) ~[asynchbase-1.7.2.jar:na] at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) [async-1.4.0.jar:na] at com.stumbleupon.async.Deferred.addCallbacks(Deferred.java:688) [async-1.4.0.jar:na] at com.stumbleupon.async.Deferred.addBothDeferring(Deferred.java:786) [async-1.4.0.jar:na] at org.hbase.async.HBaseClient.sendRpcToRegion(HBaseClient.java:1968) ~[asynchbase-1.7.2.jar:na] at org.hbase.async.HBaseClient$1RetryRpc.call(HBaseClient.java:1944) ~[asynchbase-1.7.2.jar:na] at org.hbase.async.HBaseClient$1RetryRpc.call(HBaseClient.java:1927) ~[asynchbase-1.7.2.jar:na] at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) [async-1.4.0.jar:na] at com.stumbleupon.async.Deferred.addCallbacks(Deferred.java:688) [async-1.4.0.jar:na] at com.stumbleupon.async.Deferred.addBothDeferring(Deferred.java:786) [async-1.4.0.jar:na] at org.hbase.async.HBaseClient.sendRpcToRegion(HBaseClient.java:1968) ~[asynchbase-1.7.2.jar:na] at org.hbase.async.HBaseClient$1RetryRpc.call(HBaseClient.java:1944) ~[asynchbase-1.7.2.jar:na] at org.hbase.async.HBaseClient$1RetryRpc.call(HBaseClient.java:1927) ~[asynchbase-1.7.2.jar:na] at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) [async-1.4.0.jar:na] at com.stumbleupon.async.Deferred.addCallbacks(Deferred.java:688) [async-1.4.0.jar:na] at com.stumbleupon.async.Deferred.addBothDeferring(Deferred.java:786) [async-1.4.0.jar:na] at org.hbase.async.HBaseClient.sendRpcToRegion(HBaseClient.java:1968) ~[asynchbase-1.7.2.jar:na] at org.hbase.async.HBaseClient$1RetryRpc.call(HBaseClient.java:1944) ~[asynchbase-1.7.2.jar:na] at org.hbase.async.HBaseClient$1RetryRpc.call(HBaseClient.java:1927) ~[asynchbase-1.7.2.jar:na] at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) [async-1.4.0.jar:na] at com.stumbleupon.async.Deferred.addCallbacks(Deferred.java:688) [async-1.4.0.jar:na] at com.stumbleupon.async.Deferred.addBothDeferring(Deferred.java:786) [async-1.4.0.jar:na] at org.hbase.async.HBaseClient.sendRpcToRegion(HBaseClient.java:1968) ~[asynchbase-1.7.2.jar:na] at org.hbase.async.HBaseClient$1RetryRpc.call(HBaseClient.java:1944) ~[asynchbase-1.7.2.jar:na] at org.hbase.async.HBaseClient$1RetryRpc.call(HBaseClient.java:1927) ~[asynchbase-1.7.2.jar:na] at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) [async-1.4.0.jar:na] at com.stumbleupon.async.Deferred.addCallbacks(Deferred.java:688) [async-1.4.0.jar:na] at com.stumbleupon.async.Deferred.addBothDeferring(Deferred.java:786) [async-1.4.0.jar:na] at org.hbase.async.HBaseClient.sendRpcToRegion(HBaseClient.java:1968) ~[asynchbase-1.7.2.jar:na] at org.hbase.async.HBaseClient$1RetryRpc.call(HBaseClient.java:1944) ~[asynchbase-1.7.2.jar:na] at org.hbase.async.HBaseClient$1RetryRpc.call(HBaseClient.java:1927) ~[asynchbase-1.7.2.jar:na] at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) [async-1.4.0.jar:na] at com.stumbleupon.async.Deferred.addCallbacks(Deferred.java:688) [async-1.4.0.jar:na] at com.stumbleupon.async.Deferred.addBothDeferring(Deferred.java:786) [async-1.4.0.jar:na] at org.hbase.async.HBaseClient.sendRpcToRegion(HBaseClient.java:1968) ~[asynchbase-1.7.2.jar:na] at org.hbase.async.HBaseClient$1RetryRpc.call(HBaseClient.java:1944) ~[asynchbase-1.7.2.jar:na] at org.hbase.async.HBaseClient$1RetryRpc.call(HBaseClient.java:1927) ~[asynchbase-1.7.2.jar:na] at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) [async-1.4.0.jar:na] at com.stumbleupon.async.Deferred.addCallbacks(Deferred.java:688) [async-1.4.0.jar:na] at com.stumbleupon.async.Deferred.addBothDeferring(Deferred.java:786) [async-1.4.0.jar:na] at org.hbase.async.HBaseClient.sendRpcToRegion(HBaseClient.java:1968) ~[asynchbase-1.7.2.jar:na] at org.hbase.async.HBaseClient.openScanner(HBaseClient.java:1249) ~[asynchbase-1.7.2.jar:na] at org.hbase.async.Scanner.nextRows(Scanner.java:701) ~[asynchbase-1.7.2.jar:na] at net.opentsdb.core.SaltScanner$ScannerCB.scan(SaltScanner.java:358) ~[build/:na] at net.opentsdb.core.SaltScanner.scan(SaltScanner.java:205) ~[build/:na] at net.opentsdb.core.TsdbQuery.findSpans(TsdbQuery.java:555) ~[build/:na] at net.opentsdb.core.TsdbQuery.runAsync(TsdbQuery.java:513) ~[build/:na] at net.opentsdb.tsd.QueryRpc$1BuildCB.call(QueryRpc.java:258) ~[build/:na] ... 47 common frames omitted

manolama commented 7 years ago

Hi, yeah the error isn't completely clear but the pertinent part is: Too many attempts: OpenScannerRequest(scanner=Scanner(table="tsdb", start_key=[1, 0, 0, 1, 89, -127, -9, 0], stop_key=[1, 0, 0, 1, 89, -125, 86, -112].

This is saying that when the TSD tried to read data from the region where the time series for the Java encoded byte array [1, 0, 0, 1, 89, -127, -9, 0] couldn't be found, likely because it was splitting on the HBase region server.

To fix it, you may need to tune HBase some more so that it splits less frequently or completes splits sooner. It could have to do with your write-rate.

Jiangshan00001 commented 7 years ago

i got something familiar. is there a way how to "tune HBase some more" ? http://192.168.1.172:4242/api/query?start=1503880730&end=1503884330&m=avg:down_sampler:30s-avg:sensor.value%7Bsn=B600-0046,type=pm25%7D

{"error":{"code":424,"message":"Too many attempts: OpenScannerRequest(scanner=Scanner(table=\"tsdb\", start_key=[0, 0, 1, 89, -93, 93, 0], stop_key=\"\x00\x00\x01Y\xA3y \", columns={\"t\"}, populate_blockcache=true, max_num_rows=128, max_num_kvs=4096, region=null, filter=KeyRegexpFilter(\"(?s)^.{7}(?:.{6})\Q\u0000\u0000\u0001\E(?:\Q\u0000›I\E)(?:.{6})\Q\u0000\u0000\u0004\E(?:\Q\u0000\u0000\u0004\E)(?:.{6})$\", ISO-8859-1), scanner_id=0x0000000000000000))","trace":"net.opentsdb.tsd.BadRequestException: Too many attempts: OpenScannerRequest(scanner=Scanner(table=\"tsdb\", start_key=[0, 0, 1, 89, -93, 93, 0], stop_key=\"\x00\x00\x01Y\xA3y \", columns={\"t\"}, populate_blockcache=true, max_num_rows=128, max_num_kvs=4096, region=null, filter=KeyRegexpFilter(\"(?s)^.{7}(?:.{6})\Q\u0000\u0000\u0001\E(?:\Q\u0000›I\E)(?:.{6})\Q\u0000\u0000\u0004\E(?:\Q\u0000\u0000\u0004\E)(?:.{6})$\", ISO-8859-1), scanner_id=0x0000000000000000))\n\tat net.opentsdb.tsd.QueryRpc$1ErrorCB.call(QueryRpc.java:180) [tsdb-2.2.0.jar:]\n\tat net.opentsdb.tsd.QueryRpc$1ErrorCB.call(QueryRpc.java:158) [tsdb-2.2.0.jar:]\n\tat com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.access$300(Deferred.java:430) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred$Continue.call(Deferred.java:1366) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.callback(Deferred.java:1005) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.DeferredGroup.done(DeferredGroup.java:169) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.DeferredGroup.recordCompletion(DeferredGroup.java:158) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.DeferredGroup.access$200(DeferredGroup.java:36) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.DeferredGroup$1NotifyOrdered.call(DeferredGroup.java:97) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.callback(Deferred.java:1005) [async-1.4.0.jar:na]\n\tat net.opentsdb.core.TsdbQuery$1ScannerCB.close(TsdbQuery.java:794) [tsdb-2.2.0.jar:]\n\tat net.opentsdb.core.TsdbQuery$1ScannerCB$ErrorCB.call(TsdbQuery.java:594) [tsdb-2.2.0.jar:]\n\tat net.opentsdb.core.TsdbQuery$1ScannerCB$ErrorCB.call(TsdbQuery.java:590) [tsdb-2.2.0.jar:]\n\tat com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.access$300(Deferred.java:430) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred$Continue.call(Deferred.java:1366) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.access$300(Deferred.java:430) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred$Continue.call(Deferred.java:1366) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.access$300(Deferred.java:430) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred$Continue.call(Deferred.java:1366) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.access$300(Deferred.java:430) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred$Continue.call(Deferred.java:1366) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.access$300(Deferred.java:430) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred$Continue.call(Deferred.java:1366) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.access$300(Deferred.java:430) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred$Continue.call(Deferred.java:1366) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.access$300(Deferred.java:430) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred$Continue.call(Deferred.java:1366) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.access$300(Deferred.java:430) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred$Continue.call(Deferred.java:1366) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.access$300(Deferred.java:430) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred$Continue.call(Deferred.java:1366) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.access$300(Deferred.java:430) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred$Continue.call(Deferred.java:1366) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1313) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.doCall(Deferred.java:1284) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.callback(Deferred.java:1005) [async-1.4.0.jar:na]\n\tat org.hbase.async.HBaseClient$ZKClient$ZKCallback.processResult(HBaseClient.java:3632) [asynchbase-1.7.1.jar:na]\n\tat org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:558) [zookeeper-3.4.5.jar:3.4.5-1392090]\n\tat org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495) [zookeeper-3.4.5.jar:3.4.5-1392090]\n"}}

IDerr commented 7 years ago

Same error, your TSD tries to open a scanner, but can't. check your hbase logs.

IDerr commented 6 years ago

@xorangelee

ace-han commented 6 years ago

@manolama can you provide the exact tuning configured items in Hbase as in tune HBase some more?

This error is happening in my cluster, too. Thx in advance

datto-aparrill commented 4 years ago

I get this error when my application tries to query opentsdb in parallel. If I query in series (one request after another) everything is fine.

PS: The 424 code is wrong here; it's only supposed to be used with webdav. Additionally its using a 4xx error code for a server-side issue.

johnseekins commented 4 years ago

As Alex noted, we see this error on query nodes...kinda randomly. The concern is that we'll see a consistent "too many attempts" error against one region for hours at a time. I understand this error during a region split (and that makes sense), but it shouldn't persist for 18+ hours until we restart the TSDs.

Traserve commented 3 years ago

i got the same error,but can't find anything unusual in hbase logs.

linweisen commented 3 years ago

Same error