search

OpenTSDB / opentsdb

A scalable, distributed Time Series Database.
http://opentsdb.net
GNU Lesser General Public License v2.1
4.99k stars 1.25k forks source link

opentsdb hang under load #557

Open kramerli opened 9 years ago

kramerli commented 9 years ago

I try to put data into opentsdb by put api. The data is prepared in several files. Each contains 5million records. I can manage to put a few of them. But after a few miniutes. Opentsdb throw some error in the log and the put process is failed.

The log is listed below:


2015-08-31 21:56:48,360 INFO  [New I/O worker #48] HttpQuery: [id: 0x38971101, /10.75.44.33:34549 :> /10.75.44.33:4242] HTTP /api/put?details done in 388999ms
2015-08-31 21:56:48,360 INFO  [New I/O server boss #65] ConnectionManager: [id: 0x835153bf, /10.75.44.33:34618 => /10.75.44.33:4242] OPEN
2015-08-31 21:56:48,361 INFO  [New I/O worker #40] ConnectionManager: [id: 0x835153bf, /10.75.44.33:34618 => /10.75.44.33:4242] BOUND: /10.75.44.33:4242
2015-08-31 21:56:48,361 INFO  [New I/O server boss #65] ConnectionManager: [id: 0xdf86eb92, /0:0:0:0:0:0:0:1:35607 => /0:0:0:0:0:0:0:1:4242] OPEN
2015-08-31 21:56:48,361 INFO  [New I/O worker #40] ConnectionManager: [id: 0x835153bf, /10.75.44.33:34618 => /10.75.44.33:4242] CONNECTED: /10.75.44.33:34618
2015-08-31 21:56:48,361 INFO  [New I/O worker #41] ConnectionManager: [id: 0xdf86eb92, /0:0:0:0:0:0:0:1:35607 => /0:0:0:0:0:0:0:1:4242] BOUND: /0:0:0:0:0:0:0:1:4242
2015-08-31 21:56:48,361 INFO  [New I/O worker #41] ConnectionManager: [id: 0xdf86eb92, /0:0:0:0:0:0:0:1:35607 => /0:0:0:0:0:0:0:1:4242] CONNECTED: /0:0:0:0:0:0:0:1:35607
2015-08-31 21:56:48,361 INFO  [New I/O worker #41] HttpQuery: [id: 0xdf86eb92, /0:0:0:0:0:0:0:1:35607 => /0:0:0:0:0:0:0:1:4242] HTTP /api/version done in 0ms
2015-08-31 21:56:48,361 INFO  [New I/O worker #41] ConnectionManager: [id: 0xdf86eb92, /0:0:0:0:0:0:0:1:35607 :> /0:0:0:0:0:0:0:1:4242] DISCONNECTED
2015-08-31 21:56:48,361 INFO  [New I/O worker #41] ConnectionManager: [id: 0xdf86eb92, /0:0:0:0:0:0:0:1:35607 :> /0:0:0:0:0:0:0:1:4242] UNBOUND
2015-08-31 21:56:48,362 INFO  [New I/O worker #41] ConnectionManager: [id: 0xdf86eb92, /0:0:0:0:0:0:0:1:35607 :> /0:0:0:0:0:0:0:1:4242] CLOSED

2015-08-31 21:58:23,436 ERROR [New I/O worker #40] ConnectionManager: Unexpected exception from downstream for [id: 0x835153bf, /10.75.44.33:34618 => /10.75.44.33:4242]
java.lang.OutOfMemoryError: Java heap space
    at java.nio.HeapCharBuffer.<init>(HeapCharBuffer.java:57) ~[na:1.7.0_85]
    at java.nio.CharBuffer.allocate(CharBuffer.java:331) ~[na:1.7.0_85]
    at org.jboss.netty.buffer.ChannelBuffers.decodeString(ChannelBuffers.java:1193) ~[netty-3.9.4.Final.jar:na]
    at org.jboss.netty.buffer.AbstractChannelBuffer.toString(AbstractChannelBuffer.java:551) ~[netty-3.9.4.Final.jar:na]
    at org.jboss.netty.buffer.AbstractChannelBuffer.toString(AbstractChannelBuffer.java:543) ~[netty-3.9.4.Final.jar:na]
    at net.opentsdb.tsd.HttpQuery.getContent(HttpQuery.java:459) ~[tsdb-2.1.0.jar:c775b5f]
    at net.opentsdb.tsd.HttpJsonSerializer.parsePutV1(HttpJsonSerializer.java:133) ~[tsdb-2.1.0.jar:c775b5f]
    at net.opentsdb.tsd.PutDataPointRpc.execute(PutDataPointRpc.java:102) ~[tsdb-2.1.0.jar:c775b5f]
    at net.opentsdb.tsd.RpcHandler.handleHttpQuery(RpcHandler.java:273) ~[tsdb-2.1.0.jar:c775b5f]
    at net.opentsdb.tsd.RpcHandler.messageReceived(RpcHandler.java:180) ~[tsdb-2.1.0.jar:c775b5f]
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[netty-3.9.4.Final.jar:na]
    at org.jboss.netty.handler.timeout.IdleStateAwareChannelUpstreamHandler.handleUpstream(IdleStateAwareChannelUpstreamHandler.java:36) ~[netty-3.9.4.Final.jar:na]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[netty-3.9.4.Final.jar:na]
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[netty-3.9.4.Final.jar:na]
    at org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:294) ~[netty-3.9.4.Final.jar:na]
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[netty-3.9.4.Final.jar:na]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[netty-3.9.4.Final.jar:na]
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[netty-3.9.4.Final.jar:na]
    at org.jboss.netty.handler.codec.http.HttpContentEncoder.messageReceived(HttpContentEncoder.java:82) ~[netty-3.9.4.Final.jar:na]
    at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) ~[netty-3.9.4.Final.jar:na]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[netty-3.9.4.Final.jar:na]
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[netty-3.9.4.Final.jar:na]
    at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108) ~[netty-3.9.4.Final.jar:na]
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[netty-3.9.4.Final.jar:na]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[netty-3.9.4.Final.jar:na]
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[netty-3.9.4.Final.jar:na]
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) ~[netty-3.9.4.Final.jar:na]
    at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:194) ~[netty-3.9.4.Final.jar:na]
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[netty-3.9.4.Final.jar:na]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[netty-3.9.4.Final.jar:na]
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[netty-3.9.4.Final.jar:na]
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) ~[netty-3.9.4.Final.jar:na]
2015-08-31 21:58:57,956 INFO  [New I/O worker #40] ConnectionManager: [id: 0x835153bf, /10.75.44.33:34618 :> /10.75.44.33:4242] DISCONNECTED
2015-08-31 21:58:57,956 INFO  [New I/O worker #40] ConnectionManager: [id: 0x835153bf, /10.75.44.33:34618 :> /10.75.44.33:4242] UNBOUND
2015-08-31 21:58:57,956 INFO  [New I/O worker #40] ConnectionManager: [id: 0x835153bf, /10.75.44.33:34618 :> /10.75.44.33:4242] CLOSED
2015-08-31 21:59:05,424 INFO  [New I/O server boss #65] ConnectionManager: [id: 0x9a40399c, /10.75.44.33:34625 => /10.75.44.33:4242] OPEN
2015-08-31 21:59:05,424 INFO  [New I/O worker #42] ConnectionManager: [id: 0x9a40399c, /10.75.44.33:34625 => /10.75.44.33:4242] BOUND: /10.75.44.33:4242
2015-08-31 21:59:05,424 INFO  [New I/O worker #42] ConnectionManager: [id: 0x9a40399c, /10.75.44.33:34625 => /10.75.44.33:4242] CONNECTED: /10.75.44.33:34625
2015-08-31 22:02:21,615 ERROR [New I/O worker #31] ConnectionManager: Unexpected exception from downstream for [id: 0xec533f17, /10.75.44.33:34599 => /10.75.44.33:4242]
java.lang.OutOfMemoryError: GC overhead limit exceeded
    at com.google.protobuf.ZeroCopyLiteralByteString.wrap(ZeroCopyLiteralByteString.java:52) ~[asynchbase-1.6.0.jar:na]
    at org.hbase.async.Bytes.wrap(Bytes.java:287) ~[asynchbase-1.6.0.jar:na]
    at org.hbase.async.PutRequest.toMutationProto(PutRequest.java:529) ~[asynchbase-1.6.0.jar:na]
    at org.hbase.async.MultiAction.serialize(MultiAction.java:229) ~[asynchbase-1.6.0.jar:na]
    at org.hbase.async.RegionClient.encode(RegionClient.java:1146) ~[asynchbase-1.6.0.jar:na]
    at org.hbase.async.RegionClient.sendRpc(RegionClient.java:894) ~[asynchbase-1.6.0.jar:na]
    at org.hbase.async.RegionClient.bufferEdit(RegionClient.java:757) ~[asynchbase-1.6.0.jar:na]
    at org.hbase.async.RegionClient.sendRpc(RegionClient.java:881) ~[asynchbase-1.6.0.jar:na]
    at org.hbase.async.HBaseClient.sendRpcToRegion(HBaseClient.java:1698) ~[asynchbase-1.6.0.jar:na]
    at org.hbase.async.HBaseClient.put(HBaseClient.java:1343) ~[asynchbase-1.6.0.jar:na]
    at net.opentsdb.core.TSDB.addPointInternal(TSDB.java:681) ~[tsdb-2.1.0.jar:c775b5f]
    at net.opentsdb.core.TSDB.addPoint(TSDB.java:573) ~[tsdb-2.1.0.jar:c775b5f]
    at net.opentsdb.tsd.PutDataPointRpc.execute(PutDataPointRpc.java:146) ~[tsdb-2.1.0.jar:c775b5f]
    at net.opentsdb.tsd.RpcHandler.handleHttpQuery(RpcHandler.java:273) ~[tsdb-2.1.0.jar:c775b5f]
    at net.opentsdb.tsd.RpcHandler.messageReceived(RpcHandler.java:180) ~[tsdb-2.1.0.jar:c775b5f]
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[netty-3.9.4.Final.jar:na]
    at org.jboss.netty.handler.timeout.IdleStateAwareChannelUpstreamHandler.handleUpstream(IdleStateAwareChannelUpstreamHandler.java:36) ~[netty-3.9.4.Final.jar:na]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[netty-3.9.4.Final.jar:na]
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[netty-3.9.4.Final.jar:na]
    at org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:294) ~[netty-3.9.4.Final.jar:na]
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[netty-3.9.4.Final.jar:na]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[netty-3.9.4.Final.jar:na]
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[netty-3.9.4.Final.jar:na]
    at org.jboss.netty.handler.codec.http.HttpContentEncoder.messageReceived(HttpContentEncoder.java:82) ~[netty-3.9.4.Final.jar:na]
    at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) ~[netty-3.9.4.Final.jar:na]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[netty-3.9.4.Final.jar:na]
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[netty-3.9.4.Final.jar:na]
    at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108) ~[netty-3.9.4.Final.jar:na]
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[netty-3.9.4.Final.jar:na]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[netty-3.9.4.Final.jar:na]
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[netty-3.9.4.Final.jar:na]
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) ~[netty-3.9.4.Final.jar:na]
2015-08-31 22:02:21,616 INFO  [New I/O worker #31] ConnectionManager: [id: 0xec533f17, /10.75.44.33:34599 :> /10.75.44.33:4242] DISCONNECTED
2015-08-31 22:02:21,616 INFO  [New I/O worker #31] ConnectionManager: [id: 0xec533f17, /10.75.44.33:34599 :> /10.75.44.33:4242] UNBOUND
2015-08-31 22:02:21,616 INFO  [New I/O worker #31] ConnectionManager: [id: 0xec533f17, /10.75.44.33:34599 :> /10.75.44.33:4242] CLOSED
2015-08-31 22:02:21,616 ERROR [New I/O worker #66] RegionClient: Unexpected exception from downstream on [id: 0x49bdc01a, /10.75.44.33:40244 => /10.75.44.36:18020]
java.lang.OutOfMemoryError: GC overhead limit exceeded
    at com.google.protobuf.ZeroCopyLiteralByteString.wrap(ZeroCopyLiteralByteString.java:52) ~[asynchbase-1.6.0.jar:na]
    at org.hbase.async.Bytes.wrap(Bytes.java:287) ~[asynchbase-1.6.0.jar:na]
    at org.hbase.async.GetRequest.serialize(GetRequest.java:385) ~[asynchbase-1.6.0.jar:na]
    at org.hbase.async.RegionClient.encode(RegionClient.java:1146) ~[asynchbase-1.6.0.jar:na]
    at org.hbase.async.RegionClient.sendRpc(RegionClient.java:894) ~[asynchbase-1.6.0.jar:na]
    at org.hbase.async.HBaseClient.sendRpcToRegion(HBaseClient.java:1698) ~[asynchbase-1.6.0.jar:na]
    at org.hbase.async.HBaseClient.get(HBaseClient.java:995) ~[asynchbase-1.6.0.jar:na]
    at net.opentsdb.core.TSDB.get(TSDB.java:1090) ~[tsdb-2.1.0.jar:c775b5f]
    at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:185) ~[tsdb-2.1.0.jar:c775b5f]
    at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:116) ~[tsdb-2.1.0.jar:c775b5f]
    at net.opentsdb.core.TSDB.flush(TSDB.java:730) ~[tsdb-2.1.0.jar:c775b5f]
    at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:191) ~[tsdb-2.1.0.jar:c775b5f]
    at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:116) ~[tsdb-2.1.0.jar:c775b5f]
    at net.opentsdb.core.TSDB.flush(TSDB.java:730) ~[tsdb-2.1.0.jar:c775b5f]
    at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:191) ~[tsdb-2.1.0.jar:c775b5f]
    at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:116) ~[tsdb-2.1.0.jar:c775b5f]
    at net.opentsdb.core.TSDB.flush(TSDB.java:730) ~[tsdb-2.1.0.jar:c775b5f]
    at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:191) ~[tsdb-2.1.0.jar:c775b5f]
    at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:116) ~[tsdb-2.1.0.jar:c775b5f]
    at net.opentsdb.core.TSDB.flush(TSDB.java:730) ~[tsdb-2.1.0.jar:c775b5f]
    at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:191) ~[tsdb-2.1.0.jar:c775b5f]
    at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:116) ~[tsdb-2.1.0.jar:c775b5f]
    at net.opentsdb.core.TSDB.flush(TSDB.java:730) ~[tsdb-2.1.0.jar:c775b5f]
    at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:191) ~[tsdb-2.1.0.jar:c775b5f]
    at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:116) ~[tsdb-2.1.0.jar:c775b5f]
    at net.opentsdb.core.TSDB.flush(TSDB.java:730) ~[tsdb-2.1.0.jar:c775b5f]
    at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:191) ~[tsdb-2.1.0.jar:c775b5f]
    at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:116) ~[tsdb-2.1.0.jar:c775b5f]
    at net.opentsdb.core.TSDB.flush(TSDB.java:730) ~[tsdb-2.1.0.jar:c775b5f]
    at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:191) ~[tsdb-2.1.0.jar:c775b5f]
    at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:116) ~[tsdb-2.1.0.jar:c775b5f]
    at net.opentsdb.core.TSDB.flush(TSDB.java:730) ~[tsdb-2.1.0.jar:c775b5f]
2015-08-31 22:02:21,616 INFO  [New I/O worker #66] HBaseClient: Lost connection with the .META. region
2015-08-31 22:02:21,617 ERROR [New I/O worker #67] RegionClient: Unexpected exception from downstream on [id: 0x97cc2014, /10.75.44.33:44182 => /10.75.44.35:18020]
java.lang.OutOfMemoryError: GC overhead limit exceeded
    at org.hbase.async.HBaseClient.createRegionSearchKey(HBaseClient.java:1954) ~[asynchbase-1.6.0.jar:na]
    at org.hbase.async.HBaseClient.getRegion(HBaseClient.java:1984) ~[asynchbase-1.6.0.jar:na]
    at org.hbase.async.HBaseClient.sendRpcToRegion(HBaseClient.java:1659) ~[asynchbase-1.6.0.jar:na]
    at org.hbase.async.HBaseClient.get(HBaseClient.java:995) ~[asynchbase-1.6.0.jar:na]
    at net.opentsdb.core.TSDB.get(TSDB.java:1090) ~[tsdb-2.1.0.jar:c775b5f]
    at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:185) ~[tsdb-2.1.0.jar:c775b5f]
    at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:116) ~[tsdb-2.1.0.jar:c775b5f]
    at net.opentsdb.core.TSDB.flush(TSDB.java:730) ~[tsdb-2.1.0.jar:c775b5f]
    at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:191) ~[tsdb-2.1.0.jar:c775b5f]
    at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:116) ~[tsdb-2.1.0.jar:c775b5f]
    at net.opentsdb.core.TSDB.flush(TSDB.java:730) ~[tsdb-2.1.0.jar:c775b5f]
    at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:191) ~[tsdb-2.1.0.jar:c775b5f]
    at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:116) ~[tsdb-2.1.0.jar:c775b5f]
    at net.opentsdb.core.TSDB.flush(TSDB.java:730) ~[tsdb-2.1.0.jar:c775b5f]
    at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:191) ~[tsdb-2.1.0.jar:c775b5f]
    at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:116) ~[tsdb-2.1.0.jar:c775b5f]
    at net.opentsdb.core.TSDB.flush(TSDB.java:730) ~[tsdb-2.1.0.jar:c775b5f]
    at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:191) ~[tsdb-2.1.0.jar:c775b5f]
    at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:116) ~[tsdb-2.1.0.jar:c775b5f]
    at net.opentsdb.core.TSDB.flush(TSDB.java:730) ~[tsdb-2.1.0.jar:c775b5f]
    at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:191) ~[tsdb-2.1.0.jar:c775b5f]
    at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:116) ~[tsdb-2.1.0.jar:c775b5f]
    at net.opentsdb.core.TSDB.flush(TSDB.java:730) ~[tsdb-2.1.0.jar:c775b5f]
    at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:191) ~[tsdb-2.1.0.jar:c775b5f]
    at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:116) ~[tsdb-2.1.0.jar:c775b5f]
    at net.opentsdb.core.TSDB.flush(TSDB.java:730) ~[tsdb-2.1.0.jar:c775b5f]
    at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:191) ~[tsdb-2.1.0.jar:c775b5f]
    at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:116) ~[tsdb-2.1.0.jar:c775b5f]
    at net.opentsdb.core.TSDB.flush(TSDB.java:730) ~[tsdb-2.1.0.jar:c775b5f]
    at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:191) ~[tsdb-2.1.0.jar:c775b5f]
    at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:116) ~[tsdb-2.1.0.jar:c775b5f]
    at net.opentsdb.core.TSDB.flush(TSDB.java:730) ~[tsdb-2.1.0.jar:c775b5f]
2015-08-31 22:03:09,322 INFO  [New I/O server boss #65] ConnectionManager: [id: 0x7c5ac3e7, /10.75.44.33:34633 => /10.75.44.33:4242] OPEN
2015-08-31 22:03:09,322 INFO  [New I/O worker #43] ConnectionManager: [id: 0x7c5ac3e7, /10.75.44.33:34633 => /10.75.44.33:4242] BOUND: /10.75.44.33:4242
2015-08-31 22:03:09,322 INFO  [New I/O worker #43] ConnectionManager: [id: 0x7c5ac3e7, /10.75.44.33:34633 => /10.75.44.33:4242] CONNECTED: /10.75.44.33:34633
2015-08-31 22:03:09,322 ERROR [New I/O worker #39] ConnectionManager: Unexpected exception from downstream for [id: 0xe5cfc336, /10.75.44.33:34614 => /10.75.44.33:4242]
java.lang.OutOfMemoryError: GC overhead limit exceeded
2015-08-31 22:03:09,322 INFO  [New I/O worker #39] ConnectionManager: [id: 0xe5cfc336, /10.75.44.33:34614 :> /10.75.44.33:4242] DISCONNECTED
2015-08-31 22:03:09,322 INFO  [New I/O worker #39] ConnectionManager: [id: 0xe5cfc336, /10.75.44.33:34614 :> /10.75.44.33:4242] UNBOUND
2015-08-31 22:03:09,322 INFO  [New I/O worker #39] ConnectionManager: [id: 0xe5cfc336, /10.75.44.33:34614 :> /10.75.44.33:4242] CLOSED
manolama commented 9 years ago

In this case you're just pushing too much data for the compaction queue and possibly HBase to handle. Try disabling compactions if you don't mind using a bit more storage space, or try throttling back your writes. Also instead of using the HTTP interface, you may want to dump your data into files and use the import command.

kramerli commented 9 years ago

@manolama

Really thanks for your response. I manage to avoid this issue by slow down the put speed. However I have a few more question below.

  1. How can I locate the issue. I mean to find out it is the compaction queue issue. Any hint? I was even suspecting it is a Hbase throughput issue. So I did something like pre-split region which did not solve the problem
  2. Any other bad effecting besides the storage space if I disable the compaction? Like the query speed and so on.

Thanks in advance for your help.