search

OpenTSDB / opentsdb

A scalable, distributed Time Series Database.
http://opentsdb.net
GNU Lesser General Public License v2.1
5k stars 1.25k forks source link

High CPU Uitlization on OpenTSDB #818

Open palgbpec opened 8 years ago

palgbpec commented 8 years ago

Hi,

We are using OpenTsdb 2.2.0 version. The Opentsdb CPU Utilization is going to 95-99 % and after that OpenTSDB restarts automatically. We are pushing the data at a very high rate. Also we are seeing that the OpenTSDB recovers once the client pushing the data is stopped. The problem for our OpenTSDB is somewhat similar to https://github.com/OpenTSDB/opentsdb/issues/120 . We are running three OpenTSDB instances each running on OS-Red Hat Enterprise Linux Server release 6.6 (Santiago) and 8 core cpu behind a load balancer and we are seeing high Cpu utilization on all the machines. We have taken thread dump when the CPU utilization was high using this command jstack -l and we are seeing most of the threads in a blocked state.

O/p: ..... 1 - org.jboss.netty.util.HashedWheelTimer$Worker.run() @bci=43, line=364 (Compiled frame) 1 - org.jboss.netty.util.HashedWheelTimer$Worker.run() @bci=82, line=369 (Compiled frame) 1 - org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick() @bci=81, line=445 (Compiled frame) 1 - sun.nio.ch.FileDispatcherImpl.write0(java.io.FileDescriptor, long, int) @bci=0 (Compiled frame) 1 - sun.nio.ch.FileDispatcherImpl.write0(java.io.FileDescriptor, long, int) @bci=0 (Compiled frame; information may be imprecise) 1 Attaching to process ID 12687, please wait... 1 Deadlock Detection: 1 Debugger attached successfully. 1 JVM version is 25.77-b03 1 No deadlocks found. 1 Server compiler detected. 1 Thread 12691: (state = BLOCKED) 1 Thread 12701: (state = BLOCKED) 1 Thread 12702: (state = BLOCKED) 1 Thread 12703: (state = BLOCKED) 1 Thread 12712: (state = BLOCKED) 1 Thread 12713: (state = BLOCKED) 1 Thread 12714: (state = BLOCKED) 1 Thread 12715: (state = BLOCKED) 1 Thread 12716: (state = BLOCKED) 1 Thread 12717: (state = BLOCKED) 1 Thread 12718: (state = BLOCKED) 1 Thread 12719: (state = BLOCKED) 1 Thread 12720: (state = BLOCKED) 1 Thread 12721: (state = BLOCKED) 1 Thread 12722: (state = BLOCKED) 1 Thread 12723: (state = BLOCKED)

........

palgbpec commented 8 years ago

Hi,

One more thing we noticed while analyzing the thread dump is most of the threads are in the blocked state which are waiting to acquire a lock for writing to the Opentsdb log file. Below are the logs

"OpenTSDB Old I/O server worker (parentId: 116426854, [id: 0x06f08866, /0.0.0.0:4242])" #155 prio=5 os_prio=0 tid=0x00007f037c112000 nid=0xc45d waiting for monitor entry [0x00007f030a4a2000] java.lang.Thread.State: BLOCKED (on object monitor) at ch.qos.logback.core.OutputStreamAppender.subAppend(OutputStreamAppender.java:211)

threaddumpreport