API: Hot Threads information

[ph]

Currently there is no way to get information about the running threads inside logstash other than collecting a thread dump with jstack It would be great to implement an API endpoint similar to the elasticsearch's hot threads. Ref: https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-nodes-hot-threads.html

I think this can be done easily by using the java management api, we could probably use elasticsearch's class directly and return the same results.

In my initial experimentations you have to use the java api because the ruby api will hide the pure java threads, like the one used in Manticore or the rabbit mq client.

[ph]

@purbon I know you want to add the hot threads information to the metric store, but I currently dont see how this information will be saved with the current structure type?

In my head, I would only have made this information available in the web api? Can you give me some light into that?

[purbon]

@ph you told me to use https://github.com/elastic/logstash/blob/feature/metrics/logstash-core/lib/logstash/instrument/periodic_poller/jvm.rb, something has changed? I'm just starting now the integration, so still not investigated anything.

[ph]

The periodic poller is the thing to use for the recurring task, lets take the the raw output of the hot_threads in es.

::: {Blackbird}{WiFg0SKSSNqf2-JObGulQw}{127.0.0.1}{127.0.0.1:9300}
   Hot threads at 2016-01-12T15:51:31.631Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:

    0.1% (296micros out of 500ms) cpu usage by thread 'elasticsearch[Blackbird][transport_client_timer][T#1]{Hashed wheel timer #1}'
     10/10 snapshots sharing following 5 elements
       java.lang.Thread.sleep(Native Method)
       org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:445)
       org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:364)
       org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
       java.lang.Thread.run(Thread.java:745)

I am wondering if this infos should/how to be saved in the metric store?

[purbon]

@ph this is a good question, will think about it.

[purbon]

This is the actual hot threads text ouput

::: {0.0.0.0} 
 Hot threads at 2016-01-22 13:41:12 +0100, busiestThreads=3:
     161453 micros of cpu usage by timed_waiting thread named 'Ruby-0-Thread-12'
         /Users/purbon/work/logstash/logstash-core/lib/logstash/agent.rb:83
        java.lang.Object.wait(Native Method)
        java.lang.Thread.join(Thread.java:1289)
        org.jruby.internal.runtime.NativeThread.join(NativeThread.java:75)

     60427 micros of cpu usage by waiting thread named 'pool-1-thread-2'
        sun.misc.Unsafe.park(Native Method)
        java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

     28635 micros of cpu usage by timed_waiting thread named 'pool-2-thread-1'
        java.lang.Object.wait(Native Method)
        java.lang.Object.wait(Object.java:461)
        org.jruby.RubyThread$SleepTask.run(RubyThread.java:1050)

is there something else you would like to see? anything missing?

[ph]

Cool layout!

Are we showing idle threads?

• Might be cool to add at the top the nodename, you can get it by calling the agent.
• Also cpu usage in % ?
• The time return in ISO 8601 format.

[ph]

As discussed. lets add a parameter to get longer stacktrace? The default is currently 3

[purbon]

We can call this done by the merge #4641, this does not mean we could extend or fix the format in next iterations or during the merge to master.