An Apache Storm IMetricsConsumer that forwards Storm's built-in metrics to a Graphite server for real-time graphing, visualization, and operational dashboards.
Other
75
stars
38
forks
source link
Socket timeouts are not set while connecting to graphite #26
I am facing this problem where my entire storm topology gets stuck when the socket connection open to graphite gets stuck because of the timeout not being set. So I feel the tuples which enter the GraphiteMetricsConsumer don't get acked, my queue size keeps growing and eventually new tuples stop getting emitted.
Following is the jstack corresponding to it:
"Thread-205-__metricscom.verisign.storm.metrics.GraphiteMetricsConsumer" #244 prio=5 os_prio=0 tid=0x00007f10591f1800 nid=0x6a1e runnable [0x00007f0fd3bf9000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
locked <0x000000065e5d0068> (a java.net.SocksSocketImpl)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at java.net.Socket.(Socket.java:434)
at java.net.Socket.(Socket.java:244)
at javax.net.DefaultSocketFactory.createSocket(SocketFactory.java:277)
at com.verisign.storm-graphite.com.codahale.metrics.graphite.Graphite.connect(Graphite.java:118)
at com.verisign.storm.metrics.reporters.graphite.GraphiteReporter.connect(GraphiteReporter.java:104)
at com.verisign.storm.metrics.reporters.graphite.GraphiteReporter.handleFailedSend(GraphiteReporter.java:174)
at com.verisign.storm.metrics.reporters.graphite.GraphiteReporter.appendToBuffer(GraphiteReporter.java:143)
at com.verisign.storm.metrics.GraphiteMetricsConsumer.appendToReporterBuffer(GraphiteMetricsConsumer.java:311)
at com.verisign.storm.metrics.GraphiteMetricsConsumer.handleDataPoints(GraphiteMetricsConsumer.java:227)
at backtype.storm.metric.MetricsConsumerBolt.execute(MetricsConsumerBolt.java:55)
at backtype.storm.daemon.executor$fn6647$tuple_action_fn6649.invoke(executor.clj:633)
at backtype.storm.daemon.executor$mk_task_receiver$fn__6570.invoke(executor.clj:401)
Hi,
I am facing this problem where my entire storm topology gets stuck when the socket connection open to graphite gets stuck because of the timeout not being set. So I feel the tuples which enter the GraphiteMetricsConsumer don't get acked, my queue size keeps growing and eventually new tuples stop getting emitted.
Following is the jstack corresponding to it:
"Thread-205-__metricscom.verisign.storm.metrics.GraphiteMetricsConsumer" #244 prio=5 os_prio=0 tid=0x00007f10591f1800 nid=0x6a1e runnable [0x00007f0fd3bf9000] java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
Thanks, Varsha